eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am asking this question because I am successfully training a segmentation network on my GTX 2070 on laptop with 8GB VRAM and I use exactly the same code and exactly the same software libraries installed on my desktop PC with a GTX 1080TI and it still throws out of memory. Why does this happen, considering that:  The same Windows 10 + CUDA 10.1 + CUDNN 7.6.5.32 + Nvidia Driver 418.96 (comes along with CUDA 10.1) are both on laptop and on PC.  The fact that training with TensorFlow 2.3 runs smoothly on the GPU on my PC, yet it fails allocating memory for training only with PyTorch.  PyTorch recognises the GPU (prints GTX 1080 TI) via the command : print(torch.cuda.get_device_name(0))  PyTorch allocates memory when running this command: torch.rand(20000, 20000).cuda() #allocated 1.5GB of VRAM.   What is the solution to this?

Answer: Most of the people (even in the thread below) jump to suggest that decreasing the batch_size will solve this problem. In fact, it does not in this case. For example, it would have been illogical for a network to train on 8GB VRAM and yet to fail to train on 11GB VRAM, considering that there were no other applications consuming video memory on the system with 11GB VRAM and the exact same configuration is installed and used. The reason why this happened in my case was that, when using the DataLoader object, I set a very high (12) value for the workers parameter. Decreasing this value to 4 in my case solved the problem. In fact, although at the bottom of the thread, the answer provided by Yurasyk at https://github.com/pytorch/pytorch/issues/16417#issuecomment-599137646 pointed me in the right direction. Solution: Decrease the number of workers in the PyTorch DataLoader. Although I do not exactly understand why this solution works, I assume it is related to the threads spawned behind the scenes for data fetching; it may be the case that, on some processors, such an error appears.
Question: Why am I receiving this error when I run this code? Traceback (most recent call last): File "main.py", line 13, in <module> def twoSum(self, nums: list[int], target: int) -> list[int]: TypeError: 'type' object is not subscriptable  nums = [4,5,6,7,8,9] target = 13 def twoSum(self, nums: list[int], target: int) -> list[int]: dictionary = {} answer = [] for i in range(len(nums)): secondNumber = target-nums[i] if(secondNumber in dictionary.keys()): secondIndex = nums.index(secondNumber) if(i != secondIndex): return sorted([i, secondIndex]) dictionary.update({nums[i]: i}) print(twoSum(nums, target)) 

Answer: The following answer only applies to Python < 3.9 The expression list[int] is attempting to subscript the object list, which is a class. Class objects are of the type of their metaclass, which is type in this case. Since type does not define a __getitem__ method, you can't do list[...]. To do this correctly, you need to import typing.List and use that instead of the built-in list in your type hints: from typing import List ... def twoSum(self, nums: List[int], target: int) -> List[int]:  If you want to avoid the extra import, you can simplify the type hints to exclude generics: def twoSum(self, nums: list, target: int) -> list:  Alternatively, you can get rid of type hinting completely: def twoSum(self, nums, target):
Question: These last few days, I've been trying to adapt the structure of a discord bot written in discord.py to a more OOP one (because having functions lying around isn't ideal). But I have found way more problems that I could have ever expected. The thing is that I want to encapsulate all my commands into a single class, but I don't know what decorators to use and how and which classes I must inherit. What I've achieved so far is code like the snippet down below. It runs, but at the moment of executing a command it throws errors like  discord.ext.commands.errors.CommandNotFound: Command "status" is not found  I'm using Python 3.6. from discord.ext import commands class MyBot(commands.Bot): def __init__(self, command_prefix, self_bot): commands.Bot.__init__(self, command_prefix=command_prefix, self_bot=self_bot) self.message1 = "[INFO]: Bot now online" self.message2 = "Bot still online {}" async def on_ready(self): print(self.message1) @commands.command(name="status", pass_context=True) async def status(self, ctx): print(ctx) await ctx.channel.send(self.message2 + ctx.author) bot = MyBot(command_prefix="!", self_bot=False) bot.run("token") 

Answer: To register the command you should use self.add_command(setup), but you can't have the self argument in the setup method, so you could do something like this: from discord.ext import commands class MyBot(commands.Bot): def __init__(self, command_prefix, self_bot): commands.Bot.__init__(self, command_prefix=command_prefix, self_bot=self_bot) self.message1 = "[INFO]: Bot now online" self.message2 = "Bot still online" self.add_commands() async def on_ready(self): print(self.message1) def add_commands(self): @self.command(name="status", pass_context=True) async def status(ctx): print(ctx) await ctx.channel.send(self.message2, ctx.author.name) bot = MyBot(command_prefix="!", self_bot=False) bot.run("token")
Question: I keep failing to run pip install on the tensorflow package. First it downloads the .whl file, then goes through a bunch of already satisfied requirements until it gets to installing collected packages: tensorflow, at which point here's the error I get: ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\\Users\\Borik\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python38\\site-packages\\tensorflow\\include\\external\\com_github_grpc_grpc\\src\\core\\ext\\filters\\client_channel\\lb_policy\\grpclb\\client_load_reporting_filter.h'  I've never seen anything like this before and can't seem to find anything on the net. I'm using Windows 10 and the latest versions of Python and pip.

Answer: I hit the same issue on Win10. Rather than renaming my filesystem, I found a good solution in this Python documentation. To summarize the instructions there to change MAX_PATH, either:  Enable the "Enable Win32 long paths" group policy:  Run gpedit (or searching for "Edit Group Policy" in the Control Panel) Find the "Enable Win32 long paths" option in the sidebar. It should be under Local Computer Policy -> Computer Configuration -> Administrative Templates -> System -> Filesystem (under both Windows 10 and Windows 11, currently).   Or edit the registry setting HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem and set LongPathsEnabled to 1.  This extends path from char(256) to char(32000). After this change, my 'pip install tensorflow' succeeded.
Question: I installed the package weasyprint according to the instructions Installing weasyprint (Django project). My system: win 10. I have installed gtk3 and it is present in my PATH  import weasyprint ... @staff_member_required def order_admin_pdf(request, order_id): # Получаем заказ по ID: order = get_object_or_404(Order, id=order_id) # Передаем объект в функцию render_to через генерацию шаблона pdf.html HTML в виде строки: html = render_to_string('shop/orders/order_admin_pdf.html', {'order': order}) # Создаем объект овтета с типом содержимого application/pdf и заголовком Content-Disposition: response = HttpResponse(content_type='application/pdf') response['Content-Disposition'] = 'filename=order_{}.pdf"'.format(order.id) # Вызов метода weasyprint для получения PDF документа: weasyprint.HTML(string=html).write_pdf(response, stylesheets=[weasyprint.CSS( settings.STATIC_ROOT + 'css/pdf.css')]) return response   OSError: cannot load library 'gobject-2.0': error 0x7e. Additionally, ctypes.util.find_library() did not manage to locate a library called 'gobject-2.0' 

Answer: Starting from Python 3.8 DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If you followed the installation guide from the official documentation then the following example works. import os os.add_dll_directory(r"C:\Program Files\GTK3-Runtime Win64\bin") from weasyprint import HTML HTML('https://weasyprint.org/').write_pdf('weasyprint-website.pdf')  In essence you need to call add_dll_directory() before interacting with WeasyPrint.
Question: I've installed webdrivermanager on my windows-10 system C:\Users\username>pip install webdrivermanager Requirement already satisfied: webdrivermanager in c:\python\lib\site-packages (0.8.0) Requirement already satisfied: lxml in c:\python\lib\site-packages (from webdrivermanager) (4.5.1) Requirement already satisfied: requests in c:\python\lib\site-packages (from webdrivermanager) (2.20.1) Requirement already satisfied: tqdm in c:\python\lib\site-packages (from webdrivermanager) (4.46.1) Requirement already satisfied: appdirs in c:\python\lib\site-packages (from webdrivermanager) (1.4.4) Requirement already satisfied: BeautifulSoup4 in c:\python\lib\site-packages (from webdrivermanager) (4.6.0) Requirement already satisfied: certifi>=2017.4.17 in c:\python\lib\site-packages (from requests->webdrivermanager) (2018.11.29) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python\lib\site-packages (from requests->webdrivermanager) (3.0.4) Requirement already satisfied: idna<2.8,>=2.5 in c:\python\lib\site-packages (from requests->webdrivermanager) (2.7) Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\python\lib\site-packages (from requests->webdrivermanager) (1.23)  Still whenever I am trying to use webdrivermanager I'm facing an error.  Code Block: from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager driver = webdriver.Chrome(executable_path=ChromeDriverManager().install()) driver.get('https://www.google.com/')   Console Output: C:\Users\username\Desktop\Debanjan\PyPrograms>webdriverManagerChrome.py Traceback (most recent call last): File "C:\Users\username\Desktop\Debanjan\PyPrograms\webdriverManagerChrome.py", line 2, in <module> from webdriver_manager.chrome import ChromeDriverManager ModuleNotFoundError: No module named 'webdriver_manager'    Can someone help me, if I'm missing something? Incase it adds any value, I'm using sublimetext3

Answer: Update (thanks to Vishal Kharde) The documentation now suggests: pip install webdriver-manager   Solution: Install it like that: pip install webdriver_manager  instead of pip install webdrivermanager. Requirements: The newest version, according to the documentation supports python 3.6 or newer versions:  Reference: https://pypi.org/project/webdriver-manager/
Question: I'm am trying to fetch a serverless function from a react app in development mode with the following code. let response = await fetch(url, { method: 'POST', mode: 'cors', body: "param=" + paramVar, }) .then(response => response.json());  The backend function is a Python Cloud function with the following code: def main(request): # CORS handling if request.method == 'OPTIONS': # Allows GET requests from any origin with the Content-Type # header and caches preflight response for an 3600s headers = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST', 'Access-Control-Allow-Headers': 'Content-Type', 'Access-Control-Max-Age': '3600' } return ('', 204, headers) # Set CORS headers for the main request headers = { 'Content-Type':'application/json', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'Content-Type', } # Process request return (json.dumps(response), 200, headers)  But I keep getting the following error: Access to fetch at 'url' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.  When I try to perform the same request using curl I get a proper response. Using curl to get the options gives me the following: HTTP/2 204 access-control-allow-headers: Content-Type access-control-allow-methods: GET, POST access-control-allow-origin: * access-control-max-age: 3600 content-type: text/html; charset=utf-8 date: Sun, 16 Aug 2020 01:29:41 GMT  Anyone can help me understand why I'm not able to get a response at my front-end? The 'Access-Control-Allow-Origin' is present in the headers so I really don't understand what is the cause of this error.

Answer: There was actually a bug in the backend that was only triggered by some additional headers added by the browser. In that particular case, the server was returning a 404 error which wouldn't contain my header definitions and would cause the CORS policy block. I was only able to identify the bug after I used devtools to track the request sent by the browser and replicated all the headers in my curl request. After fixing the function logic the problem was fixed.
Question: I am using the following lines of codes to visualise the gradients of an ANN model using tensorboard  tensorboard_callback = tf.compat.v1.keras.callbacks.TensorBoard(log_dir='./Graph', histogram_freq=1, write_graph = True, write_grads =True, write_images = False) tensorboard_callback .set_model(model) %tensorboard --logdir ./Graph  I received a warning message saying "WARNING:tensorflow:write_grads will be ignored in TensorFlow 2.0 for the TensorBoard Callback." I get the tensorboard output, but without gradients.  What could be the possible reason? (Note: I use 2.3.0 tensorflow version) Thank you.

Answer: Write_Grads was not implemented in TF2.x. This is one of the highly expected feature request that is still open. Please check this GitHub issue as feature request. So, we only need to import TF1.x modules and use write_grads as shown in the following code. # Load the TensorBoard notebook extension %load_ext tensorboard import tensorflow as tf import datetime # Clear any logs from previous runs !rm -rf ./logs/ # Disable V2 behavior tf.compat.v1.disable_v2_behavior() mnist = tf.keras.datasets.mnist (x_train, y_train),(x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 def create_model(): return tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(512, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10, activation='softmax') ]) model = create_model() model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S") tensorboard_callback = tf.compat.v1.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1, write_grads =True) model.fit(x=x_train, y=y_train, epochs=1, validation_data=(x_test, y_test), callbacks=[tensorboard_callback]) %tensorboard --logdir logs/fit  Output: Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz 11493376/11490434 [==============================] - 0s 0us/step Train on 60000 samples, validate on 10000 samples WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_v1.py:2048: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. 32/60000 [..............................] - ETA: 0s - loss: 2.3311 - acc: 0.0312WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0055s vs `on_train_batch_end` time: 0.0235s). Check your callbacks. 60000/60000 [==============================] - 17s 288us/sample - loss: 0.2187 - acc: 0.9349 - val_loss: 0.1012 - val_acc: 0.9690 <tensorflow.python.keras.callbacks.History at 0x7f7ebd1d3d30>
Question: I'm trying to set a conda environment using a requirements.txt file that a coworker shared with me. My coworker uses Python in a Mac without Anaconda, and I'm using it in a Windows machine with Anaconda. The file requirements.txt was generated with the command pip freeze and looks like this: absl-py==0.7.1 affine==2.3.0 agate==1.6.0 agate-dbf==0.2.0 agate-excel==0.2.1 agate-sql==0.5.2 ...  After checking the answer of this question, I tried the following in the Anaconda terminal: conda create --name my-env-name --file requirements.txt  Which fails with the following error message: PackagesNotFoundError: The following packages are not available from current channels: - appscript==1.0.1 - style==1.1.0 - senticnet==1.3 - scikits.optimization==0.3 ...  My understanding is that this happens because those packages are not available in the Anaconda package installation channels, and that they should be installed instead via pip with my conda environment activated, using pip install -r requirements.txt The problem is that this list of packages is very long, and I would like to avoid having to manually check and separating which packages are included in Anaconda channels and which should be installed via pip. Then, is there a way to tell Anaconda to create an environment by automatically recognizing the packages included in its channels, installing them, and then installing the rest using pip?

Answer: Using requirements.txt with conda There's no problem at all using a requirements.txt file when creating a conda environment. In fact, you can also set additional channels at creation time: conda create --name my-env-name --file requirements.txt --channel <NAME_OF_CHANNEL>  for example, in the case of the first package you mention, you can install it from anaconda channel. So you could run: conda create --name my-env-name --file requirements.txt --channel default --channel anaconda  Why using default channel first? Well, just to give preference to the default one (the priority of channels is specified by the order they are listed: higher priority from left to right). When at least some of the packages are not available using conda Well, when no conda channel can provide any of your required packages, there are several alternatives:  Install through conda those packages available in any of its channels.  Install through pip the rest.  Create a conda environment.yml file:  conda env export > environment.yml    When you need to recreate this environment, then you can do: conda env create --name my-env-name --file environment.yml  and it will install the packages using conda, will install pip, and then will install those packages only available with the latter one. This approach has good and bad properties:  one of the good properties is that it separates those packages installed through conda from those installed using pip. one of the bad properties is that it's only useful for conda, but not for pip alone.
Question: Are there any tricks I can employ to get IDEs to offer code completion for dynamically generated class attributes? For instance  class A: def __init__(self): setattr(self, "a", 5)  This code will set the class attribute of A called a to the value of 5. But IDEs do not know about a and therefore you do not get code completion for it. I've read that the __dir__ method can be hooked, but the suggestion made in that answer has not worked for me. Does anybody have any ideas?

Answer: I believe you will find the answer here. In short, Pycharm currently does not (and probably in the observable future will not) support dynamic attributes. That's because it analyzes code statically, and can't "see" what attributes a class might have. On the other hand, when you run your code in (say) iPython, once an instance of such class is created, the editor can get the attributes of the specific instance that resides in the memory, and thus show them in autocomplete. In such cases __dir__ helps. I assume that would be the case also with other IDEs. So if you really need the autocompletion feature, you may want to try using Jupyter notebook. Still, you will have to instantiate your variable prior to getting autocomplete. Another possible solution is to exploit that IDE supports .pyi files (Python interface file stub). You can add to your code a little snippet that instantiate the class with dynamic attributes, and writes down a class interface file (.pyi). Then IDE will use it for autocompletion. Obviously, this solution will have a "one run delay", but as in PyCharm you can switch to such file just by clicking on the asterisk near the class name, you can manually update it when having massive changes.
Question: To explode list like column to row, we can use pandas explode() function. My pandas' version '0.25.3' The given example worked for me and another answer of Stackoverflow.com works as expected but it doesn't work for my dataset.  city nested_city 0 soto ['Soto'] 1 tera-kora ['Daniel'] 2 jan-thiel ['Jan Thiel'] 3 westpunt ['Westpunt'] 4 nieuwpoort ['Nieuwpoort', 'Santa Barbara Plantation']  What I have tried: test_data['nested_city'].explode()  and test_data.set_index(['nested_city']).apply(pd.Series.explode).reset_index()  Output 0 ['Soto'] 1 ['Daniel'] 2 ['Jan Thiel'] 3 ['Westpunt'] 4 ['Nieuwpoort', 'Santa Barbara Plantation'] Name: neighbors, dtype: object 

Answer: You need to ensure that your column is of list type to be able to use pandas' explode(). Here is a working solution: from ast import literal_eval test_data['nested_city'] = test_data['nested_city'].apply(literal_eval) #convert to list type test_data['nested_city'].explode()  To explode multiple columns at a time, you can do the following: not_list_cols = [col for col in test_data.columns if col not in ['col1', 'col2']] #list of columns you are not exploding (assume col1 and col2 are being exploded) test_data = test_data.set_index(not_list_cols).apply(pd.Series.explode).reset_index()
Question: I have a numpy 2D array, and I would like to select different sized ranges of this array, depending on the column index. Here is the input array a = np.reshape(np.array(range(15)), (5, 3)) example [[ 0 1 2] [ 3 4 5] [ 6 7 8] [ 9 10 11] [12 13 14]]  Then, list b = [4,3,1] determines the different range sizes for each column slice, so that we would get the arrays [0 3 6 9] [1 4 7] [2]  which we can concatenate and flatten to get the final desired output [0 3 6 9 1 4 7 2]  Currently, to perform this task, I am using the following code slices = [] for i in range(a.shape[1]): slices.append(a[:b[i],i]) c = np.concatenate(slices)  and, if possible, I want to convert it to a pythonic format. Bonus: The same question but now considering that b determines row slices instead of columns.

Answer: We can use broadcasting to generate an appropriate mask and then masking does the job - In [150]: a Out[150]: array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11], [12, 13, 14]]) In [151]: b Out[151]: [4, 3, 1] In [152]: mask = np.arange(len(a))[:,None] < b In [153]: a.T[mask.T] Out[153]: array([0, 3, 6, 9, 1, 4, 7, 2])  Another way to mask would be - In [156]: a.T[np.greater.outer(b, np.arange(len(a)))] Out[156]: array([0, 3, 6, 9, 1, 4, 7, 2])  Bonus : Slice per row If we are required to slice per row based on chunk sizes, we would need to modify few things - In [51]: a Out[51]: array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14]]) # slice lengths per row In [52]: b Out[52]: [4, 3, 1] # Usual loop based solution : In [53]: np.concatenate([a[i,:b_i] for i,b_i in enumerate(b)]) Out[53]: array([ 0, 1, 2, 3, 5, 6, 7, 10]) # Vectorized mask based solution : In [54]: a[np.greater.outer(b, np.arange(a.shape[1]))] Out[54]: array([ 0, 1, 2, 3, 5, 6, 7, 10])