eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I downloaded the following code : from __future__ import print_function from time import sleep def callback_a(i, result): print("Items processed: {}. Running result: {}.".format(i, result)) def square(i): return i * i def processor(process, times, report_interval, callback): print("Entered processor(): times = {}, report_interval = {}, callback = {}".format( times, report_interval, callback.func_name)) # Can also use callback.__name__ instead of callback.func_name in line above. result = 0 print("Processing data ...") for i in range(1, times + 1): result += process(i) sleep(1) if i % report_interval == 0: # This is the call to the callback function # that was passed to this function. callback(i, result) processor(square, 20, 5, callback_a)  It works fine under python 2, but I get the following error under python3: Traceback (most recent call last): File "test/python/cb_demo.py", line 33, in <module> processor(square, 20, 5, callback_a) File "test/python/cb_demo.py", line 21, in processor times, report_interval, callback.func_name)) AttributeError: 'function' object has no attribute 'func_name'  I need to work under python3.

Answer: That behaviour in Python 3 is expected as it was changed from Python 2. Per the documentation here: https://docs.python.org/3/whatsnew/3.0.html#operators-and-special-methods  The function attributes named func_X have been renamed to use the __X__ form, freeing up these names in the function attribute namespace for user-defined attributes. To wit, func_closure, func_code, func_defaults, func_dict, func_doc, func_globals, func_name were renamed to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__, __name__, respectively.  You will notice the mention of func_name as one of the attributes that were renamed. You will need to use __name__. Sample code in Python 3: >>> def foo(a): ... print(a.__name__) ... >>> def c(): ... pass ... >>> >>> foo(c) c
Question: I would like to apply fillna only in numeric columns. Is possible? Right now, I'm applying it in all columns:  df = df.replace(r"^\s*$", np.nan, regex=True) 

Answer: You can select numeric columns and then fillna E.g: import pandas as pd df = pd.DataFrame({'a': [1, None] * 3, 'b': [True, None] * 3, 'c': [1.0, None] * 3}) # select numeric columns numeric_columns = df.select_dtypes(include=['number']).columns # fill -1 to all NaN df[numeric_columns] = df[numeric_columns].fillna(-1) # print print(df)
Question: I am trying to find a vectorized approach of finding the first position in an array where the values did not get higher than the maximum of n previous numbers. I thought about using the find_peaks method of scipy.signal to find a local maximum. I think it does exactly that if you define the distance to let's say 10 n is 10. But unfortunately, the condition for the distance has to be fulfilled in both directions - previous and upcoming numbers. Is there any other method or approach to finding such a thing? Example: arr1 = np.array([1. , 0.73381293, 0.75649351, 0.77693474, 0.77884614, 0.81055903, 0.81402439, 0.78798586, 0.78839588, 0.82967961, 0.8448 , 0.83276451, 0.82539684, 0.81762916, 0.82722515, 0.82101804, 0.82871127, 0.82825041, 0.82086957, 0.8347826 , 0.82666665, 0.82352942, 0.81270903, 0.81191224, 0.83180428, 0.84975767, 0.84044236, 0.85057473, 0.8394649 , 0.80000001, 0.83870965, 0.83962262, 0.85039371, 0.83359748, 0.84019768, 0.83281732, 0.83660132]) from scipy.signal import find_peaks peaks, _ = find_peaks(arr1, distance=10)  In this case, it finds positions 10 and 27. But also position 0 has 10 following elements which are not higher. How can I find those?

Answer: def rolling_window(a, window): shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) def get_peaks(arr, window): maxss = np.argmax(rolling_window(arr1, window), axis=1) return np.where(maxss == 0)[0]  >>> arr1 = np.array([1. , 0.73381293, 0.75649351, 0.77693474, 0.77884614, 0.81055903, 0.81402439, 0.78798586, 0.78839588, 0.82967961, 0.8448 , 0.83276451, 0.82539684, 0.81762916, 0.82722515, 0.82101804, 0.82871127, 0.82825041, 0.82086957, 0.8347826 , 0.82666665, 0.82352942, 0.81270903, 0.81191224, 0.83180428, 0.84975767, 0.84044236, 0.85057473, 0.8394649 , 0.80000001, 0.83870965, 0.83962262, 0.85039371, 0.83359748, 0.84019768, 0.83281732, 0.83660132]) >>> get_peaks(arr1, 10) array([ 0, 10, 27])  Credit for rolling window function : Rolling window for 1D arrays in Numpy?
Question: I am trying to implement a docker for a dummy local Django project. I am using docker-compose as a tool for defining and running multiple containers. Here I tried to containerize the Django-web-app and PostgreSQL two services.  Configuration used in Dockerfile and docker-compose.yml  Dockerfile # Pull base image FROM python:3.7-alpine # Set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Set work directory WORKDIR /code # Install dependencies COPY requirements.txt /code/ RUN pip install -r requirements.txt # Copy project COPY . /code/  docker-compose.yml version: '3.7' services: web: build: . command: python manage.py runserver 0.0.0.0:8000 volumes: - .:/code ports: - "8000:8000" depends_on: - db db: image: postgres:11 volumes: - postgres_data:/var/lib/postgresql/data/ volumes: postgres_data:  All seems okay. The path postgres integrations and all except one thing pip install -r requirements.txt. This is taking too much time to install from requirements. Last time I was giving up on this but at last the installation does completed but takes lots of time to complete.  In my scenario, the only issue is why the pip install so slow. If there is anything that I am missing? I am new to docker and any help on this topic will be highly appreciated. Thank you.  I was following this Link.

Answer: Probably this is because PyPI wheels don’t work on Alpine. Instead of using precompile files Alpine downloads the source code and compile it. Try to use python:3.7-slim image instead: # Pull base image FROM python:3.7-slim # Set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Set work directory WORKDIR /code # Install dependencies COPY requirements.txt /code/ RUN pip install -r requirements.txt # Copy project COPY . /code/  Check this article for more details: Alpine makes Python Docker builds 50× slower.
Question: I'm attempting to create a faceted plot using seaborn in python, but I'm having issues with a number of things, one thing being rotating the x-axis labels. I am currently attempting to use the following code: import pandas as pd import seaborn as sns import matplotlib.pyplot as plt vin = pd.Series(["W1","W1","W2","W2","W1","W3","W4"]) word1 = pd.Series(['pdi','pdi','tread','adjust','fill','pdi','fill']) word2 = pd.Series(['perform','perform','fill','measure','tire','check','tire']) date = pd.Series(["01-07-2020","01-07-2020","01-07-2020","01-07-2020","01-08-2020","01-08-2020","01-08-2020"]) bigram_with_dates = pd.concat([vin,word1,word2,date], axis = 1) names = ["vin", "word1","word2","date"] bigram_with_dates.columns = names bigram_with_dates['date'] = pd.to_datetime(bigram_with_dates['date']) bigram_with_dates['text_concat'] = bigram_with_dates['word1'] + "," + bigram_with_dates['word2'] plot_params = sns.FacetGrid(bigram_with_dates, col="date", height=3, aspect=.5, col_wrap = 10,sharex = False, sharey = False) plot = plot_params.map(sns.countplot, 'text_concat', color = 'c', order = bigram_with_dates['text_concat']) plot_adjust = plot.fig.subplots_adjust(wspace=0.5, hspace=0.5) for axes in plot.axes.flat: axes.set_xticklabels(axes.get_xticklabels(), rotation=90)  When I use this I get an error that states: AttributeError: 'NoneType' object has no attribute 'axes'  Which I think I understand to mean that there is no returned object so setting axes on nothing does nothing.  This code seems to work in other SO posts I've come across, but I can't seem to get it to work.  Any suggestions as to what I'm doing wrong would be greatly appreciated. Thanks, Curtis

Answer: Try this, it seems you were over-writing the 'plot' variable.: import pandas as pd import seaborn as sns import matplotlib.pyplot as plt %matplotlib inline vin = pd.Series(["W1","W1","W2","W2","W1","W3","W4"]) word1 = pd.Series(['pdi','pdi','tread','adjust','fill','pdi','fill']) word2 = pd.Series(['perform','perform','fill','measure','tire','check','tire']) date = pd.Series(["01-07-2020","01-07-2020","01-07-2020","01-07-2020","01-08-2020","01-08-2020","01-08-2020"]) bigram_with_dates = pd.concat([vin,word1,word2,date], axis = 1) names = ["vin", "word1","word2","date"] bigram_with_dates.columns = names bigram_with_dates['date'] = pd.to_datetime(bigram_with_dates['date']).dt.strftime('%m-%d-%Y') bigram_with_dates['text_concat'] = bigram_with_dates['word1'] + "," + bigram_with_dates['word2'] plot = sns.FacetGrid(bigram_with_dates, col="date", height=3, aspect=.5, col_wrap = 10,sharex = False, sharey = False) plot1 = plot.map(sns.countplot, 'text_concat', color = 'c', order = bigram_with_dates['text_concat'].value_counts(ascending = False).iloc[:5].index)\ .fig.subplots_adjust(wspace=0.5, hspace=12) for axes in plot.axes.flat: _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=90) plt.tight_layout()  Output:
Question: I follow this instruction https://www.geeksforgeeks.org/python-pandas-dataframe-get_value/ and know how to get the value from a dateframe data in pandas: df.get_value(10, 'Salary')  My question is how to get the value of 'Salary' in the last row?

Answer: First of all I would advise against using get_value since it is/will be deprecated. (see: https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.get_value.html ) There are a couple of solutions: df['Salary'].iloc[-1] df.Salary.iloc[-1]  are synonymous. Iloc is the way to retrieve items in a pandas df by their index. df['Salary'].values[-1]  creates a list of the Salary column and returns the last items df['Salary'].tail(1) df.Salary.tail(1)  returns the last row of the salary column. Which of the solutions is best, depends on the context and your personal preference.
Question: I am importing models in django shell API but i get the below error. Here is how it occurs: python manage.py shell from .models import Device  I get: File "<console>", line 1, in <module> KeyError: "'__name__' not in globals"

Answer: Try putting the app name before ".models". Here .models trying to import from models.py in the current directory but models.py is actually located in the app directory. >> from [app_name].models import Device
Question: My conda version is 4.7.11. I am trying to clone the base env to a new one so I can install some specific packages and will not mess up the base environment. I tried as some other answers suggested: conda create --name <myenv> --clone base  and conda create --name <myenv> --clone root  But none of them works. The message from terminal is "The system cannot find the file specified".  Below is my cuurent env list: base * D:\LabTest\Dave\Anaconda dlc-windowsCPU D:\LabTest\Dave\Anaconda\envs\dlc-windowsCPU dlc-windowsGPU D:\LabTest\Dave\Anaconda\envs\dlc-windowsGPU dlc-windowsGPU-dave D:\LabTest\Dave\Anaconda\envs\dlc-windowsGPU-dave dlc-windowsGPU-yc D:\LabTest\Dave\Anaconda\envs\dlc-windowsGPU-yc  I also cannot clone from my anaconda navigator.  Don't know what to do. 

Answer: I would recommend that you try the method as shown on this official documentation. In summary, you can get all the list of modules installed in the virtual environment, save it as a .txt file, and create a new environment from that .txt file. For example, conda list --explicit > spec-file.txt  Then, create a new environment using that specification. conda create --name myenv --file spec-file.txt  While this is not exactly "cloning" the base environment, you should be able to reproduce a virtual environment identical to the base through this process.
Question: I am currently using Spyder for Python, and I have this error message when I open the program: Error: You have missing dependencies! rtree>= 0.8.3: None (NOK) Please install them to avoid this message. Note: Spyder could work without some of these dependencies, however to have a smooth experience, we strongly recommend. I tried pip install rtree and got: Collecting rtree Downloading https://files.pythonhosted.org/packages/11/1d/42d6904a436076df813d1df632575529991005b33aa82f169f01750e39e4/Rtree-0.9.3.tar.gz (520kB) |████████████████████████████████| 522kB 467kB/s ERROR: Command errored out with exit status 1: command: 'C:\Users\gitte\Anaconda3\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\gitte\\AppData\\Local\\Temp\\pip-install-kmbt5h2t\\rtree\\setup.py'"'"'; __file__='"'"'C:\\Users\\gitte\\AppData\\Local\\Temp\\pip-install-kmbt5h2t\\rtree\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info cwd: C:\Users\gitte\AppData\Local\Temp\pip-install-kmbt5h2t\rtree\ Complete output (11 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\gitte\AppData\Local\Temp\pip-install-kmbt5h2t\rtree\setup.py", line 3, in <module> import rtree File "C:\Users\gitte\AppData\Local\Temp\pip-install-kmbt5h2t\rtree\rtree\__init__.py", line 1, in <module> from .index import Rtree File "C:\Users\gitte\AppData\Local\Temp\pip-install-kmbt5h2t\rtree\rtree\index.py", line 6, in <module> from . import core File "C:\Users\gitte\AppData\Local\Temp\pip-install-kmbt5h2t\rtree\rtree\core.py", line 128, in <module> raise OSError("could not find or load %s" % lib_name) OSError: could not find or load spatialindex_c-64.dll ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.  Please advise what I can do. Spyder works great so far, I just don't want to have issues along the way. Thanks!

Answer: It looks like Rtree requires libspatialindex (https://libspatialindex.org) which is not automatically installed. It seems some devs are aware of the problem and working on a fix: https://github.com/Toblerity/rtree/issues/146 https://github.com/Toblerity/rtree/issues/147
Question: When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training? Example training code using .eval().

Answer: model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn them off during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() in pair with model.eval() to turn off gradients computation: # evaluate model: model.eval() with torch.no_grad(): ... out_data = model(data) ...  BUT, don't forget to turn back to training mode after eval step: # training step ... model.train() ...
Question: When I set the learning rate and find the accuracy cannot increase after training few epochs optimizer = optim.Adam(model.parameters(), lr = 1e-4) n_epochs = 10 for i in range(n_epochs): // some training here  If I want to use a step decay: reduce the learning rate by a factor of 10 every 5 epochs, how can I do so?

Answer: You can use learning rate scheduler torch.optim.lr_scheduler.StepLR import torch.optim.lr_scheduler.StepLR scheduler = StepLR(optimizer, step_size=5, gamma=0.1)  Decays the learning rate of each parameter group by gamma every step_size epochs see docs here Example from docs # Assuming optimizer uses lr = 0.05 for all groups # lr = 0.05 if epoch < 30 # lr = 0.005 if 30 <= epoch < 60 # lr = 0.0005 if 60 <= epoch < 90 # ... scheduler = StepLR(optimizer, step_size=30, gamma=0.1) for epoch in range(100): train(...) validate(...) scheduler.step()  Example: import torch import torch.optim as optim optimizer = optim.SGD([torch.rand((2,2), requires_grad=True)], lr=0.1) scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)  for epoch in range(1, 21): scheduler.step() print('Epoch-{0} lr: {1}'.format(epoch, optimizer.param_groups[0]['lr'])) if epoch % 5 == 0:print()  Epoch-1 lr: 0.1 Epoch-2 lr: 0.1 Epoch-3 lr: 0.1 Epoch-4 lr: 0.1 Epoch-5 lr: 0.1 Epoch-6 lr: 0.010000000000000002 Epoch-7 lr: 0.010000000000000002 Epoch-8 lr: 0.010000000000000002 Epoch-9 lr: 0.010000000000000002 Epoch-10 lr: 0.010000000000000002 Epoch-11 lr: 0.0010000000000000002 Epoch-12 lr: 0.0010000000000000002 Epoch-13 lr: 0.0010000000000000002 Epoch-14 lr: 0.0010000000000000002 Epoch-15 lr: 0.0010000000000000002 Epoch-16 lr: 0.00010000000000000003 Epoch-17 lr: 0.00010000000000000003 Epoch-18 lr: 0.00010000000000000003 Epoch-19 lr: 0.00010000000000000003 Epoch-20 lr: 0.00010000000000000003  More on How to adjust Learning Rate - torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.
Question: Background and Existing Solutions I am porting some Python code into Julia (v1.3.1), and I have run into an issue with trying to reproduce the code into as easily readable code in Julia. In Python (using numpy), we have created a 101-element logarithmically spaced sequence from 0.001 to 1000: >>> X = numpy.logspace( -3, 3, 101 ) array([1.00000000e-03, 1.14815362e-03, 1.31825674e-03, ..., 1.00000000e+03])  Implementing this in Julia with PyCall would of course work like this: julia> using PyCall julia> numpy = pyimport("numpy") julia> X_python = numpy.logspace( -3, 3, 101 ) 101-element Array{Float64,1}: 0.001 0.0011481536214968829 0.0013182567385564075 ⋮ 1000.0  But I want to implement this in pure Julia for my current project. Not finding the same function from the Julia documentation, after some searching I came across an older documentation entry for logspace here. I then came across this Github pull request for deprecating logspace into its definition, so currently it seems that this is the way to create the logarithmically spaced sequence: julia> X_julia = 10 .^ range( -3, 3, length = 101 ) 101-element Array{Float64,1}: 0.001 0.0011481536214968829 0.0013182567385564075 ⋮ 1000.0   TL;DR / The Actual Question julia> LinRange(1e-3, 1e3, 101) 101-element LinRange{Float64}: 0.001,10.001,20.001,…,1000.0  Since there currently exists a simple and easy-to-read function, LinRange, for creating linear sequences (as seen above), does there exist a similar function, something like LogRange, for logarithmic sequences? I am going for simplicity and improved readability in this project, so while broadcasting a range into the exponent of 10 works from the mathematical point of view, something like LogRange(1e-3, 1e3, 101) would be easier for a beginner or part-time programmer to understand.   EDIT: When the limits of the sequence are integer exponents of 10, the code is fairly clear, but when the limits are floats, the difference in readability between LogRange() and 10 .^ () becomes more apparent: julia> 10 .^ range( log10(1.56e-2), log10(3.62e4), length = 101 ) julia> LogRange( 1.56e-2, 3.62e4, 101 ) 

Answer: You can just use range For Log-spaced values ∈ [1.0e-10, 1.0e10]: 10 .^ range(-10, stop=10, length=101)
Question: i wanted to get some proxy list from this webPage; https://free-proxy-list.net/ but i stuck in this error and dont know how to fix it. requests.exceptions.ProxyError: HTTPSConnectionPool(host='free-proxy-list.net', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000278BFFA1EB0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')))  and btw, this is my related code: import urllib import requests from bs4 import BeautifulSoup from fake_useragent import UserAgent ua = UserAgent(cache=False) header = { "User-Agent": str(ua.msie) } proxy = { "https": "http://95.66.151.101:8080" } urls = "https://free-proxy-list.net/" res = requests.get(urls, proxies=proxy) soup = BeautifulSoup(res.text,'lxml')  and i tried to scrape other web sites, but i realized that its not the way.

Answer: Your using https in the Json dict when your proxy is a http proxy Proxies should always be inside this format For a http proxy {'"http": "Http Proxy"}  For a https proxy {"https":"Https Proxy"}  And for the UserAgent {"User-Agent": "Opera/9.80 (X11; Linux x86_64; U; de) Presto/2.2.15 Version/10.00"}  Example import requests requests.get("https://example.com", proxies={"http":"http://95.66.151.101:8080"}, headers={"User-Agent": "Opera/9.80 (X11; Linux x86_64; U; de) Presto/2.2.15 Version/10.00"})  The module from fake_useragent import UserAgent You imported is irrelevant and unnecessary Extra The error could've also happened because the proxy isn't valid or responded improperly If you are looking for free lists of proxies consider checking out these sources https://pastebin.com/raw/VJwVkqRT https://proxyscrape.com/free-proxy-list https://www.freeproxylists.net/
Question: I get following error while trying to convert object (string) column in Pandas to Int32 which is integer type that allows for NA values. df.column = df.column.astype('Int32')   TypeError: object cannot be converted to an IntegerDtype  I'm using pandas version: 0.25.3

Answer: It's known bug, as explained here. Workaround is to convert column first to float and than to Int32. Make sure you strip your column from whitespaces before you do conversion: df.column = df.column.str.strip()  Than do conversion: df.column = df.column.astype('float') # first convert to float before int df.column = df.column.astype('Int32')  or simpler:  df.column = df.column.astype('float').astype('Int32') # or Int64
Question: If I have a key in a dictionary with an invalid identifier, such as A(2). How can I create a TypedDict with this field? E.g from typing import TypedDict class RandomAlphabet(TypedDict): A(2): str  is not valid Python code, resulting in the error: SyntaxError: illegal target for annotation  The same problem is with reserved keywords: class RandomAlphabet(TypedDict): return: str  throws: SyntaxError: invalid syntax 

Answer: According to PEP 589 you can use alternative syntax to create a TypedDict as follows:  Movie = TypedDict('Movie', {'name': str, 'year': int})  So, in your case, you could write: from typing import TypedDict RandomAlphabet = TypedDict('RandomAlphabet', {'A(2)': str})  or for the second example: RandomAlphabet = TypedDict('RandomAlphabet', {'return': str})   PEP 589 warns, though:  This syntax doesn't support inheritance, however, and there is no way to have both required and non-required fields in a single type. The motivation for this is keeping the backwards compatible syntax as simple as possible while covering the most common use cases.