eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have two files, file1.py which have ML model size of 1GB and file2.py which calls get_vec() method from file1 and receives vectors in return. ML model is being loaded everytime when file1 get_vec() method is called. This is where it is taking lots of time (around 10s) to load the model from disk. I want to tell file1 somehow not to reload model every time but utilize loaded model from earlier calls. Sample code is as follows # File1.py import spacy nlp = spacy.load('model') def get_vec(post): doc = nlp(post) return doc.vector  File2.py from File1 import get_vec df['vec'] = df['text'].apply(lambda x: get_vec(x))  So here, it is taking 10 to 12 seconds in each call. This seems small code but it is a part of a large project and I can not put both in the same file.  Update1: I have done some research and came to know that I can use Redis to store model in cache first time it runs and thereafter I can read the model from cache directly. I tried it for testing with Redis as follows import spacy import redis nlp = spacy.load('en_core_web_lg') r = redis.Redis(host = 'localhost', port = 6379, db = 0) r.set('nlp', nlp)  It throws an error DataError: Invalid input of type: 'English'. Convert to a bytes, string, int or float first.  Seems, type(nlp) is English() and it need to convert in a suitable format. So I tried to use pickle as well to convert it. But again, pickle is taking lots of time in encoding and decoding. Is there anyway to store this in Redis? Can anybody suggest me how can I make it faster? Thanks.

Answer: Heres how to do it Step 1) create a function in python and load your model in that function model=None def load_model(): global model model = ResNet50(weights="imagenet")  if you carefully observe first I assigned variable model to None. Then inside load_model function I loaded a model.  Also I made sure the variable model is made global so that it can be accessed from outside this function. The intuition here is we load model object in a global variable. So that we can access this variable anywhere within the code.  Now that we have our tools ready (i.e we can access the model from anywhere within this code ) lets freeze this model in your computers RAM. This is done by: if __name__ == "__main__": print(("* Loading Keras model and Flask starting server..." "please wait until server has fully started")) load_model() app.run()  Now what's the use of freezing model in RAM without using it. So, to use it I use POST request in flask @app.route("/predict", methods=["POST"]) def predict(): if flask.request.method == "POST": output=model.predict(data) #what you want to do with frozen model goes here  So using this trick you can freeze model in RAM, access it using a global variable. and then use it in your code.
Question: I have a list of fractions that I need to transform. from fractions import Fraction fractions_list=[Fraction(3,14),Fraction(1,7),Fraction(9,14)]  The output should be a list with the numerators for each fraction, followed by the least common denominator for all of them. For above example the result (3/14, 2/14, 9/14) would be represented as follows [3,2,9,14]  Is there an elegant solution for this? All I can think of involves a lot of intermediate lists to store some variables, and scales horribly.

Answer: import numpy as np fractions_list=[Fraction(3,14),Fraction(1,7),Fraction(9,14)] lcm = np.lcm.reduce([fr.denominator for fr in fractions_list]) vals = [int(fr.numerator * lcm / fr.denominator) for fr in fractions_list] vals.append(lcm)
Question: lastly I'm working on google colab I get this dataset colled celeba and it is into a google drive accout and this account is not mine but I have the access to go through it now because the internet problems and drive capacity I can not dounload the dataset then upload it to my drive ... so the question is: is there any way to let google colab get access to this dataset or such a way to import the path... I have this function definition below  create_celebahq_cond_continuous('/content/drive/My Drive/kiki96/results/tfrecords','https://drive.google.com/open?id=0B7EVK8r0v71pWEZsZE9oNnFzTm8','https://drive.google.com/open?id=0B4qLcYyJmiz0TXY1NG02bzZVRGs',4,100,False)  where I have tried to put the sharablelink of the dataset but, it does not work  please help 

Answer: To download a file to Colab If you want to download the file directly into your Google Colab instance, then you can use gdown.  Note that the file must be shared to the public. If the link to your dataset is https://drive.google.com/file/d/10vAwF6hFUjvw3pf6MmB_S0jZm9CLWbSx/view?usp=sharing, you can use: !gdown --id "10vAwF6hFUjvw3pf6MmB_S0jZm9CLWbSx"   To download the file to your Drive Instead, if you want to download it to your drive then Mount your Google Drive from google.colab import drive drive.mount('/content/drive')  Change the directory to a folder in your Google Drive cd '/content/drive/My Drive/datasets/'  Download the file into your Google Drive folder !gdown --id "10vAwF6hFUjvw3pf6MmB_S0jZm9CLWbSx"   To download a folder If you are trying to download a folder, follow these steps:  Open the shared folder    Click "Add shortcut to my drive" and select a folder    Mount your Google Drive to Google Colab Go to the folder where you added the shortcut   You can see the newly-added folder, being referenced by its Google Drive folder ID.
Question: I'm trying to plot percent change data and would like to plot it such that the y axis is symmetric about 0. i.e. 0 is in the center of the axis. import matplotlib.pyplot as plt import pandas as pd data = pd.DataFrame([1,2,3,4,3,6,7,8], columns=['Data']) data['PctChange'] = data['Data'].pct_change() data['PctChange'].plot()   This is different from How to draw axis in the middle of the figure?. The goal here is not to move the x axis, but rather, change the limits of the y axis such that the zero is in the center. Specifically in a programmatic way that changes in relation to the data.

Answer: After plotting the data find the maximum absolute value between the min and max axis values. Then set the min and max limits of the axis to the negative and positive (respectively) of that value. import matplotlib.pyplot as plt import pandas as pd data = pd.DataFrame([1,2,3,4,3,6,7,8], columns=['Data']) data['PctChange'] = data['Data'].pct_change() ax = data['PctChange'].plot() yabs_max = abs(max(ax.get_ylim(), key=abs)) ax.set_ylim(ymin=-yabs_max, ymax=yabs_max)
Question:  def fatorial(n): if n <= 1: return 1 else: return n*fatorial(n - 1) import pytest @pytest.mark.parametrize("entrada","esperado",[ (0,1), (1,1), (2,2), (3,6), (4,24), (5,120) ]) def testa_fatorial(entrada,esperado): assert fatorial(entrada) == esperado  The error:  ERROR collecting Fatorial_pytest.py ____________________________________________________________________ In testa_fatorial: indirect fixture '(0, 1)' doesn't exist  I dont know why I got "indirect fixture”. Any idea? I am using python 3.7 and windows 10 64 bits.

Answer: TL;DR - The problem is with the line  @pytest.mark.parametrize("entrada","esperado",[ ... ])  It should be written as a comma-separated string:  @pytest.mark.parametrize("entrada, esperado",[ ... ])   You got the indirect fixture because pytest couldn't unpack the given argvalues since it got a wrong argnames parameter. You need to make sure all parameters are written as one string.  Please see the documentation:   The builtin pytest.mark.parametrize decorator enables parametrization of arguments for a test function. Parameters: 1. argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings. 2. argvalues – The list of argvalues determines how often a test is invoked with different argument values.  Meaning, you should write the arguments you want to parametrize as a single string and separate them using a comma. Therefore, your test should look like this:  @pytest.mark.parametrize("n, expected", [ (0, 1), (1, 1), (2, 2), (3, 6), (4, 24), (5, 120) ]) def test_factorial(n, expected): assert factorial(n) == expected
Question: I have a dataframe which I am writing to excel using xlsxwriter and I want there to be autofilter applied to all columns where the header is not blank in my spreadsheet without having to specify a range (e.g. A1:D1). Is there any way to do this?

Answer: You will need to specify the range in some way but you can do it programatically based on the shape() of the data frame. For example: import xlsxwriter import pandas as pd df = pd.DataFrame({'A' : [1, 2, 3, 4, 5, 6, 7, 8], 'B' : [1, 2, 3, 4, 5, 6, 7, 8], 'C' : [1, 2, 3, 4, 5, 6, 7, 8], 'D' : [1, 2, 3, 4, 5, 6, 7, 8]}) writer = pd.ExcelWriter('test.xlsx', engine = 'xlsxwriter') # Convert the dataframe to an XlsxWriter Excel object. df.to_excel(writer, sheet_name='Sheet1') # Get the xlsxwriter objects from the dataframe writer object. workbook = writer.book worksheet = writer.sheets['Sheet1'] # Apply the autofilter based on the dimensions of the dataframe. worksheet.autofilter(0, 0, df.shape[0], df.shape[1]) workbook.close() writer.save()  Output:
Question: import logging import pandas as pd logger = logging.getLogger('train') logger.setLevel(logging.DEBUG) # Data data = {'Name': ['Tom', 'nick', 'krish', 'jack'], 'Age': [20, 21, 19, 18]} # Create DataFrame df = pd.DataFrame(data) logger.info(type(df)) logger.info(df.info()) . . . <other_processes> .  The above code outputs: <class 'pandas.core.frame.DataFrame'> None . . .  And at the end of the logs (after all other processes), it also outputs: <class 'pandas.core.frame.DataFrame'> RangeIndex: 4 entries, 0 to 3 Data columns (total 2 columns): Name 4 non-null object Age 4 non-null int64 dtypes: int64(1), object(1) memory usage: 144.0+ bytes  Why does it print None when I try to log df.info()? How can I get df.info() at the intended location in my logs?

Answer: Change buffer parameter in DataFrame.info to StringIO for text with .getvalue(): from io import StringIO buf = StringIO() df.info(buf=buf) logger.info(type(df)) logger.info(buf.getvalue())
Question: When I try to install Anaconda on Linux, I get to this point:  Anaconda3 will now be installed into this location: /home/jorge/anaconda3 - Press ENTER to confirm the location - Press CTRL-C to abort the installation - Or specify a different location below [/home/jorge/anaconda3] >>> PREFIX=/home/jorge/anaconda3 Unpacking payload ...  Then I receive the following error message: concurrent.futures.process._RemoteTraceback: ''' Traceback (most recent call last): File "concurrent/futures/process.py", line 367, in _queue_management_worker File "multiprocessing/connection.py", line 251, in recv TypeError: __init__() missing 1 required positional argument: 'msg' ''' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "entry_point.py", line 69, in <module> File "concurrent/futures/process.py", line 483, in _chain_from_iterable_of_lists File "concurrent/futures/_base.py", line 598, in result_iterator File "concurrent/futures/_base.py", line 435, in result File "concurrent/futures/_base.py", line 384, in __get_result concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. [1770] Failed to execute script entry_point  What can I do? I was following all the instructions step by step

Answer: Did you verify the integrity of the installer's data? because it is a common error when downloading this corrupted or incomplete since it is the previous step you have to do to make sure that the file is ok before executing the script. This post helped me a lot for the first time I installed it. https://www.digitalocean.com/community/tutorials/how-to-install-anaconda-on-ubuntu-18-04-quickstart
Question: I am using plot_confusion_matrix from sklearn.metrics. I want to represent those confusion matrices next to each other like subplots, how could I do this?

Answer: Let's use the good'ol iris dataset to reproduce this, and fit several classifiers to plot their respective confusion matrices with plot_confusion_matrix: from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier from sklearn.svm import SVC from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from matplotlib import pyplot as plt from sklearn.datasets import load_iris from sklearn.metrics import plot_confusion_matrix data = load_iris() X = data.data y = data.target  Set up - X_train, X_test, y_train, y_test = train_test_split(X, y) classifiers = [LogisticRegression(solver='lbfgs'), AdaBoostClassifier(), GradientBoostingClassifier(), SVC()] for cls in classifiers: cls.fit(X_train, y_train)  So the way you could compare all matrices at simple sight, is by creating a set of subplots with plt.subplots. Then iterate both over the axes objects and the trained classifiers (plot_confusion_matrix expects the as input) and plot the individual confusion matrices: fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15,10)) for cls, ax in zip(classifiers, axes.flatten()): plot_confusion_matrix(cls, X_test, y_test, ax=ax, cmap='Blues', display_labels=data.target_names) ax.title.set_text(type(cls).__name__) plt.tight_layout() plt.show()
Question: Is there a listing or library that has all punctuations that we might commonly come across? Normally I use string.punctuation, but some punctuation characters are not included in it, for example: >>> "'" in string.punctuation True >>> "’" in string.punctuation False 

Answer: You might do better with this check: >>> import unicodedata >>> unicodedata.category("'").startswith("P") True >>> unicodedata.category("’").startswith("P") True  The Unicode categories P* are specifically for Punctuation: connector (Pc), dash (Pd), initial quote (Pi), final quote (Pf), open (Ps), close (Pe), other (Po) To prepare the exhaustive collection, which you can subsequently use for fast membership checks, use a set comprehension: >>> import sys >>> from unicodedata import category >>> codepoints = range(sys.maxunicode + 1) >>> punctuation = {c for i in codepoints if category(c := chr(i)).startswith("P")} >>> "'" in punctuation True >>> "’" in punctuation True  Assignment expression here requires Python 3.8+, equivalent for older Python versions: chrs = (chr(i) for i in range(sys.maxunicode + 1)) punctuation = set(c for c in chrs if category(c).startswith("P"))  Beware that some of the other characters in string.punctuation are actually in Unicode category Symbol. It's easy to add those in also if you want.
Question: How would I manage to perform math.ceil such that a number is assigned to the next highest power of 10? # 0.04 -> 0.1 # 0.7 -> 1 # 1.1 -> 10 # 90 -> 100 # ...  My current solution is a dictionary that checks the range of the input number, but it's hardcoded and I would prefer a one-liner solution. Maybe I am missing a simple mathematical trick or a corresponding numpy function here?

Answer: You can use math.ceil with math.log10 to do this: >>> 10 ** math.ceil(math.log10(0.04)) 0.1 >>> 10 ** math.ceil(math.log10(0.7)) 1 >>> 10 ** math.ceil(math.log10(1.1)) 10 >>> 10 ** math.ceil(math.log10(90)) 100  log10(n) gives you the solution x that satisfies 10 ** x == n, so if you round up x it gives you the exponent for the next highest power of 10. Note that for a value n where x is already an integer, the "next highest power of 10" will be n: >>> 10 ** math.ceil(math.log10(0.1)) 0.1 >>> 10 ** math.ceil(math.log10(1)) 1 >>> 10 ** math.ceil(math.log10(10)) 10
Question: I'm trying to build a population pyramid from a pandas df using seaborn. The problem is that some data isn't displayed. As you can see from the plot I created there's some missing data. The Y-axis ticks are 21 and the df's age classes are 21 so why don't they match? What am I missing?  Here's the code I wrote:  import pandas as pd import matplotlib.pyplot as plt import numpy as np import seaborn as sns df = pd.DataFrame({'Age': ['0-4','5-9','10-14','15-19','20-24','25-29','30-34','35-39','40-44','45-49','50-54','55-59','60-64','65-69','70-74','75-79','80-84','85-89','90-94','95-99','100+'], 'Male': [-49228000, -61283000, -64391000, -52437000, -42955000, -44667000, -31570000, -23887000, -22390000, -20971000, -17685000, -15450000, -13932000, -11020000, -7611000, -4653000, -1952000, -625000, -116000, -14000, -1000], 'Female': [52367000, 64959000, 67161000, 55388000, 45448000, 47129000, 33436000, 26710000, 25627000, 23612000, 20075000, 16368000, 14220000, 10125000, 5984000, 3131000, 1151000, 312000, 49000, 4000, 0]}) AgeClass = ['100+','95-99','90-94','85-89','80-84','75-79','70-74','65-69','60-64','55-59','50-54','45-49','40-44','35-39','30-34','25-29','20-24','15-19','10-14','5-9','0-4'] bar_plot = sns.barplot(x='Male', y='Age', data=df, order=AgeClass) bar_plot = sns.barplot(x='Female', y='Age', data=df, order=AgeClass) bar_plot.set(xlabel="Population (hundreds of millions)", ylabel="Age-Group", title = "Population Pyramid") 

Answer: As explained by JohanC, the data is not missing, it's just very small compared to the other bars. Another factor is that you seem to have a white border around each of your bars, which hides the very small bars at the top. Try putting lw=0 in your call to barplot. This is what I am getting: bar_plot = sns.barplot(x='Male', y='Age', data=df, order=AgeClass, lw=0) bar_plot = sns.barplot(x='Female', y='Age', data=df, order=AgeClass, lw=0)
Question: when I run this code: #!/usr/bin/env python3 from datetime import datetime, timedelta from dateutil import tz from pytz import timezone time = "2020-01-15 10:14:00" time = datetime.strptime(time, "%Y-%m-%d %H:%M:%S") print("time1 = " + str(time)) time = time.replace(tzinfo=timezone('America/New_York')) print("time2 = " + str(time)) time = time.astimezone(tz.gettz('UTC')) # explicity convert to UTC time print("time3 = " + str(time)) time = datetime.strftime(time, "%Y-%m-%d %H:%M:%S") # output format print("done time4 = " + str(time))  I get this output: time1 = 2020-01-15 10:14:00 time2 = 2020-01-15 10:14:00-04:56 time3 = 2020-01-15 15:10:00+00:00 done time4 = 2020-01-15 15:10:00  I would have expected the final time to be "2020-01-15 15:14:00" anyone have any ideas why it's off by 4 mintutes? I don't understand why the offset in time2 would by "-04:56" instead of "-05:00"

Answer: From pytz documentation:  This library differs from the documented Python API for tzinfo implementations; if you want to create local wallclock times you need to use the localize() method documented in this document. In addition, if you perform date arithmetic on local times that cross DST boundaries, the result may be in an incorrect timezone (ie. subtract 1 minute from 2002-10-27 1:00 EST and you get 2002-10-27 0:59 EST instead of the correct 2002-10-27 1:59 EDT).   So, you are using pytz incorrectly. Following is both correct, and erroneous code Following code shows results of your use of pytz (datetime.replace(tzinfo=pytz.timezone)), and the recommended way of using pytz with datetime (pytz.timezone.localize(datetime)). from datetime import datetime, date, time, timezone from dateutil import tz import pytz d = date(2019, 1, 27) t = time(19, 32, 00) t1 = datetime.combine(d, t) t1_epoch = t1.timestamp() print("t1_epoch " + str(t1_epoch)) print("t1 " + str(t1)) # your approach/code nytz = pytz.timezone('America/New_York') t3 = t1.replace(tzinfo=nytz) t3_epoch = t3.timestamp() print("t3_epoch " + str(t3_epoch)) print("t3 " + str(t3)) # recommended approach/code using localize nytz = pytz.timezone('America/New_York') t6 = nytz.localize(t1) t6_epoch = t6.timestamp() print("t6_epoch " + str(t6_epoch)) print("t6 " + str(t6))  Output of above code: t1_epoch 1548617520.0 t1 2019-01-27 19:32:00 t3_epoch 1548635280.0 t3 2019-01-27 19:32:00-04:56 t6_epoch 1548635520.0 t6 2019-01-27 19:32:00-05:00  t3 is what you are doing, and it is giving incorrect offset (-4:56). Note that POSIX time is also incorrect in this case. POSIX time, by definition, does not change with timezone. t6 has been created using pytz.timezone.localize() method, and gives correct UTC offset (-5:00). Update: Updated language of the answer as one user found the answer confusing.
Question: i have list with lists of strings looks like allyears #[['1916'], ['1919'], ['1922'], ['1912'], ['1924'], ['1920']]  i need to have output like this: #[1916, 1919, 1922, 1912, 1924, 1920]  have been try this: for i in range(0, len(allyears)): allyears[i] = int(allyears[i])  but i have error >>> TypeError: int() argument must be a string, a bytes-like object or a number, not 'list' 

Answer: You can simply do this: allyears = [int(i[0]) for i in allyears]  Because all the elements in your allyears is a list which has ony one element, so I get it by i[0] The error is because ypu can't convert a list to an int
Question: Can I define functions other than fixtures in conftest.py. If I have a function add(), defined in conftest.py. Can I using the function inside test_add.py file just calling add()?

Answer: Two possible ways  You could just import function add from conftest.py, or move it to something more appropriate. (for example utils.py) You could create fixture that returns function and use it in your tests. Something like this.  conftest.py import pytest @pytest.fixture def add(): def inner_add(x, y): return x + y return inner_add  test_all.py def test_all(add): assert add(1, 2) == 3  I can not imagine situation when it will be a good solution. But I am sure it exists.
Question: When I run: pip3 install django==3.0.5  I get the error ERROR: Could not find a version that satisfies the requirement django==3.0.5 (from versions: 1.1.3, ... 2.2.11, 2.2.12) ERROR: No matching distribution found for django==3.0.4  I need to update some references somewhere, but I am not sure how. Plz Help. 

Answer: As noted in the comments, Django 3.x is only available for Python 3.6 or greater. If you attempt to install Django 3 while using an older version of Python (e.g. Python 3.5 in the case of the OP), pip will be unable to find a matching package. The solution is to simply upgrade to a more modern version of Python.
Question: I am trying to make a build script like this: import glob import os import subprocess import re import argparse import shutil def create_parser(): parser = argparse.ArgumentParser(description='Build project') parser.add_argument('--clean_logs', type=bool, default=True, help='If true, old debug logs will be deleted.') parser.add_argument('--run', type=bool, default=True, help="If true, executable will run after compilation.") parser.add_argument('--clean_build', type=bool, default=False, help="If true, all generated files will be deleted and the" " directory will be reset to a pristine condition.") return parser.parse_args() def main(): parser = create_parser() print(parser)  However no matter how I try to pass the argument I only get the default values. I always get Namespace(clean_build=False, clean_logs=True, run=True). I have tried: python3 build.py --run False python3 build.py --run=FALSE python3 build.py --run FALSE python3 build.py --run=False python3 build.py --run false python3 build.py --run 'False'  It's always the same thing. What am I missing?

Answer: You are misunderstanding how the argparse understands the boolean arguments. Basically you should use action='store_true' or action='store_false' instead of the default value, with the understanding that not specifying the argument will give you the opposite of the action, e.g. parser.add_argument('-x', type=bool, action='store_true')  will cause: python3 command -x  to have x set to True and python3 command  to have x set to False. While action=store_false will do the opposite.  Setting bool as type does not behave as you expect and this is a known issue. The reason for the current behavior is that type is expected to be a callable which is used as argument = type(argument). bool('False') evaluates to True, so you need to set a different type for the behavior you expect to happen.