eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Using a list's insert function is much slower than achieving the same effect using slice assignment: > python -m timeit -n 100000 -s "a=[]" "a.insert(0,0)" 100000 loops, best of 5: 19.2 usec per loop > python -m timeit -n 100000 -s "a=[]" "a[0:0]=[0]" 100000 loops, best of 5: 6.78 usec per loop  (Note that a=[] is only the setup, so a starts empty but then grows to 100,000 elements.) At first I thought maybe it's the attribute lookup or function call overhead or so, but inserting near the end shows that that's negligible: > python -m timeit -n 100000 -s "a=[]" "a.insert(-1,0)" 100000 loops, best of 5: 79.1 nsec per loop  Why is the presumably simpler dedicated "insert single element" function so much slower? I can also reproduce it at repl.it: from timeit import repeat for _ in range(3): for stmt in 'a.insert(0,0)', 'a[0:0]=[0]', 'a.insert(-1,0)': t = min(repeat(stmt, 'a=[]', number=10**5)) print('%.6f' % t, stmt) print() # Example output: # # 4.803514 a.insert(0,0) # 1.807832 a[0:0]=[0] # 0.012533 a.insert(-1,0) # # 4.967313 a.insert(0,0) # 1.821665 a[0:0]=[0] # 0.012738 a.insert(-1,0) # # 5.694100 a.insert(0,0) # 1.899940 a[0:0]=[0] # 0.012664 a.insert(-1,0)  I use Python 3.8.1 32-bit on Windows 10 64-bit. repl.it uses Python 3.8.1 64-bit on Linux 64-bit.

Answer: I think it's probably just that they forgot to use memmove in list.insert. If you take a look at the code list.insert uses to shift elements, you can see it's just a manual loop: for (i = n; --i >= where; ) items[i+1] = items[i];  while list.__setitem__ on the slice assignment path uses memmove: memmove(&item[ihigh+d], &item[ihigh], (k - ihigh)*sizeof(PyObject *));  memmove typically has a lot of optimization put into it, such as taking advantage of SSE/AVX instructions.
Question: For some reason train_test_split, despite lengths being identical and indexes look the same, triggers this error. from sklearn.model_selection import KFold data = {'col1':[30.5,45,1,99,6,5,4,2,5,7,7,3], 'col2':[99.5, 98, 95, 90,1,5,6,7,4,4,3,3],'col3':[23, 23.6, 3, 90,1,9,60,9,7,2,2,1]} df = pd.DataFrame(data) train, test = train_test_split(df, test_size=0.10) X = train[['col1', 'col2']] y2 = train['col3'] X = np.array(X) kf = KFold(n_splits=3, shuffle=True) for train_index, test_index in kf.split(X): X_train, y_train = X[train_index], y[train_index]  y is a pandas Series (same length as x). x was a dataframe with about 20 numerical columns casted to numpy array. For some reason train_test_split triggers the error despite the lengths being identical. If i dont call train_test_split it works fine. the last line triggering the error due to trying to index numpy array this way: y[train_ind]

Answer: I've tried to create a scenario for your situation. I've created following dataframe:  col1 col2 col3 0 1 2 1 1 3 4 0 2 5 6 1 3 7 8 0 4 9 10 1 5 11 12 0 6 13 14 1 7 15 16 0 8 17 18 1 9 19 20 0 10 21 22 1 11 23 24 0 12 25 26 1 13 27 28 0 14 29 30 1  I set col1 and col2 for X and col3 for y. After this I've converted X to numpy array as following. Only difference is I've used shuffle in KFold. X = df[['col1', 'col2']] y = df['col3'] X = np.array(X) kf = KFold(n_splits=3, shuffle=True) for train_index, test_index in kf.split(X): X_train, y_train = X[train_index], y[train_index]  And it worked well. So please check my code and your code and clarify it if there is something I missed. Update I assume y2 is y. So y type is still Series, you need to use .iloc for it. Following code worked well. data = {'col1':[30.5,45,1,99,6,5,4,2,5,7,7,3], 'col2':[99.5, 98, 95, 90,1,5,6,7,4,4,3,3],'col3':[23, 23.6, 3, 90,1,9,60,9,7,2,2,1]} df = pd.DataFrame(data) train, test = train_test_split(df, test_size=0.10) X = train[['col1', 'col2']] y = train['col3'] X = np.array(X) kf = KFold(n_splits=3, shuffle=True) for train_index, test_index in kf.split(X): X_train, y_train = X[train_index], y.iloc[train_index]
Question: I just updated my project to Python 3.7 and I'm seeing this error when I run mypy on the project: error: "Type[datetime]" has no attribute "fromisoformat" datetime does have a function fromisoformat in Python 3.7, but not in previous versions of Python. Why is mypy reporting this error, and how can I get it to analyze Python 3.7 correctly? Things I've tried so far:  Deleting .mypy_cache (which has a suspicious looking subfolder titled 3.6) Reinstalling mypy with pip install --upgrade --force-reinstall mypy  To reproduce:  Create a python 3.6 project install mypy 0.761 (latest) in the project venv scan the project with mypy (mypy .) update the project to python 3.7 add a file with this code in it: from datetime import datetime datetime.fromisoformat('2011-11-04 00:05:23.283')  scan the project again (mypy .) [UPDATE: this actually works fine. It was rerunning my precommit hooks without reinstalling pre-commit on the new Python version venv that was causing the problems.] 

Answer: You are running mypy under an older version of Python. mypy defaults to the version of Python that is used to run it. You have two options:  You can change the Python language version with the --python-version command-line option:  This flag will make mypy type check your code as if it were run under Python version X.Y. Without this option, mypy will default to using whatever version of Python is running mypy.  I'd put this in the project mypy configuration file; the equivalent of the command-line switch is named python_version; put it in the global [mypy] section: [mypy] python_version = 3.7   Install mypy into the virtualenv of your project, so that it uses the exact same Python version.   Note that if you see this issue (and didn't accidentally set --python-version, on the command-line or in a configuration file, you are certainly not running mypy from your project venv.
Question: How to make a dict from an enum? from enum import Enum class Shake(Enum): VANILLA = "vanilla" CHOCOLATE = "choc" COOKIES = "cookie" MINT = "mint" dct = {} for i in Shake: dct[i]=i.value print(dct)  Output: {<Shake.VANILLA: 'vanilla'>: 'vanilla', <Shake.CHOCOLATE: 'choc'>: 'choc', <Shake.COOKIES: 'cookie'>: 'cookie', <Shake.MINT: 'mint'>: 'mint'}  But I want the key to be VANILLA and not <Shake.VANILLA: 'vanilla'>

Answer: you can just use a dictionary comprehension from enum import Enum class Shake(Enum): VANILLA = "vanilla" CHOCOLATE = "choc" COOKIES = "cookie" MINT = "mint" dct = {i.name: i.value for i in Shake} print(dct)  OUTPUT {'VANILLA': 'vanilla', 'CHOCOLATE': 'choc', 'COOKIES': 'cookie', 'MINT': 'mint'}
Question: How can I create a Python C extension wheel for MacOS that is backwards compatible (MacOS 10.9+) using MacOS 10.15?  This is what I have so far: export MACOSX_DEPLOYMENT_TARGET=10.9 python -m pip wheel . -w wheels --no-deps python -m pip install delocate for whl in wheels/*.whl; do delocate-wheel -w wheels_fixed -v "$whl" done  Unfortunately, pip wheel generates a file myapp-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl, and unlike auditwheel on Linux, delocate-wheel does not modify the name of the wheel. As a result, if I upload it on PyPI using twine, only users with MacOS 10.15 are able to install it using pip. I guess I could manually rename it to myapp-0.0.1-cp37-cp37m-macosx_10_9_x86_64.whl, but this does not sound right to me.  For the builds I am just using the GitHub Actions MacOS virtual machines. Thank you. PS: The compiler used for the build is GCC9

Answer: I found the solution to my problem and I will post the answer here in case someone else has the same problem. In order to fix the problem I had to also set export MACOSX_DEPLOYMENT_TARGET=10.9 before I install python using pyenv. Now pip wheel creates my wheel with the tag macosx_10_9_x86_64.  Thank you. PS: When installing python via pyenv, python is compiled from source, and somehow it takes into account the flag MACOSX_DEPLOYMENT_TARGET.
Question: I asked the question a while back for python, but now I need to do the same thing in PySpark. I have a dataframe (df) like so: |cust_id|address |store_id|email |sales_channel|category| ------------------------------------------------------------------- |1234567|123 Main St|10SjtT |idk@gmail.com|ecom |direct | |4567345|345 Main St|10SjtT |101@gmail.com|instore |direct | |1569457|876 Main St|51FstT |404@gmail.com|ecom |direct |  and I would like to combine the last 4 fields into one metadata field that is a json like so: |cust_id|address |metadata | ------------------------------------------------------------------------------------------------------------------- |1234567|123 Main St|{'store_id':'10SjtT', 'email':'idk@gmail.com','sales_channel':'ecom', 'category':'direct'} | |4567345|345 Main St|{'store_id':'10SjtT', 'email':'101@gmail.com','sales_channel':'instore', 'category':'direct'}| |1569457|876 Main St|{'store_id':'51FstT', 'email':'404@gmail.com','sales_channel':'ecom', 'category':'direct'} |  Here's the code I used to do this in python: cols = [ 'store_id', 'store_category', 'sales_channel', 'email' ] df1 = df.copy() df1['metadata'] = df1[cols].to_dict(orient='records') df1 = df1.drop(columns=cols)  but I would like to translate this to PySpark code to work with a spark dataframe; I do NOT want to use pandas in Spark.

Answer: Use to_json function to create json object! Example: from pyspark.sql.functions import * #sample data df=spark.createDataFrame([('1234567','123 Main St','10SjtT','idk@gmail.com','ecom','direct')],['cust_id','address','store_id','email','sales_channel','category']) df.select("cust_id","address",to_json(struct("store_id","category","sales_channel","email")).alias("metadata")).show(10,False) #result +-------+-----------+----------------------------------------------------------------------------------------+ |cust_id|address |metadata | +-------+-----------+----------------------------------------------------------------------------------------+ |1234567|123 Main St|{"store_id":"10SjtT","category":"direct","sales_channel":"ecom","email":"idk@gmail.com"}| +-------+-----------+----------------------------------------------------------------------------------------+   to_json by passing list of columns: ll=['store_id','email','sales_channel','category'] df.withColumn("metadata", to_json(struct([x for x in ll]))).drop(*ll).show() #result +-------+-----------+----------------------------------------------------------------------------------------+ |cust_id|address |metadata | +-------+-----------+----------------------------------------------------------------------------------------+ |1234567|123 Main St|{"store_id":"10SjtT","email":"idk@gmail.com","sales_channel":"ecom","category":"direct"}| +-------+-----------+----------------------------------------------------------------------------------------+
Question: The title pretty much sums it up already. I have a piece of code that calls os.getenv to get both a URL as well as a token in order to connect to a service. The code lives in a module and will only be imported from there, i.e. it's not a script.  It's not a huge issue at all, since I really only need to crash and display the message saying that there are unset values, but it got me thinking about which of Python's built-in exceptions would be the best fit. I found the EnvironmentError, but that seems to function as base class from which IOError and other OS related exceptions inherit. Would it be as simple as a ValueError, as it's really just a value that's missing? Thanks!

Answer: Well most built in concrete exception classes are for specific use cases, and this one does not really fit in any but RuntimeError. But I would advise you to use a custom Exception subclass.
Question: I am trying to collect data from different .csv files, that share the same column names. However, some csv files have their headers located in different rows.  Is there a way to determine the header row dynamically based on the first row that contains "most" values (the actual header names)?  I tried the following: def process_file(file, path, col_source, col_target): global df_master print(file) df = pd.read_csv(path + file, encoding = "ISO-8859-1", header=None) df = df.dropna(thresh=2) ## Drop the rows that contain less than 2 non-NaN values. E.g. metadata df.columns = df.iloc[0,:].values df = df.drop(df.index[0])  However, when using pandas.read_csv(), it seems like the very first value determines the size of the actual dataframe as I receive the following error message:  pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 4, saw 162  As you can see in this case the header row would have been located in row 4. When adding error_bad_lines=False to read_csv, only the metadata will be read into the dataframe.  The files can have either the structure of: a "Normal" File: row1 col1 col2 col3 col4 col5 row2 val1 val1 val1 val1 val1 row3 val2 val2 val2 val2 val2 row4  or a structure with meta data before header: row1 metadata1 row2 metadata2 row3 col1 col2 col3 col4 col5 row4 val1 val1 val1 val1 val1  Any help much appreciated!

Answer: IMHO the simplest way if to forget pandas for a while:  you open the file as a text file for reading you start parsing it line by line, guessing whether the line is  metadata header the true header line data lines   A simple way is to concatenate all the lines starting from the true header line in a single string (let us call it buffer), and then use pd.read_csv(io.StringIO(buffer), ...)
Question: Does shapley support logistic regression models? Running the following code i get: logmodel = LogisticRegression() logmodel.fit(X_train,y_train) predictions = logmodel.predict(X_test) explainer = shap.TreeExplainer(logmodel ) Exception: Model type not yet supported by TreeExplainer: <class 'sklearn.linear_model.logistic.LogisticRegression'>  P.S. You are supposed to use a different explainder for different models

Answer: Shap is model agnostic by definition. It looks like you have just chosen an explainer that doesn't suit your model type. I suggest looking at KernelExplainer which as described by the creators here is   An implementation of Kernel SHAP, a model agnostic method to estimate SHAP values for any model. Because it makes not assumptions about the model type, KernelExplainer is slower than the other model type specific algorithms.  The documentation for Shap is mostly solid and has some decent examples.
Question: I have a pypi package called collectiondbf which connects to an API with a user entered API key. It is used in a directory to download files like so: python -m collectiondbf [myargumentshere..]  I know this should be basic knowledge, but I'm really stuck on the question: How can I save the keys users give me in a meaningful way so that they do not have to enter them every time? I would like to use the following solution using a config.json file, but how would I know the location of this file if my package will be moving directories? Here is how I would like to user it but obviously it won't work since the working directory will change import json if user_inputed_keys: with open('config.json', 'w') as f: json.dump({'api_key': api_key}, f) 

Answer: Most common operating systems have the concept of an application directory that belongs to every user who has an account on the system. This directory allows said user to create and read, for example, config files and settings. So, all you need to do is make a list of all distros that you want to support, find out where they like to put user application files, and have a big old if..elif..else chain to open the appropriate directory.  Or use appdirs, which does exactly that already: from pathlib import Path import json import appdirs CONFIG_DIR = Path(appdirs.user_config_dir(appname='collectiondbf')) # magic CONFIG_DIR.mkdir(parents=True, exist_ok=True) config = CONFIG_DIR / 'config.json' if not config.exists(): with config.open('w') as f: json.dumps(get_key_from_user(), f) with config.open('r') as f: keys = json.load(f) # now 'keys' can safely be imported from this module
Question: I am training a tensorflow keras sequential model on around 20+ GB text based categorical data in a postgres db and i need to give class weights to the model. Here is what i am doing. class_weights = sklearn.utils.class_weight.compute_class_weight('balanced', classes, y) model.fit(x, y, epochs=100, batch_size=32, class_weight=class_weights, validation_split=0.2, callbacks=[early_stopping])  Since i can't load the whole thing in memory i figured i can use fit_generator method in keras model.  However how can i calculate the class weights on this data? sklearn does not provide any special function for this, is it the right tool for this ? I thought of doing it on multiple random samples but is there a better approach where whole data can be used ?

Answer: You can use the generators and also you can compute the class weights.  Let's say you have your generator like this train_generator = train_datagen.flow_from_directory( 'train_directory', target_size=(224, 224), batch_size=32, class_mode = "categorical" )  and the class weights for the training set can be computed like this class_weights = class_weight.compute_class_weight( 'balanced', np.unique(train_generator.classes), train_generator.classes)  [EDIT 1] Since you mentioned about postgres sql in the comments, I am adding the prototype answer here.  first fetch the count for each classes using a separate query from postgres sql and use it to compute the class weights. you can compute it manually. The basic logic is the count of least weighed class gets the value 1, and the rest of the classes get <1 based on the relative count to the least weighed class.  for example you have 3 classes A,B,C with 100,200,150 then class weights becomes {A:1,B:0.5,C:0.66} let compute it manually after fetching the values from postgres sql.  [Query] cur.execute("SELECT class, count(*) FROM table group by classes order by 1") rows = cur.fetchall()  The above query will return rows with tuples (class name, count for each class) ordered from least to highest.  Then the below line will code will create the class weights dictionary class_weights = {} for row in rows: class_weights[row[0]]=rows[0][1]/row[1] #dividing the least value the current value to get the weight, # so that the least value becomes 1, # and other values becomes < 1
Question: I'm working with MNIST dataset from Kaggle challange and have troubles preprocessing with data. Furthermore, I don't know what are the best practices and was wondering if you could advise me on that. Disclaimer: I can't just use torchvision.datasets.mnist because I need to use Kaggle's data for training and submission.  In this tutorial, it was advised to create a Dataset object loading .pt tensors from files, to fully utilize GPU. In order to achieve that, I needed to load the csv data provided by Kaggle and save it as .pt files: import pandas as pd import torch import numpy as np # import data digits_train = pd.read_csv('data/train.csv') train_tensor = torch.tensor(digits_train.drop(label, axis=1).to_numpy(), dtype=torch.int) labels_tensor = torch.tensor(digits_train[label].to_numpy()) for i in range(train_tensor.shape[0]): torch.save(train_tensor[i], "data/train-" + str(i) + ".pt")  Each train_tensor[i].shape is torch.Size([1, 784]) However, each such .pt file has size of about 130MB. A tensor of the same size, with randomly generated integers, has size of 6.6kB. Why are these tensors so huge, and how can I reduce their size? Dataset is 42 000 samples. Should I even bother with batching this data? Should I bother with saving tensors to separate files, rather than loading them all into RAM and then slicing into batches? What is the most optimal approach here?

Answer: As explained in this discussion, torch.save() saves the whole tensor, not just the slice. You need to explicitly copy the data using clone(). Don't worry, at runtime the data is only allocated once unless you explicitly create copies. As a general advice: If the data easily fits into your memory, just load it at once. For MNIST with 130 MB that's certainly the case. However, I would still batch the data because it converges faster. Look up the advantages of SGD for more details.
Question: I'm facing the following issue. My .env files contains a line like: export SERVERNAMES="localhost domain1 domain2 domain3" <- exactly this kind of format  But the variable called SERVERNAMES is used multiple times at multiple locations of my deployment so i can't declare this as compatible list of strings that settings.py can use imidiatly. beside I don't like to set multiple variables of .env for basically the same thing. So my question is how can I format my ALLOWED_HOSTS to be compatible with my settings.py. Something like this does not seem to work it seems: ALLOWED_HOSTS = os.environ.get('SERVERNAMES').split(',')  Thanks and kind regards

Answer: Simply split your SERVERNAMES variable using space as separator instead of comma ALLOWED_HOSTS = os.environ.get('SERVERNAMES').split(' ')
Question: I have data that is over 400,000 lines long. When running this code: f=pd.read_csv(filename,error_bad_lines=False)  I get the following error: pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at row 454751  My data by the end of the file looks like this: BTC 9948 8718 1.57E+12 ASK BTC 52 8718 1.57E+12 ASK BTC 120 8718 1.57E+12 ASK BTC 200 8718 1.57E+12 ASK BTC 150 8718 1.57E+12 ASK BTC 50 8718 1.57E+12 ASK BTC 10 8718 1.57E+12 ASK BTC 57 8718 1.57E+12 ASK BTC 50 8718 1.57E+12 ASK BTC 50191 8718  Line 454751 is this one: BTC 50 8718 1.57E+12 ASK I tried running error_bad_lines=False as seen above but that still doesnt work. I also searched for quotes in my file but I do not have any.

Answer: Changing the Parser engine from C to Python should solve your problem. Use the following line to read your csv: f = pd.read_csv(filename, error_bad_lines=False, engine="python")  From the read_csv documentation:  engine : {‘c’, ‘python’}, optional Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.
Question: i am trying to use jupyter notebooks from vs code and installed jupyter notebook extension and i am using (base)conda environment for execution. while this happened Error: Jupyter cannot be started. Error attempting to locate jupyter: at A.startServer (c:\Users\DELL\.vscode\extensions\ms-python.python-2020.2.63990\out\client\extension.js:1:784356) at async A.ensureServerAndNotebookImpl (c:\Users\DELL\.vscode\extensions\ms-python.python-2020.2.63990\out\client\extension.js:1:783811) at async A.ensureServerAndNotebook (c:\Users\DELL\.vscode\extensions\ms-python.python-2020.2.63990\out\client\extension.js:1:783612) at async A.submitCode (c:\Users\DELL\.vscode\extensions\ms-python.python-2020.2.63990\out\client\extension.js:1:780564) at async A.reexecuteCell (c:\Users\DELL\.vscode\extensions\ms-python.python-2020.2.63990\out\client\extension.js:75:879318)   how to resolve this issue ?

Answer: I had exactly the same problem when I installed Visual Studio Code and tried to run some Python code from a jupyter notebook on my fresh Ubuntu 18.04. How I solved it:  Make sure you have installed the Jupyter Extension in VS Code. (for those who don't read the SO question :))  Press Command+Shift+P to open a new command pallete  Type >Python: Select Intepreter to start jupyter notebook server  Open the notebook again   And it worked fine. Hope it works for you.
Question: Since python3.6, you can use underscore to separate digits of an integer. For example x = 1_000_000 print(x) #1000000  This feature was added to easily read numbers with many digits and I found it very useful. But when you print the number you always get a number not separated with digits. Is there a way to print the number with its digits separated with underscore. P.S. I want the output as integer not as string. Not "1_000_000" but 1_000_000

Answer: Try using this: >>> x = 1_000_000 >>> print(f"{x:_}") 1_000_000  Here are details Another way would be to use format explicitly: >>> x = 1_000_000 >>> print(format(x, '_d')) 1_000_000