eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a sample data as follows: import pandas as pd df = pd.DataFrame({"x": [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120], "id": [1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], "label": ["a", "a", "a", "b", "a", "b", "b", "b", "a", "b", "a", "b"]})  So my data look like this  x id label 10 1 a 20 1 a 30 1 a 40 1 b 50 2 a 60 2 b 70 3 a 80 3 a 90 4 b 100 4 a 110 5 b 120 5 a  I would like to split this data into two groups (train, test) based on label distribution given the number of test samples (e.g. 6 samples). My settings prefers to define size of test set as integer representing the number of test samples rather than percentage. However, with my specific domain, any id MUST be allocated in ONLY one group. For example, if id 1 was assigned to the training set, other samples with id 1 cannot be assigned to the test set. So the expected output are 2 dataframes as follows: Training set  x id label 10 1 a 20 1 a 30 1 a 40 1 b 50 2 a 60 2 b  Test set  x id label 70 3 a 80 3 a 90 4 b 100 4 a 110 5 b 120 5 a  Both training set and test set have the same class distribution (a:b is 4:2) and id 1, 2 were assigned to only the training set while id 3, 4, 5 were assigned to only the test set. I used to do with sklearn train_test_split but I could not figure out how to apply it with such a condition. May I have your suggestions how to handle such conditions?

Answer: sklearn.model_selection has several other options other than train_test_split. One of them, aims at solving what you're after. In this case you could use GroupShuffleSplit, which as mentioned inthe docs it provides randomized train/test indices to split data according to a third-party provided group. You also have GroupKFold for these cases which is very useful. from sklearn.model_selection import GroupShuffleSplit X = df.drop('label',1) y=df.label  You can now instantiate GroupShuffleSplit, and do as you would with train_test_split, with the only difference of specifying a group column, which will be used to split X and y so the groups are split according the the groups values: gs = GroupShuffleSplit(n_splits=2, test_size=.6, random_state=0) train_ix, test_ix = next(gs.split(X, y, groups=X.id))  Now you can index the dataframe to create the train and test sets: X_train = X.loc[train_ix] y_train = y.loc[train_ix] X_test = X.loc[test_ix] y_test = y.loc[test_ix]   Giving: print(X_train) x id 4 50 2 5 60 2 8 90 4 9 100 4 10 110 5 11 120 5  And for the test set: print(X_test) x id 0 10 1 1 20 1 2 30 1 3 40 1 6 70 3 7 80 3
Question: I want to upload files to EC2 instance using pysftp library (Python script). So I have created small Python script which is using below line to connect pysftp.Connection( host=Constants.MY_HOST_NAME, username=Constants.MY_EC2_INSTANCE_USERNAME, private_key="./mypemfilelocation.pem", ) some code here ..... pysftp.put(file_to_be_upload, ec2_remote_file_path)  This script will upload files from my local Windows machine to EC2 instance using .pem file and it works correctly. Now I want to do this action using AWS lambda with API Gateway functionality. So I have uploaded Python script to AWS lambda. Now I am not sure how to use pysftp library in AWS lambda, so I found solution that add pysftp library Layer in AWS lambda Layer. I did it with  pip3 install pysftp -t ./library_folder  And I make zip of above folder and added in AWS lambda Layer. But still I got so many errors like one by one :-  No module named 'pysftp' No module named 'paramiko' Undefined Symbol: PyInt_FromLong cannot import name '_bcrypt' from partially initialized module 'bcrypt' (most likely due to a circular import) cffi module not found  I just fade up of above errors I didn't find the proper solution. How can I can use pysftp library in my AWS lambda seamlessly?

Answer: I build pysftp layer and tested it on my lambda with python 3.8. Just to see import and basic print: import json import pysftp def lambda_handler(event, context): # TODO implement print(dir(pysftp)) return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }  I used the following docker tool to build the pysftp layer:  https://github.com/lambci/docker-lambda  So what I did for pysftp was: # create pysftp fresh python 3.8 environment python -m venv pysftp # activate it source pysftp/bin/activate cd pysftp # install pysftp in the environemnt pip3 install pysftp # generate requirements.txt pip freeze > requirements.txt # use docker to construct the layer docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.8 python3.8 -m pip --isolated install -t ./mylayer -r requirements.txt zip -r pysftp-layer.zip .  And the rest is uploading the zip into s3, creating new layer in AWS console, setting Compatible runtime to python 3.8 and using it in my test lambda function. You can also check here how to use this docker tool (the docker command I used is based on what is in that link).  Hope this helps
Question: Dockerfile FROM python:3.7.4-alpine ENV PYTHONUNBUFFERED 1 ENV PYTHONDONTWRITEBYTECODE 1 ENV LANG C.UTF-8 MAINTAINER "mail@gmail.com" RUN apk update && apk add postgresql-dev gcc musl-dev RUN apk --update add build-base jpeg-dev zlib-dev RUN pip install --upgrade setuptools pip RUN mkdir /code WORKDIR /code COPY requirements.txt /code/ RUN pip install -r requirements.txt COPY . /code/ #CMD ["gunicorn", "--log-level=DEBUG", "--timeout 90", "--bind", "0.0.0.0:8000", "express_proj.wsgi:application"] ENTRYPOINT ["./docker-entrypoint.sh"]  docker-entrypoint.sh #!/bin/bash # Prepare log files and start outputting logs to stdout touch /code/gunicorn.log touch /code/access.log tail -n 0 -f /code/*.log & # Start Gunicorn processes echo Starting Gunicorn. exec gunicorn express_proj.wsgi:application \ --name express \ --bind 0.0.0.0:8000 \ --log-level=info \ --log-file=/code/gunicorn.log \ --access-logfile=/code/access.log \ --workers 2 \ --timeout 90 \ "$@"  Getting Error standard_init_linux.go:211: exec user process caused "no such file or directory" Need help. Some saying to use dos2unix(i do not know hoe to use it.)

Answer: The "shebang" line at the start of a script says what interpreter to use to run it. In your case, your script has specified #!/bin/bash, but Alpine-based Docker images don't typically include GNU bash; instead, they have a more minimal /bin/sh that includes just the functionality in the POSIX shell specification. Your script isn't using any of the non-standard bash extensions, so you can just change the start of the script to #!/bin/sh
Question: While updating my packages I've noticed that there is a package named "matplotlib-base". I couldn't figure out what the difference to "matplotlib" is, neither on the official website nor here on Stack Overflow, and I also couldn't find any repository to compare the code. Any ideas? 

Answer: The packages are similar, but differ in their dependencies: matplotlib depends on matplotlib-base and pyqt. Therefore installing matplotlib will also pull in the qt stack, while installing matplotlib-base does not. Users that do not need qt backends and prefer a slim installation will prefer matplotlib-base over matplotlib. See also: https://conda-forge.org/docs/maintainer/knowledge_base.html#matplotlib
Question: I just can't figure out what "==" means at the second line: - It is not a test, there is no if statement... - It is not a variable declaration... I've never seen this before, the thing is data.ctage==cat is a pandas Series and not a test... for cat in data["categ"].unique(): subset = data[data.categ == cat] # Création du sous-échantillon print("-"*20) print('Catégorie : ' + cat) print("moyenne:\n",subset['montant'].mean()) print("mediane:\n",subset['montant'].median()) print("mode:\n",subset['montant'].mode()) print("VAR:\n",subset['montant'].var()) print("EC:\n",subset['montant'].std()) plt.figure(figsize=(5,5)) subset["montant"].hist(bins=30) # Crée l'histogramme plt.show() # Affiche l'histogramme 

Answer: It is testing each element of data.categ for equality with cat. That produces a vector of True/False values. This is passed as in indexer to data[], which returns the rows from data that correspond to the True values in the vector. To summarize, the whole expression returns the subset of rows from data where the value of data.categ equals cat. (Seems possible the whole operation could be done more elegantly using data.groupBy('categ').apply(someFunc).)
Question: The program takes an optional command line argument (which is meant to be a directory path) I am using python pathlib and shutil to move files. Here's the code: from pathlib import Path path = Path(sys.argv[1]) shutil.move(path / file, path / e.upper())  Where e is just a string representing certain file extension; Input:  python3 app.py /home/user/Desktop  This code generates an error:  'PosixPath' object has no attribute 'rstrip'  The / operator works fine if I don't specify the second argument in the command line (and use Path.cwd() as the path instead)

Answer: Use the rename function of Path to move a file, if you're using the pathlib module. ie. (path / file).rename(path / e.upper())  Otherwise, if you wish to use the shutil module, then you must convert your paths to strings before passing them to shutil.move() ie. shutil.move(str(path / file), str(path / e.upper()))
Question: Are NumPy and TensorFlow the same thing? I just started learning programming; I was learning AI and found TensorFlow. I started to look at videos and I saw the code snippets below: import tensorflow as tf tf.ones([1,2,3]) tf.zeros([2,3,2])  import numpy as np np.zeros([2,3,2]) np.ones([1,2,3]) 

Answer: Although the method names and parameters look identical, they are not the same thing. This becomes clear in the debugger. Just assign the results to variables and inspect them:  As you can see, Tensorflow gives you an EagerTensor and NumPy gives you an NDArray. Tensorflow is a library for artificial intelligence, especially machine learning. Numpy is a library for doing numerical calculations. They are often used in combination, because it's often required to pre-process data, which can be done with NumPy, and then do the machine learning on the processed data with Tensorflow.
Question: I have an existing solution to split a dataframe with one column into 2 columns. df['A'], df['B'] = df['AB'].str.split(' ', 1).str  Recently, I got the following warning FutureWarning: Columnar iteration over characters will be deprecated in future releases. How to fix this warning? I'm using python 3.7

Answer: That's not entirely correct, plus the trailing .str does not make sense. Since split with expand returns a DataFrame, this is easier: df[['A', 'B']] = df['AB'].str.split(' ', n=1, expand=True)   Your existing method without expand returns a single Series with a list of columns. I'm not sure what version of pandas used to work with your code but AFAIK you'll need to make some tweaks for this to work with pandas (>= 1.0) today. Assignment in this way is tedious but still possible. s = df['AB'].str.split(' ', n=1) df['A'], df['B'] = s.str[0], s.str[1]  I prefer the expand solution as it's a line shorter.
Question: I have a dataframe  'A' 'B' 'C' 'X' ,'Y' , 'Z' 0 1 2 3 4 5  and a list l=[A,B,C,D,E,F] I want to use that that list to select columns that are in the list but ignore the ones that don't appear. So the expected output is   'A' 'B' 'C' 0 1 2 3 4 5 

Answer: Use DataFrame.loc for select all rows by : and columns by mask created by Index.isin : df = df.loc[:, df.columns.isin(l)]  Or get columns names by Index.intersection: df = df[df.columns.intersection(l)]   print (df) A B C 0 NaN NaN NaN 1 NaN NaN NaN 2 NaN NaN NaN 3 NaN NaN NaN 4 NaN NaN NaN 5 NaN NaN NaN
Question: I have a list like this: list_1 = [np.NaN, np.NaN, 1, np.NaN, np.NaN, np.NaN, 0, np.NaN, 1, np.NaN, 0, 1, np.NaN, 0, np.NaN, 1, np.NaN]  So there are intervals that begin with 1 and end with 0. How can I replace the values in those intervals, say with 1? The outcome will look like this: list_2 = [np.NaN, np.NaN, 1, 1, 1, 1, 0, np.NaN, 1, 1, 0, 1, 1, 0, np.NaN, 1, np.NaN]  I use NaN in this example, but a generalized solution that can apply to any value will also be great 

Answer: Pandas solution: s = pd.Series(list_1) s1 = s.eq(1) s0 = s.eq(0) m = (s1 | s0).where(s1.cumsum().ge(1),False).cumsum().mod(2).eq(1) s.loc[m & s.isna()] = 1 print(s.tolist()) #[nan, nan, 1.0, 1.0, 1.0, 1.0, 0.0, nan, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, nan, 1.0, 1.0]   but if there is only 1, 0 or NaN you can do: s = pd.Series(list_1) s.fillna(s.ffill().where(lambda x: x.eq(1))).tolist()  output [nan, nan, 1.0, 1.0, 1.0, 1.0, 0.0, nan, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, nan, 1.0, 1.0]
Question: I'm attempting to use freezegun in my unit tests to patch a field in a dataclass that is set to the current date when the object is initialised. I would imagine the question is relevant to any attempt to patch a function being used as a default_factory outside of just freezegun. The dataclass is frozen so its immutable. For example if my dataclass is: @dataclass(frozen=True) class MyClass: name: str timestamp: datetime.datetime = field(init=False, default_factory=datetime.datetime.now)  When I patch datetime with freezegun, it has no impact on the initialisation of the timestamp in MyClass (it still sets timestamp to the current date returned by now() in the unit test, causing the test to fail).  I'm assuming it has to do with the default factory and module being loaded well before the patch is in place. I have tried patching datetime, and then reloading the module with importlib.reload but with no luck. The solution I have at the moment is: @dataclass(frozen=True) class MyClass: name: str timestamp: datetime.datetime = field(init=False) def __post_init__(self): object.__setattr__(self, "timestamp", datetime.datetime.now())  which works.  Ideally though, I would like a non-invasive solution that doesn't require me changing my production code to enable my unit tests. 

Answer: You're right, the dataclass creation process does something strange here which leads to your current problem. It binds the factory function during class creation, which means that it holds a reference of the code before freezegun had a chance to patch it. Here is an example without dataclasses that runs into the same issue: from datetime import datetime from freezegun import freeze_time class Foo: # looks up the function at class creation time now_func = datetime.now def __init__(self): # asks datetime for a reference at instance creation time self.timestamp_a = datetime.now() # uses an old reference we couldn't patch self.timestamp_b = Foo.now_func() with freeze_time(datetime(2020, 1, 1)): foo = Foo() assert foo.timestamp_a == datetime(2020, 1, 1) # works assert foo.timestamp_b == datetime(2020, 1, 1) # raises an AssertionError   As to how to solve the problem, you can theoretically hack MyClass.__init__.__closure__ during your tests to switch out the functions, but that's a bit mad.  Something that is still a bit better than overwriting timestamp in a __post_init__ might be to just delegate the function call with a lambda so that the name lookup is delayed to instantiation time: timestamp: datetime = field(init=False, default_factory=lambda: datetime.now())  Or you can start using a different datetime library like pendulum that supports freezing time out of the box. FWIW, this is what I ended up doing.
Question: I would like to downgrade the Torch version used in my Google Colab notebooks. How could I do that?

Answer: Run from your cell: !pip install torch==version  Where version could be, for example, 1.3.0 (default is 1.4.0). You may have to downgrade torchvision appropriately as well so you would go with: !pip install torch==1.3.0 torchvision==0.4.1  On the other hand PyTorch provides backward compatibility between major versions so there should be no need for downgrading. Remember you have to restart your Google Colab for changes to take effect
Question: I am using AWS Lambda to create my APIs and want to return an array's data in JSON format. However, when I call the lambda, it is able to return the required JSON data but it is coming as a string in double quotes. I tried running the same code in my Python IDE and everything works fine but when I try to return it in Lambda it is coming as a string. Has this got something to do with how Lambda handles the return statement in Python functions? This is how I'm returning data in my Lambda: return { 'statusCode': 200, 'headers': { "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": 'GET, POST, PUT, DELETE, OPTIONS' }, 'body': json.dumps(json_data,default = myconverter) }  Here json_data is a python list which is populated with the data that is being retrieved from the database for the specific unique ID passed by the user and myconverter is the JSON encoder that I've written. The output that I'm getting is: { "statusCode": 200, "headers": { "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, OPTIONS" }, "body": "[{\"Dp_Record_Id\": 2, \"DP_TYPE\": \"NSDL\", \"DP_ID\": \"40877589\", \"CLIENT_ID\": \"1232\", \"Default_flag\": \"Y\"}]" }  Here, I want the "body" to just return an array of the data without double quotes, like this: "body": [{\"Dp_Record_Id\": 2, \"DP_TYPE\": \"NSDL\", \"DP_ID\": \"40877589\", \"CLIENT_ID\": \"1232\", \"Default_flag\": \"Y\"}]  Please let me know if this is possible and how can it be done. Any help on this would be appreciated

Answer: I'm not sure if this is what you want, but you can just do 'body': json_data. I tested this now in my λ function: Lambda function import json def lambda_handler(event, context): json_data = [{"Dp_Record_Id": 2, "DP_TYPE": "NSDL", "DP_ID": "40877589", "CLIENT_ID": "1232", "Default_flag": "Y"}] return {'statusCode': 200, 'headers': { "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": 'GET, POST, PUT, DELETE, OPTIONS' }, 'body': json_data }  Result Response: { "statusCode": 200, "headers": { "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, OPTIONS" }, "body": [ { "Dp_Record_Id": 2, "DP_TYPE": "NSDL", "DP_ID": "40877589", "CLIENT_ID": "1232", "Default_flag": "Y" } ] }
Question: I am trying to download songs from youtube using python 3.8 and youtube_dl 2020.3.24. But the weird thing is that most songs I try to download don't get downloaded. I'm talking 99% of them. The ones that do get downloaded get the following Error from youtube_dl: ERROR: unable to download video data: HTTP Error 403: Forbidden It is worth saying that this happened overnight and I did not change any code. before this everything worked fine. I have friends who ran the same code and they did not get this Error

Answer: Same problem many times .. solution: youtube-dl --rm-cache-dir Cause of the problem: Sometimes I download playlists of large videos and I force it to stop downloading, the next time I run the command to resume the download, the 403 problem arises At the moment, the cache directory is used only to store youtube players for obfuscated signatures. Since all videos in playlist use simple signatures Playlist caching is an obvious way to detect changed titles or changed playlists in general
Question: I tried to use bert-tensorflow in Google Colab, but I got the following error:  --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in () 1 import bert ----> 2 from bert import run_classifier_with_tfhub # run_classifier 3 from bert import optimization 4 from bert import tokenization 1 frames /usr/local/lib/python3.6/dist-packages/bert/optimization.py in () 85 86 ---> 87 class AdamWeightDecayOptimizer(tf.train.Optimizer): 88 """A basic Adam optimizer that includes "correct" L2 weight decay.""" 89  AttributeError: module 'tensorflow._api.v2.train' has no attribute 'Optimizer'  Here is the code I tried:  Install the libraries:  !pip install --upgrade --force-reinstall tensorflow !pip install --upgrade --force-reinstall tensorflow-gpu !pip install tensorflow_hub !pip install sentencepiece !pip install bert-tensorflow  Run this code:  from sklearn.model_selection import train_test_split import pandas as pd from datetime import datetime from tensorflow.keras import optimizers import bert from bert import run_classifier from bert import optimization from bert import tokenization  I've also tried import tensorflow.compat.v1 as tf tf.disable_v2_behavior() But got the same error.

Answer: I did some experimentation in my own colab notebook (please provide a link next time) and I found that in the error message, there was  class AdamWeightDecayOptimizer(tf.train.Optimizer):  this being the header of the class. But there is nothing like tf.train.optimizer instead it should be : class AdamWeightDecayOptimizer(tf.compat.v1.train.Optimizer):  The link where there is exact issue with (lol) exact same line is here
Question: In resume, I have two keys in the same dictionary where each one has their corresponding lists. I try to compare both list to check common and differential elements. It means that the output I will count how many elements are identical or present in only one key's list. from the beginning I am inserting the elements using the files as arguments and they are read in the function def shared(list): dict_shared = {} for i in list: infile = open(i, 'r') if i not in dict_shared: dict_shared[i] = [] for line in infile: dict_shared[spacer].append(record.id) return dict_shared  Now I am stuck trying to find a way to compare the lists created and present in the dictionary. dict = {a:[1,2,3,4,5], b:[2,3,4,6]}  My intention is to compare the lists in order to have the lines shared between two texts. a: [1,5] b: [6] a-b: [2,3,4]  From now I can't find a way to solve this. Any suggestion?

Answer: A solution with list comprehension would be: dictionary = {'a':[1,2,3,4,5], 'b':[2,3,4,6]} only_in_a = [x for x in dictionary['a'] if not x in dictionary['b']] only_in_b = [x for x in dictionary['b'] if not x in dictionary['a']] in_both = [x for x in dictionary['a'] if x in dictionary['b']]  Note that this is not especially wise in terms of complexity, for larger lists.
Question: What is the HTML equivalent for &nbsp (space) in Dash? html.Div( [ dcc.Input(), <add horizontal space here> dcc.Input() ] ) 

Answer: If you want to add some space between components you can simply use CSS-properties for this: html.Div( [ dcc.Input(), dcc.Input(style={"margin-left": "15px"}) ] )  This adds a margin to the left of your second Input. Have a look at the layout-section in the Plotly Dash documentation and CSS documentation about margin:  https://dash.plotly.com/layout https://www.w3schools.com/csSref/pr_margin-left.asp
Question: I am trying to load a model state_dict I trained on Google Colab GPU, here is my code to load the model: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") model = models.resnet50() num_ftrs = model.fc.in_features model.fc = nn.Linear(num_ftrs, n_classes) model.load_state_dict(copy.deepcopy(torch.load("./models/model.pth",device))) model = model.to(device) model.eval()  Here is the error:  state_dict = state_dict.copy() AttributeError: 'function' object has no attribute 'copy'  Pytorch : >>> import torch >>> print (torch.__version__) 1.4.0 >>> import torchvision >>> print (torchvision.__version__) 0.5.0  Please help I have searched everywhere to no avail [full error details][1] https://i.sstatic.net/s22DL.png

Answer: I am guessing this is what you did by mistake. You saved the function  torch.save(model.state_dict, 'model_state.pth') instead of the state_dict() torch.save(model.state_dict(), 'model_state.pth') Otherwise, everything should work as expected. (I tested the following code on Colab) Replace model.state_dict() with model.state_dict to reproduce error import copy model = TheModelClass() torch.save(model.state_dict(), 'model_state.pth') device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") model.load_state_dict(copy.deepcopy(torch.load("model_state.pth",device)))