eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a bucket in s3 called "sample-data". Inside the Bucket I have folders labelled "A" to "Z". Inside each alphabetical folder there are more files and folders. What is the fastest way to download the alphabetical folder and all it's content? For example --> sample-data/a/foo.txt,more_files/foo1.txt In the above example the bucket sample-data contains an folder called a which contains foo.txt and a folder called more_files which contains foo1.txt I know how to download a single file. For instance if i wanted foo.txt I would do the following.  s3 = boto3.client('s3') s3.download_file("sample-data", "a/foo.txt", "foo.txt")  However i am wondering if i can download the folder called a and all it's contents entirely? Any help would be appreciated.

Answer: I think your best bet would be the awscli aws s3 cp --recursive s3://mybucket/your_folder_named_a path/to/your/destination  From the docs:  --recursive (boolean) Command is performed on all files or objects under the specified directory or prefix.  EDIT: However, to do this with boto3 try this: import os import errno import boto3 client = boto3.client('s3') def assert_dir_exists(path): try: os.makedirs(path) except OSError as e: if e.errno != errno.EEXIST: raise def download_dir(bucket, path, target): # Handle missing / at end of prefix if not path.endswith('/'): path += '/' paginator = client.get_paginator('list_objects_v2') for result in paginator.paginate(Bucket=bucket, Prefix=path): # Download each file individually for key in result['Contents']: # Calculate relative path rel_path = key['Key'][len(path):] # Skip paths ending in / if not key['Key'].endswith('/'): local_file_path = os.path.join(target, rel_path) # Make sure directories exist local_file_dir = os.path.dirname(local_file_path) assert_dir_exists(local_file_dir) client.download_file(bucket, key['Key'], local_file_path) download_dir('your_bucket', 'your_folder', 'destination')
Question: I have written the following code to plot 6 pie charts in different subplots, but I get an error. This code works correctly if I use it to plot only 2 charts, but produces an an error for anything more than that. I have 6 categorical variables in my dataset, the names of which are stored in the list cat_cols. The charts are to be plotted from the training data train. CODE fig, axes = plt.subplots(2, 3, figsize=(24, 10)) for i, c in enumerate(cat_cols): train[c].value_counts()[::-1].plot(kind = 'pie', ax=axes[i], title=c, autopct='%.0f', fontsize=18) axes[i].set_ylabel('') plt.tight_layout()  ERROR AttributeError: 'numpy.ndarray' object has no attribute 'get_figure'  How do we rectify this?

Answer:  The issue is plt.subplots(2, 3, figsize=(24, 10)) creates two groups of 3 subplots, not one group of six subplots.  array([[<AxesSubplot:xlabel='radians'>, <AxesSubplot:xlabel='radians'>, <AxesSubplot:xlabel='radians'>], [<AxesSubplot:xlabel='radians'>, <AxesSubplot:xlabel='radians'>, <AxesSubplot:xlabel='radians'>]], dtype=object)   Unpack all of the subplot arrays from axes, using axes.ravel().  numpy.ravel, which returns a flattened array. A list comprehension will also work, axe = [sub for x in axes for sub in x] In practical terms, axes.ravel(), axes.flat, and axes.flatten(), can be used similarly. See What is the difference between flatten and ravel functions in numpy? & numpy difference between flat and ravel().   Assign each plot to one of the subplots in axe. How to resolve AttributeError: 'numpy.ndarray' object has no attribute 'get_figure' when plotting subplots is a similar issue.  import pandas as pd import numpy as np # sinusoidal sample data sample_length = range(1, 6+1) rads = np.arange(0, 2*np.pi, 0.01) data = np.array([np.sin(t*rads) for t in sample_length]) df = pd.DataFrame(data.T, index=pd.Series(rads.tolist(), name='radians'), columns=[f'freq: {i}x' for i in sample_length]) # crate the figure and axes fig, axes = plt.subplots(2, 3, figsize=(24, 10)) # unpack all the axes subplots axe = axes.ravel() # assign the plot to each subplot in axe for i, c in enumerate(df.columns): df[c].plot(ax=axe[i])
Question: My OS is ubuntu 20.04 and my default python is 3.8.2. I'm trying to create a virtual environment with pipenv and python 3.7. The following error occurs when I run pipenv install --python 3.7: Creating a virtualenv for this project… Using /usr/bin/python3.7m (3.7.0) to create virtualenv… ⠋RuntimeError: failed to query /usr/bin/python3.7m with code 1 err: 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py", line 16, in <module>\n from distutils.command.install import SCHEME_KEYS\nModuleNotFoundError: No module named \'distutils.command\'\n' Error while trying to remove the /home/yuhao/.local/share/virtualenvs/electrode-mimic-j_E-dTLW env: No such file or directory Virtualenv location: Warning: Your Pipfile requires python_version 3.7, but you are using None (/bin/python). $ pipenv check will surely fail. Creating a virtualenv for this project… Using /usr/bin/python3 (3.8.2) to create virtualenv… ⠙created virtual environment CPython3.8.2.final.0-64 in 162ms creator CPython3Posix(dest=/home/yuhao/.local/share/virtualenvs/electrode-mimic-j_E-dTLW, clear=False, global=False) seeder FromAppData(download=False, progress=latest, msgpack=latest, pytoml=latest, packaging=latest, setuptools=latest, contextlib2=latest, retrying=latest, pip=latest, pep517=latest, idna=latest, CacheControl=latest, appdirs=latest, requests=latest, pkg_resources=latest, webencodings=latest, distlib=latest, certifi=latest, distro=latest, ipaddr=latest, wheel=latest, six=latest, pyparsing=latest, urllib3=latest, chardet=latest, colorama=latest, lockfile=latest, html5lib=latest, via=copy, app_data_dir=/home/yuhao/.local/share/virtualenv/seed-app-data/v1.0.1.debian) activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator Virtualenv location: /home/yuhao/.local/share/virtualenvs/electrode-mimic-j_E-dTLW Pipfile.lock not found, creating… Locking [dev-packages] dependencies… ^C Aborted!  Why is this happening?

Answer: Did you install the python3-distutils package ? If not you can install it with :  sudo apt-get install python3-distutils  If you need it for a python3 version that is not the system default, specify the python 3 version : sudo apt-get install python3.X-distutils  example: python3.8-distutils
Question: I uploaded my django project on github and I have a lot of commits on my project. I don't want to delete my project and reupload it again. what is the easiest way to hide secret key after uploading project to github and after a lot of commits?

Answer: In the same directory where manage.py is, create a file whose name is .env, and put inside it: SECRET_KEY = '....your secret key ....' # --- the one indicated in your settings.py, cut an paste it here  where SECRET_KEY = '....your secret key ....' is the one indicated in your settings.py. So cut this line from your settings.py and paste it in the .env file. In the same directory, create a file whose name is .gitignore, and put inside it: .env  Then in your settings.py, where previously you had SECRET_KEY = '....your secret key ....', put: from decouple import config SECRET_KEY = config("SECRET_KEY") # this is to replace the secret key you cut away before  then in your command prompts run: pip install python-decouple pip freeze > requirements.txt  then add, commit and push on Github. Here you can find out more information on how .gitignore works.
Question: I run a query from AWS Athena console and takes 10s. The same query run from Sagemaker using PyAthena takes 155s. Is PyAthena slowing it down or is the data transfer from Athena to sagemaker so time consuming? What could I do to speed this up?

Answer: Just figure out a way of boosting the queries: Before I was trying: import pandas as pd from pyathena import connect conn = connect(s3_staging_dir=STAGIN_DIR, region_name=REGION) pd.read_sql(QUERY, conn) # takes 160s  Figured out that using a PandasCursor instead of a connection is way faster import pandas as pd pyathena import connect from pyathena.pandas.cursor import PandasCursor cursor = connect(s3_staging_dir=STAGIN_DIR, region_name=REGION, cursor_class=PandasCursor).cursor() df = cursor.execute(QUERY).as_pandas() # takes 12s  Ref: https://github.com/laughingman7743/PyAthena/issues/46
Question: $ sudo apt-get install python-pip Reading package lists... Done Building dependency tree Reading state information... Done Package python-pip is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source However the following packages replace it: python3-pip  both my pip and pip3 are installed in python 3 pip -V pip 20.1.1 from /usr/lib/python3/dist-packages/pip (python 3.8) pip3 -V pip 20.1.1 from /usr/lib/python3/dist-packages/pip (python 3.8)  Now i cant install pip ...it shows above error

Answer: If you have python(python2) installed you then you can use following command to install pip(for python2). curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py python get-pip.py  Now you can check for pip2 pip2 --version  I hope these will help you
Question: I have a dataframe df like below  NETWORK config_id APPLICABLE_DAYS Case Delivery 0 Grocery 5399 SUN 10 1 1 Grocery 5399 MON 20 2 2 Grocery 5399 TUE 30 3 3 Grocery 5399 WED 40 4  I want to implode( combine Applicable_days from multiple rows into single row like below) and get the average case and delivery per config_id  NETWORK config_id APPLICABLE_DAYS Avg_Cases Avg_Delivery 0 Grocery 5399 SUN,MON,TUE,WED 90 10  using the groupby on network,config_id i can get the avg_cases and avg_delivery like below. df.groupby(['network','config_id']).agg({'case':'mean','delivery':'mean'})  But How do i be able to join APPLICABLE_DAYS while performing this aggregation?

Answer: If you want the "opposite" of explode, then that means bringing it into a list in Solution #1. You can also join as a string in Solution #2: Use lambda x: x.tolist() for the 'APPLICABLE_DAYS' column within your .agg groupby function: df = (df.groupby(['NETWORK','config_id']) .agg({'APPLICABLE_DAYS': lambda x: x.tolist(),'Case':'mean','Delivery':'mean'}) .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1) .reset_index()) df Out[1]: NETWORK config_id APPLICABLE_DAYS Avg_Cases Avg_Delivery 0 Grocery 5399 [SUN, MON, TUE, WED] 25 2.5   Use lambda x: ",".join(x) for the 'APPLICABLE_DAYS' column within your .agg groupby function:  df = (df.groupby(['NETWORK','config_id']) .agg({'APPLICABLE_DAYS': lambda x: ",".join(x),'Case':'mean','Delivery':'mean'}) .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1) .reset_index()) df Out[1]: NETWORK config_id APPLICABLE_DAYS Avg_Cases Avg_Delivery 0 Grocery 5399 SUN,MON,TUE,WED 25 2.5  If you are looking for the sum, then you can just change mean to sum for the Cases and Delivery columns.
Question: I have a tfevent file already present on my Drive and I have successfully connected it to Google Colab. After searching within the issues of Tensorboard Github, I found that I had to set dom.serviceWorkers.enabled to True which I have done. But on Google Colab after performing the two steps:  %load_ext tensorboard %tensorboard --logdir path/to/logs  I am getting a Error 403 on the second step cell: I am using Firefox version 81.0.1 (64-bit) and my default mode is Private Window, so history and cache are cleared after I close all browser window. Can someone help me with this?

Answer: It seems that Tensorboard needs you to enable third party cookies to run without returning a HTTP 403 (Forbidden) error. I had the same issue using Chrome and fixed it by just allowing everything:  You can do the same on Firefox like so:  You could also find out which exact cookie is needed and then just allow that one.
Question: I a model as seen in the code below, but when trying to evaluate it or using earlystopping on it it gives me the following error:  numdigits = int(np.log10(self.target)) + 1 OverflowError: cannot convert float infinity to integer  I must state that without using .EarlyStopping or model.evaluate everything works well. I know that np.log10(0) gives -inf so that could be a potential cause, but why is there a 0 there in the first place and how can it be prevented? How can this problem be fixed? NOTES this is the code I use: import tensorflow as tf from tensorflow import keras TRAIN_PERCENT = 0.9 model = keras.Sequential([ keras.layers.Dense(128, input_shape=(100,), activation='relu'), keras.layers.Dense(128, activation='relu'), keras.layers.Dense(100) ]) earlystop_callback = keras.callbacks.EarlyStopping(min_delta=0.0001, patience=1 , monitor='accuracy' ) optimizer = keras.optimizers.Adam(lr=0.01) model.compile(optimizer=optimizer, loss="mse", metrics=['accuracy']) X_set, Y_set = some_get_data_function() sep = int(len(X_set)/TRAIN_PERCENT) X_train, Y_train = X_set[:sep], Y_set[:sep] X_test, Y_test = X_set[sep:], Y_set[sep:] model.fit(X_train, Y_train, batch_size=16, epochs=5, callbacks=[earlystop_callback]) ev = model.evaluate(X_test, Y_test) print(ev)  X,Y sets are np arrays. X is an array of arrays of 100 integers between 0 and 10. Y is an array of arrays of 100 integers, all of them are either 0 or 1.

Answer: Well it's hard to tell exactly as I can't run code without some_get_data_function() realization but recently I've got same error when mistakenly passed EMPTY array to model.evaluate. Taking into account that @meTchaikovsky comment solved your issue it's certainly due to messed up input arrays.
Question: Context: I am developping a simple Python application using a PySide2 GUI. It currently works fine in Windows, Linux and Mac. On Windows, I could use PyInstaller and InnoSetup to build a simple installer. Then I tried to do the same thing on Mac. It soon broke, because the system refused to start the command or the app generated by PyInstaller because it was not correctly signed. And as I am not an apple developper, I cannot sign anything... After some research, I tried py2app. I can go one step further here. With python setup.py py2app -A  I can create a runnable app. Which obviously cannot be ported to a different system because it uses my development folders. And if I use python setup.py py2app the generated program cannot start because py2app did not copy all the required Qt stuff. I tried to add one by one the missing libraries, but on the end the system could not find the plugins and I gave up... Question: Can someone help me with a recipe to convert a python script or package using a Qt GUI into a portable app on Mac? Ideally, the recipe should say how to use a custom application icon, but this is not required. References:  Python 3.8.5 macOS 10.15.7 Catalina PySide2 5.15.1 PyInstaller 4.0 py2app 0.22  As my real package is too large for a SO question I trimmed it down to a minimal reproducible example: from PySide3.QtWidgets import * import sys class MainWindow(QMainWindow): def __init__(self): super().__init__() hello = QLabel('Hello', self) hello.move(50, 50) def run(args): app = QApplication(args) main = MainWindow() main.show() sys.exit(app.exec_()) if __name__ == '__main__': run(sys.argv)  And here is the setup.py file used for py2app: from setuptools import setup APP = ['app.py'] DATA_FILES = [] OPTIONS = {} setup( app=APP, data_files=DATA_FILES, options={'py2app': OPTIONS}, setup_requires=['py2app'], ) 

Answer: Requirements  works with Python 3.8.5 macOS 10.15.7 Catalina uses PySide2 and py2app  Problems  PySide2 must be added under OPTIONS to the packages list when running the app then still an error occurs: Library not loaded: @rpath/libshiboken2.abi3.5.15.dylib, Reason: image not found  Solution The slightly modified setup.py could look like this: from setuptools import setup APP = ['app.py'] DATA_FILES = [] OPTIONS = { 'packages': ['PySide2'], 'iconfile': 'some_icon.icns', 'plist': { 'CFBundleDevelopmentRegion': 'English', 'CFBundleIdentifier': "com.ballesta.xxx", 'CFBundleVersion': "1.0.0", 'NSHumanReadableCopyright': u"Copyright © 2020, Serge Ballesta, All Rights Reserved" } } setup( app=APP, data_files=DATA_FILES, options={'py2app': OPTIONS}, setup_requires=['py2app'], )  Additionally, an icon definition and a few plist entries for some basic information have been added. The whole build is best triggered with a script that could look like this: #!/bin/sh python3 -m venv venv . venv/bin/activate pip install PySide2 pip install py2app python setup.py py2app cp ./venv/lib/python3.8/site-packages/shiboken2/libshiboken2.abi3.5.15.dylib ./dist/app.app/Contents/Resources/lib/python3.8/lib-dynload/shiboken2  Test Here the screenshot of a test run:
Question: I ran collectstatic a few weeks back and I would like to remove most (99.5%) of the collected files so that I do not have to store them when deploying to production. I tried collectstatic --clear but this removed them and then placed the deleted files back afterwards (not sure what the practical point of the command is. The docs state it is used to "to remove stale static files" but it certainly doesn't do that). Is there a way to erase the collected files? I've been using this page of the Django docs to try and find a solution but haven't had any luck.

Answer: It took an interesting combination of commands and actions to get the job done. After replacing my static folder with a new folder with only my desired contents, the base command of collectstatic --noinput --clear --no-post-process  got most of the files to be cleared and not re-copied. However, the js and css files were still being copied. To get the system to ignore these files and not copy them I used the --ignore tag for each JS and CSS filled-folder that was being copied over. For me personally, this additional command looked like this: --ignore facebook --ignore gis  In the end the full command pattern for another user would look something like this: collectstatic --noinput --clear --no-post-process --ignore [JSfolder0] --ignore [JSfolder1] --ignore [JSfolder2]
Question: How can I know whether tensorflow tensor is in cuda or cpu? Take this very simple example: import tensorflow as tf tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/device:GPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # print tensor a print(a) # Run on the GPU c = tf.matmul(a, b) print(c)  The code runs fine. Here, I am physically placing tensor 'a' and 'b' on the GPU. While printing 'a', I get: tf.Tensor( [[1. 2. 3.] [4. 5. 6.]], shape=(2, 3), dtype=float32)  It does not give any info whether 'a' in CPU or GPU. Now, suppose that there is an intermediate tensor like tensor 'c' which gets created during some operation. How can I know that tensor 'c' is a CPU or a GPU tensor? Also, suppose the tensor is placed on GPU. How can I move it to CPU?

Answer: As of Tensorflow 2.3 you can use .device property of a Tensor: import tensorflow as tf a = tf.constant([1, 2, 3]) print(a.device) # /job:localhost/replica:0/task:0/device:CPU:0  More detailed explanation can be found here
Question: How can i change the size of cells in a plotly express heatmap? I would need bigger cells import plotly.express as px fig1 = px.imshow(df[col],color_continuous_scale='Greens') fig1.layout.yaxis.type = 'category' fig1.layout.xaxis.type = 'category' fig1.layout.yaxis.tickmode = 'linear' fig1.layout.xaxis.tickmode = 'linear' fig1.layout.xaxis.tickangle = 65 fig1.layout.autosize = True fig1.layout.height = 500 fig1.layout.width = 500 fig1.show()  Result (very narrow) 

Answer: 'px' may not make it square due to the color bar, so why not use 'go'? import plotly.graph_objects as go fig = go.Figure(data=go.Heatmap( z=[[1, 20, 30], [20, 1, 60], [30, 60, 1]])) fig.show()   Set the graph size. fig.layout.height = 500 fig.layout.width = 500   Examples at px import plotly.express as px data=[[1, 25, 30, 50, 1], [20, 1, 60, 80, 30], [30, 60, 1, 5, 20]] fig = px.imshow(data, labels=dict(x="Day of Week", y="Time of Day", color="Productivity"), x=['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'], y=['Morning', 'Afternoon', 'Evening'] ) fig.update_xaxes(side="top") fig.layout.height = 500 fig.layout.width = 500 fig.show()
Question: I am using PyCharm for a project and I need to access pyc files. Unfortunately, IDE seems not to show *.pyc files nor __pycache__, even in searches with double Shift+Shift. I cannot find settings or documentation about it. Do you have any idea on how can I show these files and folders in IDE?

Answer: The setting you are looking for is Settings... | Editor | File Types | Ignore Files and Folders. Remove *.pyc and __pycache__ from the list to see the files.
Question: Is there a way to implement a validation in WTFforms that enforces the fact that at least one of the fields is required? For example, I have two StringFields and I want to make sure the user writes something in at least one of the fields before clicking on "submit". field1 = StringField('Field 1', validators=[???]) field2 = StringField('Field 2', validators=[???])  What should I write in place of the ???? InputRequired() in this case wouldn't do the job as I need to assign it to one of the fields or to both. How can I do that?

Answer: Override your form's validate method. For example: class CustomForm(FlaskForm): field1 = StringField('Field 1') field2 = StringField('Field 2') def validate(self, extra_validators=None): if super().validate(extra_validators): # your logic here e.g. if not (self.field1.data or self.field2.data): self.field1.errors.append('At least one field must have a value') return False else: return True return False  Note, you can still add individual validators to the fields e.g. input length.
Question: I have this dataframe: df = pd.DataFrame({'value':[1,2,3,4,2,42,12,21,21,424,34,12,42], 'type':['big','small','medium','big','big','big','big','medium','small','small','small','medium','small'], 'entity':['R','R','R','P','R','P','P','P','R','R','P','R','R']}) value type entity 0 1 big R 1 2 small R 2 3 medium R 3 4 big P 4 2 big R 5 42 big P 6 12 big P 7 21 medium P 8 21 small R 9 424 small R 10 34 small P 11 12 medium R 12 42 small R  The operation consists of grouping by column 'entity' doing a count operation based on a two logical conditions applied to a column 'value' and column 'type'. In my case, I have to count the values greater than 3 in the column 'name' and are not equal to 'medium' in the column 'type'. The result must be R=3 and P=4. After this, I must add the result to the original dataframe creating a new column named ‘Count’. I know this operation can be done in R with the next code: df[y!='medium' & value>3 , new_var:=.N,by=entity] df[is.na(new_var),new_var:=0,] df[,new_var:=max(new_var),by=entity]  In a previous task, I had to calculate only the values greater than 3 as condition. In that case, the result was R=3 and P=4 and I got it applying the next code: In []: df.groupby(['entity'])['value'].apply(lambda x: (x>3).sum()) Out[]: entity P 5 R 4 Name: value, dtype: int64 In []: DF=pd.DataFrame(DF) In []: DF.reset_index(inplace=True) In []: df.merge(DF,on=['entity'],how='inner') In []: df=df.rename(columns={'value_x':'value','value_y':'count'},inplace=True) Out[]: value type entity count 0 1 big R 4 1 2 small R 4 2 3 medium R 4 3 2 big R 4 4 21 small R 4 5 424 small R 4 6 12 medium R 4 7 42 small R 4 8 4 big P 5 9 42 big P 5 10 12 big P 5 11 21 medium P 5 12 34 small P 5  My questions are: How do I do it for the two conditions case? In fact, How do I do it for a general case with multiples different conditions?

Answer: Create mask by your conditions - here for greater by Series.gt with not equal by Series.ne chained by & for bitwise AND and then use GroupBy.transform for count Trues by sum: mask = df['value'].gt(3) & df['type'].ne('medium') df['count'] = mask.groupby(df['entity']).transform('sum')  Solution with helper column new: mask = df['value'].gt(3) & df['type'].ne('medium') df['count'] = df.assign(new = mask).groupby('entity')['new'].transform('sum')   print (df) value type entity count 0 1 big R 3 1 2 small R 3 2 3 medium R 3 3 4 big P 4 4 2 big R 3 5 42 big P 4 6 12 big P 4 7 21 medium P 4 8 21 small R 3 9 424 small R 3 10 34 small P 4 11 12 medium R 3 12 42 small R 3