eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here is the batch data set i created before to fit in the model: train_ds = tf.keras.preprocessing.image_dataset_from_directory( train_path, label_mode = 'categorical', #it is used for multiclass classification. It is one hot encoded labels for each class validation_split = 0.2, #percentage of dataset to be considered for validation subset = "training", #this subset is used for training seed = 1337, # seed is set so that same results are reproduced image_size = img_size, # shape of input images batch_size = batch_size, # This should match with model batch size ) valid_ds = tf.keras.preprocessing.image_dataset_from_directory( train_path, label_mode ='categorical', validation_split = 0.2, subset = "validation", #this subset is used for validation seed = 1337, image_size = img_size, batch_size = batch_size, )  if i run a for loop, i am able to access the img array and labels: for images, labels in train_ds: print(labels)  But if i try to access them like this: ATTEMPT 1) images, labels = train_ds  I get the following value error: ValueError: too many values to unpack (expected 2) ATTEMPT 2: If i try to unpack it like this: images = train_ds[:,0] # get the 0th column of all rows labels = train_ds[:,1] # get the 1st column of all rows  I get the following error: TypeError: 'BatchDataset' object is not subscriptable Is there a way for me to extract the labels and images without going trough a for loop?

Answer: Just unbatch your dataset and convert the data to lists: import tensorflow as tf import pathlib dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz" data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True) data_dir = pathlib.Path(data_dir) batch_size = 32 train_ds = tf.keras.utils.image_dataset_from_directory( data_dir, validation_split=0.2, subset="training", seed=123, batch_size=batch_size) train_ds = train_ds.unbatch() images = list(train_ds.map(lambda x, y: x)) labels = list(train_ds.map(lambda x, y: y)) print(len(labels)) print(len(images))  Found 3670 files belonging to 5 classes. Using 2936 files for training. 2936 2936
Question: I am trying to pull the week number given a date and then add that week number to the corresponding row in a pandas/python dataframe. When I run a for loop it is only storing the last calculated value instead of recording each value. I've tried .append but haven't been able to get anything to work. import datetime from datetime import date for i in df.index: week_number = date(df.year[i],df.month[i],df.day[i]).isocalendar() df['week'] = (week_number[1]) Expected values: day month year week 8 12 2021 49 19 12 2021 50 26 12 2021 51 Values I'm getting: day month year week 8 12 2021 51 19 12 2021 51 26 12 2021 51 

Answer: You can simply use Pandas .apply method to make it a one-liner: df["week"] = df.apply(lambda x: date(x.year, x.month, x.day).isocalendar()[1], axis=1)
Question: [ {"923390702359048212": 5}, {"462291477964259329": 1}, {"803390252265242634": 3}, {"824114065445486592": 2}, {"832041337968263178": 4} ]  This is a list of user ids that I just randomly made and some sample number that each id has. In this case lets call it a number of goals scored in a season in a video game. As I try to update the amount of goals scored at the end of the game, I have to go in a roundabout way which first gets all the member ids in the server, and then compares it to the data with the following code. amount = 0 for member in server: if member.id != server.member.id: amount = amount + 1 else: print(amount) print(str(member.id), str(server.member.id)) print(jsondata[amount][str(server.member.id)]) break jsondata[amount][str(server.member.id) = jsondata[amount][str(server.member.id)] + 1  Is there a better way to do what I am working on? I know I am going to run into problems eventually as I don't list members on the json until they score a goal and also I feel like I am wasting a ton of time with my program by having it check a lot of members (I edited out my real list and made this one to make it easier as nobody needs to see 50+ entries to get the idea). I am still a beginner when it comes to this so please respond in simple terms or leave links to places I can learn more complicated ideas. Thanks  def goalscored(): amount = 0 for member in server: if member.id != server.member.id: amount = amount + 1 else: print(amount) break with open('Goals.json','w') as f: jsondata = json.load(f) jsondata[amount][str(server.member.id) = jsondata[amount][str(server.member.id)] + 1 json.dump(jsondata,f) def firstgoal(server.member.id): with open('Goals.json','w') as f: jsondata = json.load(f) amount = 0 goalscored = 1 totalmembers = server.members.amount for member in server: if membed.id !=server.member.id: amount = amount + 1 if amount == totalmembers: NewScore = {user.id:goalscored} json.dump(NewScore,f)  Code that I'm using

Answer: Not sure why you couldn't get it work earlier, but storing it as dict would be much, much easier. # file.json { "923390702359048212": 5, "462291477964259329": 1, "803390252265242634": 3, "824114065445486592": 2, "832041337968263178": 4 } def goalscored(): with open("Goals.json", "r") as f: jsondata = json.load(f) # Make sure your id is not int but str if server.member.id not in jsondata: # first goal case jsondata[server.member.id] = 0 jsondata[server.member.id] += 1 with open("Goals.json", "w") as f: json.dump(jsondata, f)
Question: i try to use python requests library but i got this error i use psiphon VPN most of time in Windows 10 and got this below error after calling requests.get('[API URL]') Exception has occurred: SSLError HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)'))) During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: File "C:\Users\Hessam\Desktop\QWE.py", line 3, in <module> r = requests.get('https://api.github.com/user', auth=('user', 'pass')) 

Answer: You should try to add verify=False to your request: import requests r = requests.get('https://api.github.com/user', verify=False)  requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and requests will throw an SSLError if it’s unable to verify the certificate. In your specific case, you most likely have a problem with the SSL certificate on your VPN. Note that when verify is set to False, requests will accept any TLS certificate presented by the server, and will ignore hostname mismatches and/or expired certificates, which will make your application vulnerable to man-in-the-middle (MitM) attacks. Setting verify to False may be useful during local development or testing.
Question: I'm trying to get the correlation between a single column and the rest of the numerical columns of the dataframe, but I'm stuck. I'm trying with this: corr = IM['imdb_score'].corr(IM)  But I get the error  operands could not be broadcast together with shapes  which I assume is because I'm trying to find a correlation between a vector (my imdb_score column) with the dataframe of several columns. How can this be fixed?

Answer: The most efficient method it to use corrwith. Example: df.corrwith(df['A'])  Setup of example data: import numpy as np import pandas as pd df = pd.DataFrame(np.random.randint(10, size=(5, 5)), columns=list('ABCDE')) # A B C D E # 0 7 2 0 0 0 # 1 4 4 1 7 2 # 2 6 2 0 6 6 # 3 9 8 0 2 1 # 4 6 0 9 7 7  output: A 1.000000 B 0.526317 C -0.209734 D -0.720400 E -0.326986 dtype: float64
Question: I have a pandas multi index dataframe like shown below.  M EM A ... EA M0 EM0 Component EN EZ NZ EN EZ NZ EN EZ ... EZ NZ EN EZ NZ EN EZ NZ Date ... 2020-07-15 0.001682 0.000963 0.001292 0.000737 0.000635 0.000907 -0.048716 0.022769 ... 0.013103 0.016042 0.003619 0.001009 0.001718 0.000829 0.000685 0.000880 2020-07-16 0.000198 0.000178 -0.001219 0.000586 0.000548 0.000691 0.014514 0.009234 ... 0.010467 0.013240 0.000271 0.000238 -0.001365 0.000592 0.000541 0.000750 2020-07-17 0.000810 -0.000322 -0.000682 0.000445 0.000654 0.000595 0.045604 -0.014437 ... 0.014070 0.011086 0.000966 -0.000665 -0.000886 0.000680 0.000564 0.000584 2020-07-18 0.000287 -0.000887 -0.000329 0.000631 0.000815 0.000534 0.038145 -0.001408 ... 0.016091 0.010116 -0.000147 -0.000928 -0.000342 0.000760 0.000654 0.000519 2020-07-19 0.000805 0.000673 -0.001189 0.000537 0.000513 0.000462 0.051083 0.026574 ... 0.010684 0.009200 0.000746 0.001316 -0.001138 0.000896 0.000493 0.000470  I need to calculate the mean of EN, EZ, and NZ columns under M and EM to new columns (e.g. Mean) (also under M and EM, respectively), like shown below.  M EM Component EN EZ NZ Mean EN EZ NZ Mean Date 2020-07-15 0.001682 0.000963 0.001292 x.YYYYYY 0.000737 0.000635 0.000907 y.ZZZZZZ  Can someone show me a workaround for this? Thanks in advance!

Answer: Try: df_mean = pd.concat({'mean': df.groupby(level=0, axis=1).mean()}, axis=1).swaplevel(axis=1) df = df.join(df_mean).sort_index(level=0, axis=1) print(df) # Output: EM M Component EN EZ NZ mean EN EZ NZ mean Date 2020-07-15 4 5 6 5.0 1 2 3 2.0 2020-07-16 14 15 16 15.0 11 12 13 12.0  Setup to be reproducible: import io s = """\ ,M,M,M,EM,EM,EM Component,EN,EZ,NZ,EN,EZ,NZ Date,,,,,, 2020-07-15,1,2,3,4,5,6 2020-07-16,11,12,13,14,15,16 """ df = pd.read_csv(io.StringIO(s), header=[0, 1], index_col=0)  Update  I was wondering how may I calculate the median under EM instead of mean. I want to have only the mean for M and only median for EM  df1 = pd.concat([df['M'].mean(axis=1), df['EM'].median(axis=1)], keys=[('M', 'mean'), ('EM', 'median')]) df = df.join(df1.unstack('Date').T).sort_index(level=0, axis=1) print(df) # Output: EM M Component EN EZ NZ median EN EZ NZ mean Date 2020-07-15 4 6 6 6.0 1 3 3 2.333333 2020-07-16 14 17 16 16.0 11 11 13 11.666667
Question: Folium allow to create Markers with tooltip or popup text. I would like to do the same with my GeoJSON polygons. My GeoJSON has a property called "name" (feature.properties.name -> let's assume it is the name of each US state). I would like to be able to display this as a label in my choropleth map, in addition to the unemployment rate in each state. I also have the same information in the "State" column from the pandas dataframe. Is this possible? I would be happy with a solution that allows this to be a popup, tooltip or a simple text label written on top. import pandas as pd url = ( "https://raw.githubusercontent.com/python-visualization/folium/master/examples/data" ) state_geo = f"{url}/us-states.json" state_unemployment = f"{url}/US_Unemployment_Oct2012.csv" state_data = pd.read_csv(state_unemployment) m = folium.Map(location=[48, -102], zoom_start=3) folium.Choropleth( geo_data=state_geo, name="choropleth", data=state_data, columns=["State", "Unemployment"], key_on="feature.id", fill_color="YlGn", fill_opacity=0.7, line_opacity=0.2, legend_name="Unemployment Rate (%)", ).add_to(m) folium.LayerControl().add_to(m) m 

Answer: I've had to use folium's GeoJsonTooltip() and some other steps to get this done in the past. I'm curious to know if someone has a better way  Capture the return value of the Choropleth function Add a value(eg unemployment) to the Chorpleth's underlying geojson obj Create GeoJsonTooltip with that value from step 2 Add that tooltip to the choropleth's geojson  url = ( "https://raw.githubusercontent.com/python-visualization/folium/master/examples/data" ) state_geo = f"{url}/us-states.json" state_unemployment = f"{url}/US_Unemployment_Oct2012.csv" state_data = pd.read_csv(state_unemployment) m = folium.Map(location=[48, -102], zoom_start=3) # capturing the return of folium.Choropleth() cp = folium.Choropleth( geo_data=state_geo, name="choropleth", data=state_data, columns=["State", "Unemployment"], key_on="feature.id", fill_color="YlGn", fill_opacity=0.7, line_opacity=0.2, legend_name="Unemployment Rate (%)", ).add_to(m) # creating a state indexed version of the dataframe so we can lookup values state_data_indexed = state_data.set_index('State') # looping thru the geojson object and adding a new property(unemployment) # and assigning a value from our dataframe for s in cp.geojson.data['features']: s['properties']['unemployment'] = state_data_indexed.loc[s['id'], 'Unemployment'] # and finally adding a tooltip/hover to the choropleth's geojson folium.GeoJsonTooltip(['name', 'unemployment']).add_to(cp.geojson) folium.LayerControl().add_to(m) m
Question: Python just gave me weird advice: >>> import random >>> random.sample({1: 2, 3: 4, 5: 6}, 2) Traceback (most recent call last): File "<pyshell#11>", line 1, in <module> random.sample({1: 2, 3: 4, 5: 6}, 2) File "C:\Users\*****\AppData\Local\Programs\Python\Python310\lib\random.py", line 466, in sample raise TypeError("Population must be a sequence. For dicts or sets, use sorted(d).") TypeError: Population must be a sequence. For dicts or sets, use sorted(d).  Note the second part of the last error line. Why should I sort if I'm going to randomize a moment later anyway? Seems like wasting O(n log n) time.

Answer: From the commit history of cpython - My emphasis: github  In the future, the population must be a sequence. Instances of :class:set are no longer supported. The set must first be converted to a :class:list or :class:tuple, preferably in a deterministic order so that the sample is reproducible.  If you don't care about reproducibility sorting is not necessary.
Question: I want to generate a environments.yml file of an existing Python environment. I tried the following command: python env export --from-history -f environment.yml  This throws the following error: can't open file 'env': [Errno 2] No such file or directory  Note: This is not a conda environment.

Answer: pip freeze > requirements.txt to save the venv pip install -r requirements.txt to create venv.
Question: I've read in few places that generally, Python doesn't provide backward compatibility, which means that any newer version of Python may break code that worked fine for earlier versions. If so, what is my way as a developer to know what versions of Python can execute my code successfully? Is there any set of rules/guarantees regarding this? Or should I just tell my users: Just run this with Python 3.8 (for example) - no more no less...?

Answer: 99% of the time, if it works on Python 3.x, it'll work on 3.y where y >= x. Enabling warnings when running your code on the older version should pop DeprecationWarnings when you use a feature that's deprecated (and therefore likely to change/be removed in later Python versions). Aside from that, you can read the What's New docs for each version between the known good version and the later versions, in particular the Deprecated and Removed sections of each. Beyond that, the only solution is good unit and component tests (you are using those, right? 😉) that you rerun on newer releases to verify stuff still works & behavior doesn't change.
Question:  Please see the data in excel above. When use df.columns this is the printout: Index(['Country of Citizenship', '2015', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4', 'Unnamed: 5', 'Unnamed: 6', 'Unnamed: 7', 'Unnamed: 8', 'Unnamed: 9', ... 'Unnamed: 108', 'Unnamed: 109', 'Unnamed: 110', 'Unnamed: 111', 'Unnamed: 112', 'Unnamed: 113', 'Unnamed: 114', 'Unnamed: 115', 'Unnamed: 116', 'Unnamed: 117'], dtype='object', length=118)  I want to turn this multi-level column into a single level column with column name changed as 2015-Jan. Thanks a lot for help

Answer: You can read a excel file into a pandas dataframe with multi-indexes, like with the following example: import pandas as pd df = pd.read_excel('your_file.xlsx', header=[0,1,2], index_col=[0])  If you want to know how to navigate and use multi-indexes i recommend: this guide on indexes Alternatively there are also alot of Stackoverflow questions regarding Multiindex. Its hard to show without example data, but basically to flatten the multiindex you can do this: df.columns = df.columns.to_flat_index()
Question: We are using Quart (Flask+asyncio) Python web framework. Every time the request is processed and the response is sent to a client, this (or similar) message is logged:  WARNING:asyncio:Executing <Task pending name='Task-11' coro=<ASGIHTTPConnection.handle_request() running at /usr/local/lib/python3.8/site-packages/quart/asgi.py:102> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f742ba41ee0>()] created at /usr/local/lib/python3.8/asyncio/base_events.py:422> cb=[_wait.._on_completion() at /usr/local/lib/python3.8/asyncio/tasks.py:518] created at /usr/local/lib/python3.8/site-packages/quart/asgi.py:46> took 2.700 seconds  Since it is WARNING, we are kind of worried about what this could be. Does anyone have any idea why a log like this appears? Also, I have seen more logs starting <Task pending name... before. Does anyone know what these logs are?  To replicate a similar log message, it is enough just to do this: import time from quart import Quart app = Quart(__name__) @app.route('/', methods=['POST']) async def endpoint(): time.sleep(0.5) return '', 200  If I set sleep() to a lower value (e.g. 0.05), the log message is not printed out.

Answer: asyncio and other event loops require the tasks to yield control back to the event loop periodically so that it can switch to another task and execute tasks concurrently. This warning is indicating that a task is taking a long time between yields, thereby 'blocking' the event loop. It is likely this is happening as your code is either doing something CPU intensive, or more likely is using non-asyncio IO e.g. using requests. You should investigate this as it will degrade your servers ability to serve multiple requests concurrently.
Question: I'm trying to setup Graphene, but have a following exception raised when open http://localhost:8000/graphql/ in browser: TemplateDoesNotExist at /graphql/ graphene/graphiql.html Request Method: GET Request URL: http://localhost:8000/graphql/ Django Version: 3.2.10  Added did whole setup, added to urls, configured schema, queries and mutations. But still not work. And even don't remember that ever needed to configure templates for Graphene.

Answer: Looks like forgot to add following in settings.py, so it wasn't fully configured, at least for DEBUG mode: INSTALLED_APPS = [ # ... "graphene_django", # ... ]
Question: I have a small PySpark DataFrame df: index col1 0 1 1 3 2 4  And a dictionary: LOOKUP = {0: 2, 1: 5, 2: 5, 3: 4, 4: 6}  I now want to add an extra column col2 to df, equal to the LOOKUP values of col1. My output should look like this: index col1 col2 0 1 5 1 3 4 2 4 6  I tried using: df = df.withColumn(col("col2"), LOOKUP[col("col1")])  But this gave me errors, as well as using expr. How to achieve this in PySpark?

Answer: You can use a map column that you create from the lookup dictionary: from itertools import chain from pyspark.sql import functions as F lookup = {0: 2, 1: 5, 2: 5, 3: 4, 4: 6} lookup_map = F.create_map(*[F.lit(x) for x in chain(*lookup.items())]) df1 = df.withColumn("col2", lookup_map[F.col("col1")]) df1.show() #+-----+----+----+ #|index|col1|col2| #+-----+----+----+ #| 0| 1| 5| #| 1| 3| 4| #| 2| 4| 6| #+-----+----+----+  Another way would be to create a lookup_df from the dict then join with your dataframe
Question: from pyspark.sql.functions import isnan, when, count, sum , etc... It is very tiresome adding all of it. Is there a way to import all of it at once?

Answer: You can try to use from pyspark.sql.functions import *. This method may lead to namespace coverage, such as pyspark sum function covering python built-in sum function. Another insurance method: import pyspark.sql.functions as F, use method: F.sum.
Question: I set up my development environment on Fedora 35 and when I run any brownie command such as $ brownie console or even brownie --version I get the following error: Traceback (most recent call last): File "/home/philippbunke/.local/bin/brownie", line 5, in <module> from brownie._cli.__main__ import main File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/brownie/__init__.py", line 6, in <module> from brownie.project import compile_source, run File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/brownie/project/__init__.py", line 3, in <module> from .main import ( # NOQA 401 File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/brownie/project/main.py", line 44, in <module> from brownie.network import web3 File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/brownie/network/__init__.py", line 4, in <module> from .account import Accounts File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/brownie/network/account.py", line 12, in <module> import eth_account File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/eth_account/__init__.py", line 1, in <module> from eth_account.account import ( # noqa: F401 File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/eth_account/account.py", line 8, in <module> from cytoolz import ( File "/home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/cytoolz/__init__.py", line 3, in <module> from .functoolz import * ImportError: /home/philippbunke/.local/pipx/venvs/eth-brownie/lib64/python3.10/site-packages/cytoolz/functoolz.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send  Setup:  Python=3.10.1  Cython=0.29.26  gcc/gcc-c=11.2.1-7.fc35.x86_64  Ganache CLI=v6.12.2 $ pipx list venvs are in /home/philippbunke/.local/pipx/venvs apps are exposed on your $PATH at /home/philippbunke/.local/bin package eth-brownie 1.16.4, Python 3.10.1 - brownie  $ $PATH /home/philippbunke/.local/bin:/home/philippbunke/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin  I have spent a whole day trying to fix this error, please help me out!

Answer: The problem here seems to be Python 3.10.1! I used anaconda to create a new virtual environment with Python 3.8.12, installed brownie using pipx install --python python3.8 eth-brownie and it worked! The trick here was, to also tell pipx to use another python version, otherwise it would create a dependency to the global python version, which is python 3.10 in my case.
Question: What is exactly the 'right' way for type hinting? My IDE (and resulting code) works fine for type hints using either of below options, but some types can be imported from the typing module. Is there a preference for using the import from the typing module over builtins (like list or dict)? Examples: from typing import Dict def func_1(arg_one: Dict) -> Dict: pass  and def func_2(arg_one: dict) -> dict: pass 

Answer: The "right" way is to use builtins when possible (e.g. dict over typing.Dict). typing.Dict is only needed if you use Python < 3.9. In older versions you couldn't use generic annotations like dict[str, Any] with builtins, you had to use Dict[str, Any]. See PEP 585