eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: The basic idea involved: I am trying to make an application where students can write code related to a specific problem(say to check if the number is even) The code given by the student is then checked by the application by comparing the output given by the user's code with the correct output given by the correct code which is already present in the application. The basic version of the project I am working on: An application in which you can write a python script (in tkinter text box). The contents of the text box are first stored in a test_it.py file. This file is then imported (on the click of a button) by the application. The function present in test_it.py is then called to get the output of the code(by the user). The problem: Since I am "importing" the contents of test_it.py , therefore, during the runtime of the application the user can test his script only once. The reason is that python will import the test_it.py file only once. So even after saving the new script of the user in test_it.py , it wont be available to the application. The solution: Reload test_it.py every time when the button to test the script is clicked. The actual problem: While this works perfectly when I run the application from the script, this method fails to work for the compiled/executable version(.exe) of the file (which is expected since during compilation all the imported modules would be compiled too and so modifying them later will not work) The question: I want my test_it.py file to be reloaded even after compiling the application.  If you would like to see the working version of the application to test it yourself. You will find it here.

Answer: Even for the bundled application imports work the standard way. That means whenever an import is encountered, the interpreter will try to find the corresponding module. You can make your test_it.py module discoverable by appending the containing directory to sys.path. The import test_it should be dynamic, e.g. inside a function, so that it won't be discovered by PyInstaller (so that PyInstaller won't make an attempt to bundle it with the application). Consider the following example script, where the app data is stored inside a temporary directory which hosts the test_it.py module: import importlib import os import sys import tempfile def main(): with tempfile.TemporaryDirectory() as td: f_name = os.path.join(td, 'test_it.py') with open(f_name, 'w') as fh: # write the code fh.write('foo = 1') sys.path.append(td) # make available for import import test_it print(f'{test_it.foo=}') with open(f_name, 'w') as fh: # update the code fh.write('foo = 2') importlib.reload(test_it) print(f'{test_it.foo=}') main()
Question: I have the following step definitions, which result in an error because the @given fixture is not found, even though it is defined in target_fixture: import pytest from pytest_bdd import scenario, given, when, then, parsers from admin import Admin @scenario('../features/Admin.feature', 'register a new user') def test_admin(): pass @given('I\'m logged in as an admin at <host_name> with email <admin_email> and password <admin_password>', target_fixture="admin_login") def admin_login(host_name, admin_email, admin_password): admin = Admin(admin_email, admin_password) admin.login(host_name) # assert admin.status_code == 200 return admin @when('I call the register method for host <host_name> with email <user_email> and password <user_password> and firstName <first_name> and last name <last_name>') def test_register(admin_login, host_name, user_email, first_name, last_name): admin_login.register(host_name, user_email, first_name, last_name) assert admin_login.status_code == 200 @then('the user will be able to log in to <host_name> with email <user_email> and password <user_password>') def test_login(admin_login): print(admin_login) assert 3 == 3  This results in the error: platform darwin -- Python 3.8.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 rootdir: /Users/davidjoseph/work/ plugins: bdd-4.0.1 collected 3 items tests/step_defs/test_admin.py EEF [100%] ======================================================================== ERRORS ========================================================================= ____________________________________________________________ ERROR at setup of test_register ____________________________________________________________ file /Users/davidjoseph/work/tests/step_defs/test_admin.py, line 18 @when('I call the register method for host <host_name> with email <user_email> and password <user_password> and firstName <first_name> and last name <last_name>') def test_register(admin_login, host_name, user_email, first_name, last_name): E fixture 'admin_login' not found > available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestbdd_given_I'm logged in as an admin at <host_name> with email <admin_email> and password <admin_password>, pytestbdd_given_trace, pytestbdd_then_the user will be able to log in to <host_name> with email <user_email> and password <user_password>, pytestbdd_then_trace, pytestbdd_when_I call the register method for host <host_name> with email <user_email> and password <user_password> and firstName <first_name> and last name <last_name>, pytestbdd_when_trace, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory > use 'pytest --fixtures [testpath]' for help on them.  Can anyone tell me why admin_login is being recognized as a fixture?

Answer: Either downgrade to pytest-bdd<4 where this behaviour is still accepted, or rename the steps by removing the test_ prefix to prevent pytest from recognizing them as separate tests. @when(...) def register(admin_login, ...): ... @then(...) def login(admin_login): ...  should work.
Question: I'm doing a fresh install on a new Windows 10 laptop. I installed Python 3.9 and PyCharm Community 2020.2, then started a new project. In the project settings, I created a new project interpreter in a venv, inside the /venv folder. Everything looks to get set up correctly, but I can't install anything to the project interpreter. When I try to do so, e.g. when I try to install pandas or anything else, I get None-zero exit code (2) with the following message:  Usage: D:\MyProject\project\venv\Scripts\python.exe -m pip install [options] [package-index-options] ... D:\MyProject\project\venv\Scripts\python.exe -m pip install [options] -r [package-index-options] ... D:\MyProject\project\venv\Scripts\python.exe -m pip install [options] [-e] ... D:\MyProject\project\venv\Scripts\python.exe -m pip install [options] [-e] ... D:\MyProject\project\venv\Scripts\python.exe -m pip install [options] <archive url/path> ... no such option: --build-dir  When I go to the Terminal and just 'pip install pandas' per PyCharm's 'proposed solution', it installs fine, and pandas and its dependencies appear as usual in the list of installed modules in the interpreter. I've not encountered this before, and don't see anywhere in the settings where I can specify how exactly PyCharm will invoke pip in this situation.

Answer: PyCharm relies on --build-dir to install packages and the flag was removed in the latest pip 20.3. The fix for PyCharm is ready and will be released this week in 2020.3 release (and backported to 2020.2.5 and 2020.1.5). The workaround is to downgrade pip to the previous version - close PyCharm and run python -m pip install pip==20.2.4 in the terminal using the corresponding virtual environment. Update 1 2020.1.5 and 2020.2.5 with the fix were released - please update.
Question: This is my DataFrame in PySpark: utc_timestamp data feed 2015-10-13 11:00:00+00:00 1 A 2015-10-13 12:00:00+00:00 5 A 2015-10-13 13:00:00+00:00 6 A 2015-10-13 14:00:00+00:00 10 B 2015-10-13 15:00:00+00:00 11 B  The values of data are cumulative. I want to get this result (differences between consecutive rows, grouped by feed): utc_timestamp data feed 2015-10-13 11:00:00+00:00 1 A 2015-10-13 12:00:00+00:00 4 A 2015-10-13 13:00:00+00:00 1 A 2015-10-13 14:00:00+00:00 10 B 2015-10-13 15:00:00+00:00 1 B  In pandas I would do it this way: df["data"] -= (df.groupby("feed")["data"].shift(fill_value=0))  How can I do the same thing in PySpark?

Answer: You can use lag as a substitute for shift, and coalesce( , F.lit(0)) as a substitute for fill_value=0 from pyspark.sql.window import Window import pyspark.sql.functions as F window = Window.partitionBy("feed").orderBy("utc_timestamp") data = F.col("data") - F.coalesce(F.lag(F.col("data")).over(window), F.lit(0)) df.withColumn("data", data)
Question: I have deployed a Python 3.7 function in Google Cloud. I need to get the project-id through code to find out where it is deployed. I write a small Python 3.7 script and test it through Google shell command line import urllib import urllib.request url="http://metadata.google.internal/computeMetadata/v1/project/project-id" x=urllib.request.urlopen(url) with x as response: x.read()  Unfortunately this gives me only b'' as response. I am not getting the project id though I have set it using gcloud config set project my-project  I am new to Google Cloud and Python. Issue 2 This is an additional issue below: In my local system I have installed gcloud and if I run the above python3.7 script from there: x=urllib.request.urlopen(url) #this line  I am getting this exception from the line above: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1350, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1240, in request self._send_request(method, url, body, headers, encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1286, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1235, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1006, in _send_output self.send(msg) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 946, in send self.connect() File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 917, in connect self.sock = self._create_connection( File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/socket.py", line 787, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/socket.py", line 918, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno 8] nodename nor servname provided, or not known  The below changes as suggested by the answer also gives me empty string: 

Answer: In the Python 3.7 runtime, you can get the project ID via an environment variable: import os project_id = os.environ['GCP_PROJECT']  In future runtimes, this environment variable will be unavailable, and you'll need to get the project ID from the metadata server: import urllib.request url = "http://metadata.google.internal/computeMetadata/v1/project/project-id" req = urllib.request.Request(url) req.add_header("Metadata-Flavor", "Google") project_id = urllib.request.urlopen(req).read().decode()  Running this locally will produce an error because your local machine can't resolve the http://metadata.google.internal URL -- this is only available to deployed functions.
Question: I use sqlalchemy for Postgres DB. engine = create_engine( "postgresql://postgres:postgres@localhost/test" ...  Create engine without Postgres dialect(psycopg2, pg8000, or other). So my question: what is a default transaction isolation level? And what is a default Postgres dialect?

Answer: Per the docs, the default postgres driver used by SQLAlchemy is psycopg2. The default transaction isolation level is configured on the DB side, not by the client. Out-of-the-box, it is READ COMMITTED.
Question: I have a dataframe that has multiple entries for users. These users can also be assigned to multiple ID's. I would like to group by the users and then store a list of these ID's in another column as shown below: I'd like to go from this: df1 = pd.DataFrame({'USER': ['BOB','STEVE','PAUL','KEITH','STEVE','STEVE','BOB'],'ID':[1,2,3,4,5,6,7]})   To this. Only showing values if that user is attached to multiple ID's 

Answer: groupby + map u = df1.groupby("USER")["ID"].agg(list) df1["MULTI_IDS"] = df1["USER"].map(u[u.str.len().ge(2)])   USER ID MULTI_IDS 0 BOB 1 [1, 7] 1 STEVE 2 [2, 5, 6] 2 PAUL 3 NaN 3 KEITH 4 NaN 4 STEVE 5 [2, 5, 6] 5 STEVE 6 [2, 5, 6] 6 BOB 7 [1, 7]
Question: I want to limit CPU cores and threads. So I found three ways to limit these. 1) "Keras backend + Tensorflow" from keras import backend as K import tensorflow as tf config = tf.ConfigProto(intra_op_parallelism_threads=2, \ inter_op_parallelism_threads=4, \ allow_soft_placement=True, \ device_count = {'CPU': 1}) session = tf.Session(config=config) K.set_session(session)  2) "Keras from Tensorflow" import tensorflow as tf from tensorflow import keras tf.config.threading.set_intra_op_parallelism_threads(2) tf.config.threading.set_inter_op_parallelism_threads(4)  3) "keras from Tensorflow" import os os.environ['TF_NUM_INTRAOP_THREADS'] = '2' os.environ['TF_NUM_INTEROP_THREADS'] = '4'  These three ways are same affects? Lastly I understood for the parameters like I wrote below  intra_op_parallelism_threads("number of CPU cores") inter_op_parallelism_threads("number of threads")  is this right? If I miss-understanding please let me know. Thank you.

Answer: Not exactly, it's not as simple as that. As per official documentation - intra_op_parallelism_threads - Certain operations like matrix multiplication and reductions can utilize parallel threads for speedups. A value of 0 means the system picks an appropriate number. Refer this inter_op_parallelism_threads - Determines the number of parallel threads used by independent non-blocking operations. 0 means the system picks an appropriate number. Refer this So technically you can not limit the number of CPUs but only the number of parallel threads, which, for the sake of limiting resource consumption, is sufficient.  Regarding the methods, you are using - The third approach allows you to directly set the environment variables using os library. import os os.environ['TF_NUM_INTRAOP_THREADS'] = '2' os.environ['TF_NUM_INTEROP_THREADS'] = '4'  The second approach is a method in tf2 that does exactly the same (sets environment variables), the difference being that Keras is packaged into tf2 now. import tensorflow as tf from tensorflow import keras tf.config.threading.set_intra_op_parallelism_threads(2) tf.config.threading.set_inter_op_parallelism_threads(4)  The first approach is for standalone Keras. This approach will work if keras is set to tensorflow backend. Again, it does the same thing which is set environment variables indirectly. from keras import backend as K import tensorflow as tf config = tf.ConfigProto(intra_op_parallelism_threads=2, \ inter_op_parallelism_threads=4, \ allow_soft_placement=True, \ device_count = {'CPU': 1}) session = tf.Session(config=config) K.set_session(session)  If you still have doubts, you can check what happens to the environment variables after running all 3 independently and then check the specific variable using os with - print(os.environ.get('KEY_THAT_MIGHT_EXIST'))  For a better understanding of the topic, you can check this link that details it out quite well.  TLDR; You can use the second or third approach if you are working with tf2. Else use the first or third approach if you are using standalone Keras with tensorflow backend.
Question: Simplified dfs: df = pd.DataFrame( { "ID": [6, 2, 4], "to ignore": ["foo", "whatever", "idk"], "value": ["A", "B", "A"], } ) df2 = pd.DataFrame( { "ID_number": [1, 2, 3, 4, 5, 6], "A": [0.91, 0.42, 0.85, 0.84, 0.81, 0.88], "B": [0.11, 0.22, 0.45, 0.38, 0.01, 0.18], } ) ID to ignore value 0 6 foo A 1 2 whatever B 2 4 idk A A B ID_number 0 0.91 0.11 1 1 0.42 0.22 2 2 0.85 0.45 3 3 0.84 0.38 4 4 0.81 0.01 5 5 0.88 0.18 6  I want to add a column to df which includes combinations of df['ID'] to df2['ID_number'] and df['value'] to the df2 column matching the value in df[value] (either 'A' or 'B'). We can add a column of matching values where the lookup column name in df2 is given, 'A': df["NewCol"] = df["ID"].map( df2.drop_duplicates("ID_number").set_index("ID_number")["A"] )  Which gives:  ID to ignore value NewCol 0 6 foo A 0.88 1 2 whatever B 0.42 2 4 idk A 0.84  But this doesn't give values for B, so the value '0.42' above when looking for 'B' should instead be '0.22'. df["NewCol"] = df["ID"].map( df2.drop_duplicates("ID_number").set_index("ID_number")[df["value"]] )  obviously doesn't work. How can do I this?

Answer: You can set ID_number as index in df2,then use pd.Index.get_indexer here. df2 = df2.set_index('ID_number') r = df2.index.get_indexer(df['ID']) c = df2.columns.get_indexer(df['value']) df['new_col'] = df2.values[r, c] df ID to ignore value new_col 0 6 foo A 0.88 1 2 whatever B 0.22 2 4 idk A 0.84  Timeits Benchmarked using the below setup: Tested on Ubuntu 20.04.1 LTS(focal), Cpython3.8.5, Ipython shell(7.18.1), pandas(1.1.4), numpy(1.19.2) Setup df2 = pd.DataFrame( { "ID_number": np.arange(1, 1_000_000 + 1), "A": np.random.rand(1_000_000), "B": np.random.rand(1_000_000), } ) df = pd.DataFrame( { "ID": np.random.randint(1, 1_000_000, 50_000), "to ignore": ["anything"] * 50_000, "value": np.random.choice(["A", "B"], 50_000), } )  Resutls: @Vaishali In [57]: %%timeit ...: mapper = df2.set_index('ID_number').to_dict('index') ...: df['NewCol'] = df.apply(lambda x: mapper[x['ID']][x['value']], axis = ...: 1) ...: ...: 2.09 s ± 68.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) @Ch3steR In [58]: %%timeit ...: t = df2.set_index('ID_number') ...: r = t.index.get_indexer(df['ID']) ...: c = t.columns.get_indexer(df['value']) ...: df['new_col'] = df2.values[r, c] ...: ...: 49.7 ms ± 2.69 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) @Mayank In [59]: %%timeit ...: x = df2.set_index('ID_number').stack() ...: y = df.set_index(['ID', 'value']) ...: y['NewCol'] = y.index.to_series().map(x.to_dict()) ...: y.reset_index(inplace=True) ...: ...: 3.41 s ± 226 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) @Jezrael In [60]: %%timeit ...: df11 = (df2.melt('ID_number', value_name='NewCol', var_name='value') ...: .drop_duplicates(['ID_number','value']) ...: .rename(columns={'ID_number':'ID'})) ...: df.merge(df11, on=['ID','value'], how='left') ...: ...: 693 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: Im coding a discord bot and I got this error randomly. From what I can interpret, I am temporarily blocked from discord.py api, but what does the "exceeding rate limits part mean?"  discord.errors.HTTPException: 429 Too Many Requests (error code: 0): You are being blocked from accessing our API temporarily due to exceeding our rate limits frequently. Please read our docs at https://discord.com/developers/docs/topics/rate-limits to prevent this moving forward. 

Answer: Exceeding the rate limit means that the discord API is explicitly telling you that you cannot read any more data from their API for a given amount of time. Looking at their rate limit docs, the rate limit varies depending on the endpoint you're talking to:  The HTTP API implements a process for limiting and preventing excessive requests in accordance with RFC 6585. API users that regularly hit and ignore rate limits will have their API keys revoked, and be blocked from the platform. For more information on rate limiting of requests, please see the Rate Limits section.  To help, they conveniently return some information on where you stand with respect to their rate limiting: X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1470173023 X-RateLimit-Bucket: abcd1234 ...  If you're using the requests library you can easily check if you're close to exceeding the rate limit: req = requests.get("https://discord.com/api/path/to/the/endpoint") req.headers["X-RateLimit-Remaining"] # How many more requests you can make before `X-RateLimitReset`
Question: I know the canonical way to unpack a tuple is like this a, b, c = (1, 2, 3) # or (a,b,c) = (1, 2, 3)  but noticed that you can unpack a tuple like this [a, b, c] = (1, 2, 3)  Does the second method incur any extra cost due to some sort of cast or list construction? Is there a way to inspect how the python interpreter is dealing with this akin to looking at the assembly from a compiler?

Answer: No, those are all exactly equivalent. One way to look at this empirically is to use the dis dissasembler: >>> import dis >>> dis.dis("a, b, c = (1, 2, 3)") 1 0 LOAD_CONST 0 ((1, 2, 3)) 2 UNPACK_SEQUENCE 3 4 STORE_NAME 0 (a) 6 STORE_NAME 1 (b) 8 STORE_NAME 2 (c) 10 LOAD_CONST 1 (None) 12 RETURN_VALUE >>> dis.dis("(a, b, c) = (1, 2, 3)") 1 0 LOAD_CONST 0 ((1, 2, 3)) 2 UNPACK_SEQUENCE 3 4 STORE_NAME 0 (a) 6 STORE_NAME 1 (b) 8 STORE_NAME 2 (c) 10 LOAD_CONST 1 (None) 12 RETURN_VALUE >>> dis.dis("[a, b, c] = (1, 2, 3)") 1 0 LOAD_CONST 0 ((1, 2, 3)) 2 UNPACK_SEQUENCE 3 4 STORE_NAME 0 (a) 6 STORE_NAME 1 (b) 8 STORE_NAME 2 (c) 10 LOAD_CONST 1 (None) 12 RETURN_VALUE >>>  From the formal language specification, this is detailed here. This is part of the "target list", A relevant quote:  Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows....
Question: Being very new to FastAPI I am strugling to test slightly more difficult code than I saw in the tutorial. I use fastapi_cache module and Redis like this: from fastapi import Depends, FastAPI, Query, Request from fastapi_cache.backends.redis import CACHE_KEY, RedisCacheBackend from fastapi_cache import caches, close_caches app = FastAPI() def redis_cache(): return caches.get(CACHE_KEY) @app.get('/cache') async def test( cache: RedisCacheBackend = Depends(redis_cache), n: int = Query( ..., gt=-1 ) ): # code that uses redis cache @app.on_event('startup') async def on_startup() -> None: rc = RedisCacheBackend('redis://redis') caches.set(CACHE_KEY, rc) @app.on_event('shutdown') async def on_shutdown() -> None: await close_caches()  test_main.py looks like this: import pytest from httpx import AsyncClient from .main import app @pytest.mark.asyncio async def test_cache(): async with AsyncClient(app=app, base_url="http://test") as ac: response = await ac.get("/cache?n=150")  When I run pytest, it sets cache variable to None and test fails. I think I understand why the code is not working. But how do I fix it to test my caching properly?

Answer: The point is that httpx does not implement lifespan protocol and trigger startup event handlers. For this, you need to use LifespanManager. Install: pip install asgi_lifespan The code would be like so: import pytest from asgi_lifespan import LifespanManager from httpx import AsyncClient from .main import app @pytest.mark.asyncio async def test_cache(): async with LifespanManager(app): async with AsyncClient(app=app, base_url="http://localhost") as ac: response = await ac.get("/cache")  More info here: https://github.com/encode/httpx/issues/350
Question: I'm looking for a Python library for parsing code into its abstract syntax tree representation. There exists a built-in module, named ast, however, it is only designed for parsing Python code, to my understanding. I'm wondering if there is a similar Python library that suits the same purpose, but works with other programming languages. In particular, I'm looking for one that can parse JavaScript code. If one does not exist, any direction on how I could get started designing my own?

Answer: In general, when you need to parse code written in a language, it’s almost always better to use that language instead. For parsing JavaScript from Python, you may want to check out this module, which can be installed using pip and should work well enough.