eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an SQL Alchemy engine where I try to insert parameters via sqlalchemy.sql.text to protect against SQL injection. The following code works, where I code variables for the condition and conditions values. from sqlalchemy import create_engine from sqlalchemy.sql import text db_engine = create_engine(...) db_engine.execute( text( 'SELECT * FROM table_name WHERE :condition_1 = :condition_1_value'), condition_1="name", condition_1_value="John" ) ).fetchall()  However, when I try to code the variable name for table_name, it returns an error. from sqlalchemy import create_engine from sqlalchemy.sql import text db_engine = create_engine(...) db_engine.execute( text( 'SELECT * FROM :table_name WHERE :condition_1 = :condition_1_value'), table_name="table_1", condition_1="name", condition_1_value="John" ) ).fetchall()  Any ideas why this does not work? EDIT: I know that it has something to do with the table_name not being a string, but I am not sure how to do it in another way.

Answer:  Any ideas why this does not work?  Query parameters are used to supply the values of things (usually column values), not the names of things (tables, columns, etc.). Every database I've seen works that way. So, despite the ubiquitous advice that dynamic SQL is a "Bad Thing", there are certain cases where it is simply necessary. This is one of them. table_name = "table_1" # NOTE: Do not use untrusted input here! stmt = text(f'SELECT * FROM "{table_name}" …')  Also, check the results you get from trying to parameterize a column name. You may not be getting what you expect. stmt = text("SELECT * FROM table_name WHERE :condition_1 = :condition_1_value") db_engine.execute(stmt, dict(condition_1="name", condition_1_value="John"))  will not produce the equivalent of SELECT * FROM table_name WHERE name = 'John'  It will render the equivalent of SELECT * FROM table_name WHERE 'name' = 'John'  and will not throw an error, but it will also return no rows because 'name' = 'John' will never be true.
Question: See update at bottom - question slightly changed I'm trying to download a file from s3 to a file-like object using boto3's .download_fileobj method, however when I try to inspect the downloaded bytestream, it's empty. I'm not sure what I'm doing wrong however: client = boto3.client('s3') data = io.BytesIO() client.download_fileobj(Bucket='mybucket', Key='myfile.wav', Fileobj=data) print(data.read())  Which yields an empty bytestring: b''  UPDATE : kINDA SOLVED. So it turns out that adding data.seek(0) after the download_fileobj line solves the problem. In light of this, I am now looking for an answer that explains what this snippet does and why it fixes the problem.

Answer: Thank you for this, I was running into the same problem. :o) The reason this works is that file buffer objects work with an internal pointer to the current spot to read from or write to. This is important when you pass the read() method a number of bytes to read, or to continually write to the next section of the file. As the client.download_fileobj() writes to the byte stream, or any operation writes to any stream, the pointer is positioned to the end of the last write. So you need to tell the file buffer object that you want to read from the beginning of what was just written.
Question: I have a dataframe organised like so:  x y e A 0 0.0 1.0 0.01 1 0.1 0.9 0.03 2 0.2 1.3 0.02 ... B 0 0.0 0.5 0.02 1 0.1 0.6 0.02 2 0.2 0.9 0.04 ...  etc. I would like to select rows of a A/B/etc. that fall between certain values in x. This, for example, works: p,q=0,1 indices=df.loc[("A"),"x"].between(p,q) df.loc[("A"),"y"][indices] Out: [1.0,0.9]  However, this takes two lines of code, and uses chain indexing. However, what is to me the obvious way of one-lining this doesn't work: p,q=0,1 df.loc[("A",df[("A"),"x"].between(p,q)),"y"] Out: [1.0,0.9]  How can I avoid chain indexing here? (Also, if anyone wants to explain how to make the "x" column into the indices and thereby avoid the '0,1,2' indices, feel free!) [Edited to clarify desired output]

Answer: You can merge your 2 lines of code by using a lambda function. >>> df.loc['A'].loc[lambda A: A['x'].between(p, q), 'y'] 1 0.9 2 1.3 Name: y, dtype: float64  The output of your code: indices=df.loc[("A"),"x"].between(p,q) output=df.loc[("A"),"y"][indices] print(output) # Output 1 0.9 2 1.3 Name: y, dtype: float64
Question: I couldn't find the answer for this question maybe someone could help me please? Is the order of execution defined in case of using two contexts like that?  with open('a.txt', 'w') as f1, open('b.txt', 'w') as f2: <some operation>  Am I guaranteed that the first context (here opening 'a.txt') will be executed before second (here opening 'b.txt')?

Answer: According to the language reference:  with A() as a, B() as b: SUITE  is semantically equivalent to: with A() as a: with B() as b: SUITE   So yes, since A() will execute before the body is executed.
Question: Suppose I have the df below. I would like to combine the price columns and value columns so that all prices are in one column and all volumes are in another column. I would also like a third column that identified the price level. For example, unit1, unit2 and unit3. import numpy as np import pandas as pd df = pd.DataFrame( { 'uid': ['U100', 'U200', 'E100', 'E200', 'E300', 'A100', 'A200', 'A300', 'A400', 'A500'], 'location': ['US', 'US', 'EU', 'EU', 'EU', 'Asia', 'Asia', 'Asia', 'Asia', 'Asia'], 'unit1_price': [10, 20, 15, 10, 10, 10, 20, 20, 25, 25], 'unit1_vol': [100, 150, 100, 200, 150, 150, 100, 200, 200, 200], 'unit2_price': [10, 25, 30, 20, 10, 10, 10, 10, 20, 20], 'unit2_vol': [200, 200, 150, 300, 300, 200, 150, 225, 225, 250], 'unit3_price': [0, 0, 0, 20, 20, 20, 20, 20, 20, 20], 'unit3_vol': [0, 0, 0, 500, 500, 500, 500, 500, 500, 500] } ) df   This is what the final df should look like:   I tried using melt and I think almost have the right answer. pd.melt(df, id_vars=['uid', 'location'], value_vars=['unit1_price', 'unit1_vol', 'unit2_price', 'unit2_vol', 'unit3_price', 'unit3_vol'])  This is what the partial df looks like with melt:  The problem with the above is that volume and price are in the same column but I want them to be in 2 separate columns. Did I use the right function?

Answer: You can form two dataframe using pd.melt first and combine it back to become one dataframe. df1 = df.melt(id_vars=['uid', 'location'], value_vars=['unit1_price', 'unit2_price', 'unit3_price'],var_name='unit',value_name='price') df2 = df.melt(id_vars=['uid', 'location'], value_vars=['unit1_vol', 'unit2_vol', 'unit3_vol'],var_name='unit', value_name="volume") ddf = pd.concat([df1,df2['volume']],axis=1).sort_values(by=['uid','unit'],ignore_index=True) ddf['unit']=ddf['unit'].str.split('_',expand=True)[0]
Question: How do i set xticks to 'a different interval' For instance: plt.plot(1/(np.arange(0.1,3,0.1)))  returns:  If I would like the x axis to be on a scale from 0 to 3, how can i do that? I've tried plt.xticks([0,1,2])  but that returns: 

Answer: You want to learn about plt.xlim and adjacent functions. This causes the X axis to have limits (minimum, maximum) that you specify. Otherwise Matplotlib decides for you based on the values you try to plot. y = 1 / np.arange(0.1,3,0.1) plt.plot(y) plt.xlim(0, 3) # minimum 0, maximum 3 plt.show()   Your plot uses only Y values, so the X values are automatically chosen to be 1, 2, 3, ... to pair up with each Y value you provide. If you desire to determine the X too, you can do that: x = np.arange(0.1,3,0.1) y = 1/x plt.plot(x, y) plt.xticks([0,1,2,3]) # ticks at those positions, if you don't like the automatic ones plt.show()
Question: A seemingly simple problem, but one that's proving a bit vexing. I have a list of chromosomes (there are 23 chromosome - chromosomes 1 to 21, then chromosome X and chromosome Y) like so: ['chr11','chr14','chr16','chr13','chr4','chr13','chr2','chr1','chr2','chr3','chr14','chrX',] I would like to sort this in the following order : ['chr1', 'chr2','chr2','chr3','chr4','chr11','chr13','chr13', 'chr14','chr14','chr16','chrX'] However, due to the lexicographical nature of python's sort it will sort chr1, chr10, chr11, chr12...chr2, etc. as I have chromosome X, sorting by their integer values also doesn't seem like an option. would I potentially have to specify a unique key by which to sort the list? Or is there some sort of obvious solution I'm missing.

Answer: You can use natsorted, what you want is natural sorting after all ;) l = ['chr11','chr14','chr16','chr13','chr4','chr13','chr2', 'chr1','chr2','chr3','chr14','chrX','chrY'] from natsort import natsorted out = natsorted(l)  output: ['chr1', 'chr2', 'chr2', 'chr3', 'chr4', 'chr11', 'chr13', 'chr13', 'chr14', 'chr14', 'chr16', 'chrX', 'chrY']
Question: I have a project based on django (3.2.10) and sentry-sdk (1.16.0) There is my sentry-init file: from os import getenv SENTRY_URL = getenv('SENTRY_URL') if SENTRY_URL: from sentry_sdk import init from sentry_sdk.integrations.django import DjangoIntegration from sentry_sdk.integrations.redis import RedisIntegration from sentry_sdk.integrations.celery import CeleryIntegration init( dsn=SENTRY_URL, integrations=[DjangoIntegration(), RedisIntegration(), CeleryIntegration()], traces_sample_rate=1.0, send_default_pii=True, debug=True, )  I have a CustomError inherited from Exception Every time I raise the CustomError sentry-sdk sends it to the dsn-url. I want to ignore some class of error or something like this. How can I do this?

Answer: You can pass a function that filters the errors to be sent: from os import getenv SENTRY_URL = getenv('SENTRY_URL') if SENTRY_URL: from sentry_sdk import init from sentry_sdk.integrations.django import DjangoIntegration from sentry_sdk.integrations.redis import RedisIntegration from sentry_sdk.integrations.celery import CeleryIntegration def before_send(event, hint): if 'exc_info' in hint: exc_type, exc_value, tb = hint['exc_info'] if isinstance(exc_value, CustomError): # Replace CustomError with your custom error return None return event init( dsn=SENTRY_URL, integrations=[DjangoIntegration(), RedisIntegration(), CeleryIntegration()], traces_sample_rate=1.0, send_default_pii=True, debug=True, before_send=before_send )  You can find more info in the documentation.
Question: I want to insert NaN at specific locations in A. However, there is an error. I attach the expected output. import numpy as np from numpy import NaN A = np.array([10, 20, 30, 40, 50, 60, 70]) C=[2,4] A=np.insert(A,C,NaN,axis=0) print("A =",[A])  The error is <module> A=np.insert(A,C,NaN,axis=0) File "<__array_function__ internals>", line 5, in insert File "C:\Users\USER\anaconda3\lib\site-packages\numpy\lib\function_base.py", line 4678, in insert new[tuple(slobj)] = values ValueError: cannot convert float NaN to integer  The expected output is [array([10, 20, NaN, 30, 40, NaN, 50, 60, 70])] 

Answer: Designate a type for your array of float32 (or float16, float64, etc. as appropriate) import numpy as np A = np.array([10, 20, 30, 40, 50, 60, 70], dtype=np.float32) C=[2,4] A=np.insert(A,C,np.NaN,axis=0) print("A =",[A])   A = [array([10., 20., nan, 30., 40., nan, 50., 60., 70.], dtype=float32)]
Question: I want to convert Python dictionary into DataFrame. Dictionary value is a List with different length. Example: import pandas as pd data = {'A': [1], 'B': [1,2], 'C': [1,2,3]} df = pd.DataFrame.from_dict(data)  But, the above code doesn't work with the following error: ValueError: All arrays must be of the same length  The output that I would like to get is as follows: name value 'A' [1] 'B' [1,2] 'C' [1,2,3] 

Answer: You can use: df = pd.DataFrame({'name':data.keys(), 'value':data.values()}) print (df) name value 0 A [1] 1 B [1, 2] 2 C [1, 2, 3]   df = pd.DataFrame(data.items(), columns=['name','value']) print (df) name value 0 A [1] 1 B [1, 2] 2 C [1, 2, 3]  Also working Series: s = pd.Series(data) print (s) A [1] B [1, 2] C [1, 2, 3] dtype: object
Question: How would I add 1 year to a column? I've tried using map and apply but I failed miserably. I also wonder why pl.date() accepts integers while it advertises that it only accepts str or pli.Expr. A small hack workaround is: col = pl.col('date').dt df = df.with_columns(pl.when(pl.col(column).is_not_null()) .then(pl.date(col.year() + 1, col.month(), col.day())) .otherwise(pl.date(col.year() + 1,col.month(), col.day())) .alias("date"))  but this won't work for months or days. I can't just add a number or I'll get a: > thread 'thread '<unnamed>' panicked at 'invalid or out-of-range date<unnamed>', ' panicked at '/github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.19/src/naive/date.rsinvalid or out-of-range date:', 173:/github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.19/src/naive/date.rs51 :note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace  Most likely because day and month cycle while year goes to infinity. I could also do this: df = df.with_columns( pl.when(col.month() == 1) .then(pl.date(col.year(), 2, col.day())) .when(col.month() == 2) .then(pl.date(col.year(), 3, col.day())) .when(col.month() == 3) .then(pl.date(col.year(), 4, col.day())) .when(col.month() == 4) .then(pl.date(col.year(), 5, col.day())) .when(col.month() == 5) .then(pl.date(col.year(), 6, col.day())) .when(col.month() == 6) .then(pl.date(col.year(), 7, col.day())) .when(col.month() == 7) .then(pl.date(col.year(), 8, col.day())) .when(col.month() == 8) .then(pl.date(col.year(), 9, col.day())) .when(col.month() == 9) .then(pl.date(col.year(), 10, col.day())) .when(col.month() == 10) .then(pl.date(col.year(), 11, col.day())) .when(col.month() == 11) .then(pl.date(col.year(), 12, col.day())) .otherwise(pl.date(col.year() + 1, 1, 1)) .alias("valid_from") ) 

Answer: Polars allows to do addition and subtraction with python's timedelta objects. However above week units things get a bit more complicated as we have to take different days of the month and leap years into account. For this polars has offset_by under the dt namespace. (pl.DataFrame({ "dates": pl.datetime_range(pl.datetime(2000, 1, 1), pl.datetime(2026, 1, 1), "1y", eager=True) }).with_columns( pl.col("dates").dt.offset_by("1y").alias("dates_and_1_yr") ))  shape: (27, 2) ┌─────────────────────┬─────────────────────┐ │ dates ┆ dates_and_1_yr │ │ --- ┆ --- │ │ datetime[μs] ┆ datetime[μs] │ ╞═════════════════════╪═════════════════════╡ │ 2000-01-01 00:00:00 ┆ 2001-01-01 00:00:00 │ │ 2001-01-01 00:00:00 ┆ 2002-01-01 00:00:00 │ │ 2002-01-01 00:00:00 ┆ 2003-01-01 00:00:00 │ │ 2003-01-01 00:00:00 ┆ 2004-01-01 00:00:00 │ │ 2004-01-01 00:00:00 ┆ 2005-01-01 00:00:00 │ │ … ┆ … │ │ 2022-01-01 00:00:00 ┆ 2023-01-01 00:00:00 │ │ 2023-01-01 00:00:00 ┆ 2024-01-01 00:00:00 │ │ 2024-01-01 00:00:00 ┆ 2025-01-01 00:00:00 │ │ 2025-01-01 00:00:00 ┆ 2026-01-01 00:00:00 │ │ 2026-01-01 00:00:00 ┆ 2027-01-01 00:00:00 │ └─────────────────────┴─────────────────────┘
Question: I am trying to create a requirements.txt to use pytorch but would like it to work on both GPU and non-GPU platforms. I do something like on my Linux GPU system: --find-links https://download.pytorch.org/whl/cu113/torch_stable.html torch==1.10.2+cu113 torchvision==0.11.3+cu113 pytorch-lightning==1.5.10  This works fine and the packages are installed and I can use the GPU-enabled pytorch. I wonder how I can modify this for the mac and non GPU users to install the non cuda package for torch and torchvision? Do I need to maintain separate requirements.txt files?

Answer: February 2024 update Check https://pytorch.org/. You will see that "CUDA is not available on MacOS, please use default package". However, you can still get performance boosts (this will depend on your hardware) by installing the MPS accelerated version of pytorch by: # MPS acceleration is available on MacOS 12.3+ pip3 install torch torchvision torchaudio  This command can be generated here: https://pytorch.org/  And in order to install different Torch versions on different platforms you can use conditionals in your requirements.txt like this # for CUDA 11.8 torch on Linux --index-url https://download.pytorch.org/whl/cu118; sys_platform == "linux" torch; sys_platform == "linux" torchvision; sys_platform == "linux" pytorch-lightning; sys_platform == "linux" # for MPS accelerated torch on Mac torch; sys_platform == "darwin" torchvision; sys_platform == "darwin" pytorch-lightning; sys_platform == "darwin"  This will install CUDA enabled torch and torchvision on Linux but the MPS accelerated version of them on MacOS
Question: Given this code: from abc import ABC, abstractmethod from typing import TYPE_CHECKING class Foo(ABC): @property @staticmethod @abstractmethod def name() -> str: pass class Bar(Foo): name = "bar" class Baz(Foo): name = "baz" instances = [Bar(), Baz()] print(instances[0].name) if TYPE_CHECKING: reveal_type(instances[0].name)  Ran by the python interpreter, it prints (as expected): bar  However, ran by mypy type checker, it prints (unexpectedly): main.py:23: note: Revealed type is "def () -> builtins.str"  It looks like the type is wrongly inferred (see playground). Is it possible to fix that somehow? If possible, I would like to fix it "once and for all" (like in the Foo class) because I'm accessing .name in multiple places.

Answer: Python 3.11 disallowed wrapping of @property using class decorators such as @classmethod and @staticmethod (see GH#89519). This means the provided snippet shouldn't be considered as valid Python code, and latest versions of Mypy actually warns about it: main.py:6: error: Only instance methods can be decorated with @property [misc] main.py:23: note: Revealed type is "def () -> builtins.str" Found 1 error in 1 file (checked 1 source file)  The easiest workaround is simply to use an instance method like so: class Foo(ABC): @property @abstractmethod def name(self) -> str: pass  See also: GH#13035.
Question: In Python, is it a good practice to rely on import to execute code, like in the example below? The code in mod.py is supposed to load some config, and needs to be executed once only. It can use more complex logic, but its purpose is to establish values of some parameters, later used as configuration by main.py. # --- mod.py --- param1 = 'abc' param2 = 'def' # ... # --- main.py --- import mod p1 = mod.param1 p2 = mod.param2 # (then calls functions from other components, which use p1, p2, ... as arguments) 

Answer: Defining things in an additional module is perfectly fine - variables, classes, functions etc. When the module is imported, as long as you don't use from ... import * your namespace does not get cluttered and you can extract a standalone and/or repeated fragments to have cleaner code. It's pretty much an intended use for modules. What is not so good, is having code with side-effects that gets executed on import. This here gives a nice example why it's not a good idea:  Say “no” to import side‐effects in Python
Question: I Have a main.py file for my Tkinter app. I export it to a standalone.exe with Pyinstaller. I want that when I start the .exe to update the .exe if a new version was deployed in the directory where I export my program. Apparently we can do that with PyUpdater but I didn't find how on StackOverflow.

Answer: If you're trying to push updates/patches to a frozen python program, read this explanation in the PyUpdater documentation. Instead of attempting make your app auto-update (i.e. push updates to frozen apps) a much more straightforward approach would be to use Inno Setup.
Question: I am developing Actor class and ray.wait() to collect the results. Below is the code and console outputs which is collecting the result for only 2 Actors when there are 3 Actors. import time import ray @ray.remote class Tester: def __init__(self, param): self.param = param def run(self): return self.param params = [0,1,2] testers = [] for p in params: tester = Tester.remote(p) testers.append(tester) runs = [] for i, tester in enumerate(testers): runs.append(tester.run.remote()) while len(runs): done_id, result_ids = ray.wait(runs) #runs size is not decreasing result = ray.get(done_id[0]) print('result:{}'.format(result)) time.sleep(1)  result:2 (pid=819202) (pid=819200) (pid=819198) result:1 result:0 result:0 result:0 result:0 result:0 ... ... ...  The console is printing out forever because the runs variable's size is not reduced. When I call ray.wait(runs) and get the done_id, runs's element with the done_id should be removed, but it is not removed. I want the console output to be like below. result:2 (pid=819202) (pid=819200) (pid=819198) result:1 result:0 

Answer: The script you provided is using ray.wait incorrectly. The following code does what you want: import time import ray @ray.remote class Tester: def __init__(self, param): self.param = param def run(self): return self.param params = [0, 1, 2] # I use list comprehensions instead of for loops for terseness. testers = [Tester.remote(p) for p in params] not_done_ids = [tester.run.remote() for tester in testers] # len() is not required to check that the list is empty. while not_done_ids: # Replace not_done_ids with the list of object references that aren't # ready. Store the list of object references that are ready in done_ids. # timeout=1 means sleep at most 1 second, do not sleep if there are # new object references that are ready. done_ids, not_done_ids = ray.wait(not_done_ids, timeout=1) # ray.get can take an iterable of object references. done_return_values = ray.get(done_ids) # Process each result. for result in done_return_values: print(f'result: {result}')  I added the following fixes:  ray.wait returns two lists, a list of objects that are ready, and a list of objects that may or may not be ready. You should iterate over the first list to get all object references that are ready. Your while loop goes forever until the list is empty. I simply replaced the runs list with not_done_ids so that once all object references are ready, the while loop breaks. ray.wait supports sleeping, with timeout. I removed your sleep and added timeout=1, which enables the program to run more efficiently (there is no sleep if another object is ready!).
Question: I'm running Ubuntu 20.04 and I want to start a project using Python 3.10. I used an install guide for Python 3.10 (this one), installed it using the deadsnakes PPA, and that was fine: $ python3.10 Python 3.10.5 (main, Jun 11 2022, 16:53:24) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>  I even switched my default python to 3.10 for good measure, using this, and that also works. $ python Python 3.10.5 (main, Jun 11 2022, 16:53:24) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>  Yet I cannot build a virtual environment: $ virtualenv myenv -p python3.10 RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.10'  And if I try to rely on the default, it gives me python3.8. I had been using python3.8 before alright, but I don't know where that setting is coming from. Pyenv is installed, I don't know if that's interfering nor not. $ virtualenv myenv created virtual environment CPython3.8.10.final.0-64 in 110ms creator CPython3Posix(dest=/home/jokea/FlorA/fl-scraper/myenv, clear=False, global=False) seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/jokea/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1) activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator  I just want to make a virtualenv with Python3.10. What could I be missing?

Answer: I was running into the same issue with a fresh python3.7 install. I managed to fix it by installing distutils. sudo apt-get install python3.7-distutils