eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to plot x1 and x2 against each other, and if class == 1, then marker should be +, if class == 0, then the marker should just be a dot. The file.csv is structured like this, the class column which is the conditional will only be either 1 or 0:    x1 x2 mark     1 2 0   9 4 1   0 5 1   2 6 0    Here's the code I have: df = pd.read_csv('file.csv') print(df) x1 = df['x1'].to_numpy() x2 = df['x2'].to_numpy() mark = df['class'].to_numpy() figure, ax = plt.subplots() for i in range(0,80,1): if mark[i] > 0: ax.plot(x1[i], color="red", marker='o') else: ax.plot(x1[i], color="blue", marker='x')  This is what the result SHOULD look like: 

Answer: I would plot it this way for all the data, if you're going to select a subset, consider using a new dataframe to select the subset: import matplotlib.pyplot as plt # the rows where class is 1 and plot them with '+' marker plt.scatter(df[df['class']==1]['x1'], df[df['class']==1]['x2'], marker='+') # the rows where class is 0 and plot them with 'o' marker plt.scatter(df[df['class']==0]['x1'], df[df['class']==0]['x2'], marker='o') plt.xlabel('x1') plt.ylabel('x2') plt.title('Scatter plot x1 vs x2') plt.show()
Question: When writing pytest tests for a function that is supposed to print something to the console, to verify the output string I am using the capsys fixture and capsys.readouterr(). This is the code I am currently using: @pytest.mark.parametrize( "values,expected", [ ([], "guessed so far: \n"), (["single one"], "guessed so far: single one\n"), (["one", "two", "three", "4"], "guessed so far: 4, three, two, one\n"), ], ) def test_print_guesses(capsys, values: list, expected: str) -> None: hm.print_guesses(values) assert capsys.readouterr().out == expected  I am also using the mypy extension in VS Code, so for now I am getting the warning: Function is missing a type annotation for one or more arguments  I'd like to get rid of that. What is the appropriate type annotation for the capsys argument?

Answer: Per the documentation for capsys, it:  Returns an instance of CaptureFixture[str].  This class indeed has a readouterr method (returning a CaptureResult, which has an out attribute). So your test should look like: @pytest.mark.parametrize( # ... ) def test_print_guesses(capsys: pytest.CaptureFixture[str], values: list, expected: str) -> None: hm.print_guesses(values) assert capsys.readouterr().out == expected
Question: If two fields exist, the corresponding fields are Boolean values.  x_field(bool value) y_field(bool value)  I want to generate all cases that can be represented as a combination of multiple Boolean values. For example, there are a total of 4 combinations that can be expressed by two Boolean fields as above.  x_field(true), y_field(true) x_field(true), y_field(false) x_field(false), y_field(true) x_field(false), y_field(false)  If you have an array with two fields that are Boolean type, can you generate all cases and express like this form? # before calculate ... fields = ["x_field", "y_field"] # after calculate ... result = [{"x_field": True, "y_field": True}, {"x_field": True, "y_field": False}, {"x_field": False, "y_field": True}, {"x_field": False, "y_field": False},]   My Attempted I thought I could solve this problem with the itertools module, but I wasn't sure which function to use. I tried to implement using various itertools module functions, but it failed. from itertools import combinations, permutations boolean_fields = ["x_field", "y_field"] # --> boolean_fields = [True, False, True, False] x = list(combination(boolean_fields, 2)) 

Answer: I'm surprised no one has proposed this simple one-liner yet... from itertools import product fields = ['x', 'y', 'z'] [dict(zip(fields, values)) for values in product([True,False], repeat=len(fields))]  Output: [{'x': True, 'y': True, 'z': True}, {'x': True, 'y': True, 'z': False}, {'x': True, 'y': False, 'z': True}, {'x': True, 'y': False, 'z': False}, {'x': False, 'y': True, 'z': True}, {'x': False, 'y': True, 'z': False}, {'x': False, 'y': False, 'z': True}, {'x': False, 'y': False, 'z': False}]  Explanation: product([True,False], repeat=len(fields)) creates all the 2**len(fields) combinations of True or False values. Then iterating over these combinations, zip(fields, values) pairs up every field name to its corresponding value. Finally turn every such pairing into a dict, and turn that into a list of dicts as you iterate over all the combinations (through list comprehension syntax).
Question: In Python I have a large xarray dataarray (da) of the shape In [1]: da.shape Out[1]: (744, 24, 30, 131, 215)  I need to perform the operation da = 10**da however, I'm running into a memory error  MemoryError: Unable to allocate 56.2 GiB for an array with shape (744, 24, 30, 131, 215) and data type float32  Is there a way to do the operation iteratively? I tried to manually loop but that gives the same error. for ii in range(len(da[:,0,0,0,0])): da[ii,:,:,:,:] = 10**da[ii,:,:,:,:]  It works fine with smaller arrays e.g., (24, 24, 30, 131, 215).

Answer: Chunking the dataarray did the trick. da = da.chunk({'dim0':50,'dim1': 24, 'dim2': 30, 'dim3': 131, 'dim4': 215}) da = 10**da
Question: Take a look at the following Route Handler @some_router.post("/some-route") async def handleRoute(something = Depends(lambda request : do_something(request, "some-argument"))): return JSONResponse(content = {"msg": "Route"}, status_code = 200)  So the dependency function do_something takes in a request and a string value. But, passing it in like so makes it so that it is recognized as a query parameter now even though it shouldn't. It is just an argument that is needed for the dependency function.  The do_something implementation async def do_something(request: Request, value: str):  pass This is obviously an example code. How can I pass in the request along with the string value into the do_something dependency function without making it recognized as a query parameter?  Rewrote Route Handler  Used default arguments but that's not how FastAPI's dependency injections works so no good  Maybe the use of lambda itself forces FastAPI to recognize it as a Query Parameter?  Tried setting a type for "request" as "Request" but then I get a syntax issue, tried wrapping it in parentheses and it still didn't work.  

Answer: My current solution to this is the following: def get_do_something(value: str): async def fixed_do_something(request: Request): return await do_something(request, roles) return fixed_do_something  Notice how instead of directly passing the request with your additional parameters, you are now using a wrapper function. This wrapper function handles the additional parameters and passes them to the nested function. Following this convention simplifies development and makes the process more efficient. @some_router.post("/some-route") async def handleRoute(something = Depends(get_do_something('some-value'))): return JSONResponse(content = {"msg": "Route"}, status_code = 200)
Question: I have a Pandas dataframe that looks like df = pd.DataFrame([['John', '1/1/2017','10'], ['John', '2/2/2017','15'], ['John', '2/2/2017','20'], ['John', '3/3/2017','30'], ['Sue', '1/1/2017','10'], ['Sue', '2/2/2017','15'], ['Sue', '3/2/2017','20'], ['Sue', '3/3/2017','7'], ['Sue', '4/4/2017','20']], columns=['Customer', 'Deposit_Date','DPD'])  And I want to create a new row called PreviousMean. This column is the year to date average of DPD for that customer. i.e. Includes all DPDs up to but not including rows that match the current deposit date. If no previous records existed then it's null or 0. So the desired outcome looks like  Customer Deposit_Date DPD PreviousMean 0 John 2017-01-01 10 NaN 1 John 2017-02-02 15 10.0 2 John 2017-02-02 20 10.0 3 John 2017-03-03 30 15.0 4 Sue 2017-01-01 10 NaN 5 Sue 2017-02-02 15 10.0 6 Sue 2017-03-02 20 12.5 7 Sue 2017-03-03 7 15.0 8 Sue 2017-04-04 20 13.0  And after some researching on the site and internet here is one solution: df['PreviousMean'] = df.apply( lambda x: df[(df.Customer == x.Customer) & (df.Deposit_Date < x.Deposit_Date)].DPD.mean(), axis=1)  And it works fine. However, my actual datafram is much larger (~1 million rows) and the above code is very slow. Is there any better way to do it? Thanks

Answer: You could use a custom groupby.apply with expanding.mean and a mask on the duplicated date to ffill the output: df['Deposit_Date'] = pd.to_datetime(df['Deposit_Date']) df['PreviousMean'] = (df.groupby('Customer') .apply(lambda s: s['DPD'].expanding().mean().shift() .mask(s['Deposit_Date'].duplicated()) .ffill(), include_groups=False) .droplevel(0) )  NB. this is assuming the dates are sorted. Output:  Customer Deposit_Date DPD PreviousMean 0 John 2017-01-01 10 NaN 1 John 2017-02-02 15 10.0 2 John 2017-02-02 20 10.0 3 John 2017-03-03 30 15.0 4 Sue 2017-01-01 10 NaN 5 Sue 2017-02-02 15 10.0 6 Sue 2017-03-02 20 12.5 7 Sue 2017-03-03 7 15.0 8 Sue 2017-04-04 20 13.0  Intermediates:  Customer Deposit_Date DPD expanding.mean shift duplicated mask PreviousMean 0 John 2017-01-01 10 10.00 NaN False NaN NaN 1 John 2017-02-02 15 12.50 10.0 False 10.0 10.0 2 John 2017-02-02 20 15.00 12.5 True NaN 10.0 3 John 2017-03-03 30 18.75 15.0 False 15.0 15.0 4 Sue 2017-01-01 10 10.00 NaN False NaN NaN 5 Sue 2017-02-02 15 12.50 10.0 False 10.0 10.0 6 Sue 2017-03-02 20 15.00 12.5 False 12.5 12.5 7 Sue 2017-03-03 7 13.00 15.0 False 15.0 15.0 8 Sue 2017-04-04 20 14.40 13.0 False 13.0 13.0  Note: the generalization of this answer to multiple groups is discussed here.
Question: I am dealing with Python objects containing Pandas DataFrame and Numpy Series objects. These can be large, several millions of rows. E.g.  @dataclass class MyWorld: # A lot of DataFrames with millions of rows samples: pd.DataFrame addresses: pd.DataFrame # etc.  I need to cache these objects, and I am hoping to find an efficient and painless way to serialise them, instead of standard pickle.dump(). Are there any specialised Python serialisers for such objects that would pickle Series data with some efficient codec and compression automatically? Alternatively, I need to hand construct several Parquet files, but that requires a lot of more manual code to deal with this, and I'd rather avoid that if possible. Performance here may mean  Speed File size (can be related, as you need to read less from the disk/network)  I am aware of joblib.dump() which does some magic for these kind of objects, but based on the documentation I am not sure if this is relevant anymore.

Answer: What about storing huge structures in parquet format while pickling it, this can be automated easily: import io from dataclasses import dataclass import pickle import numpy as np import pandas as pd @dataclass class MyWorld: array: np.ndarray series: pd.Series frame: pd.DataFrame @dataclass class MyWorldParquet: array: np.ndarray series: pd.Series frame: pd.DataFrame def __getstate__(self): for key, value in self.__annotations__.items(): if value is np.ndarray: self.__dict__[key] = pd.DataFrame({"_": self.__dict__[key]}) if value is pd.Series: self.__dict__[key] = self.__dict__[key].to_frame() stream = io.BytesIO() self.__dict__[key].to_parquet(stream) self.__dict__[key] = stream return self.__dict__ def __setstate__(self, data): self.__dict__.update(data) for key, value in self.__annotations__.items(): self.__dict__[key] = pd.read_parquet(self.__dict__[key]) if value is np.ndarray: self.__dict__[key] = self.__dict__[key]["_"].values if value is pd.Series: self.__dict__[key] = self.__dict__[key][self.__dict__[key].columns[0]]  Off course we will have some trade off between performance and volumetry as reducing the second requires format translation and compression. Lets create a toy dataset: N = 5_000_000 data = { "array": np.random.normal(size=N), "series": pd.Series(np.random.uniform(size=N), name="w"), "frame": pd.DataFrame({ "c": np.random.choice(["label-1", "label-2", "label-3"], size=N), "x": np.random.uniform(size=N), "y": np.random.normal(size=N) }) }  We can compare the parquet conversion trade off (about 300 ms more): %timeit -r 10 -n 1 pickle.dumps(MyWorld(**data)) # 1.57 s ± 162 ms per loop (mean ± std. dev. of 10 runs, 1 loop each) %timeit -r 10 -n 1 pickle.dumps(MyWorldParquet(**data)) # 1.9 s ± 71.3 ms per loop (mean ± std. dev. of 10 runs, 1 loop each)  And the volumetry gain (about 40 Mb spared): len(pickle.dumps(MyWorld(**data))) / 2 ** 20 # 200.28876972198486 len(pickle.dumps(MyWorldParquet(**data))) / 2 ** 20 # 159.13739013671875  Indeed those metrics will strongly depends on the actual dataset to be serialized.
Question: My database has about 600Mio entries that I want to query (Pandas is too slow). This local dbSNP only contains rsIDs and genomic positions. I used: import sqlite3 import gzip import csv rsid_db = sqlite3.connect('rsid.db') rsid_cursor = rsid_db.cursor() rsid_cursor.execute( """ CREATE TABLE rsids ( rsid TEXT, chrom TEXT, pos INTEGER, ref TEXT, alt TEXT ) """ ) with gzip.open('00-All.vcf.gz', 'rt') as vcf: # from https://ftp.ncbi.nih.gov/snp/organisms/human_9606/VCF/00-All.vcf.gz reader = csv.reader(vcf, delimiter="\t") i = 0 for row in reader: if not ''.join(row).startswith('#'): rsid_cursor.execute( f""" INSERT INTO rsids (rsid, chrom, pos, ref, alt) VALUES ('{row[2]}', '{row[0]}', '{row[1]}', '{row[3]}', '{row[4]}'); """ ) i += 1 if i % 1000000 == 0: print(f'{i} entries written') rsid_db.commit() rsid_db.commit() rsid_db.close()  I want to query multiple rsIDs and get their genomic position and alteration (query rsid and get chrom, pos, ref, alt and rsid). One entry looks like:    rsid chrom pos ref alt     rs537152180 1 4002401 G A,C    I query using: import sqlite3 import pandas as pd def query_rsid(rsid_list, rsid_db_path='rsid.db'): with sqlite3.connect(rsid_db_path) as rsid_db: rsid_cursor = rsid_db.cursor() rsid_cursor.execute( f""" SELECT * FROM rsids WHERE rsid IN ('{"', '".join(rsid_list)}'); """ ) query = rsid_cursor.fetchall() return query  It takes about 1.5 minutes no matter how many entries. Is there a way to speed this up?

Answer: Others have suggested defining your rsid column as the primary key, or alternatively creating a unique index on it. That's a good idea. Another thing: rsid IN ('dirty','great','list','of',items') may use a so-called skip-scan to get its results. If your rsid_list is very large, or if it pulls in values that are lexically widely separated, you may get a benefit from putting the items in the list into a temporary table then doing SELECT rsids.* FROM rsids JOIN temp_rsids_list ON rsids.rsid = temp_rsids_list.rsids  to get a more efficient lookup. I would declare the table like this: CREATE TABLE rsids ( rsid TEXT PRIMARY KEY COLLATE BINARY, chrom TEXT, pos INTEGER, ref TEXT, alt TEXT ) WITHOUT ROWID  COLLATE BINARY is the default. But still, it's helpful to show it because you know upfront you don't want case-insensitive matching on that column. This will remind your future self and your colleagues of that important optimization. WITHOUT ROWID tells SQLite to organize the table as a so-called "clustered index" where the other values are stored along with the easily searchable primary key. If you can make your primary key into an INTEGER that is a good idea for the sake of performance.
Question: This example shows that python calcuation with float numbers is faster than with integer numbers. I am wondering why calculation with integer is not faster than with float import time # Number of operations N = 10**7 # Integer addition start_time = time.time() int_result = 0 for i in range(N): int_result += 1 int_time = time.time() - start_time # Float addition start_time = time.time() float_result = 0.0 for i in range(N): float_result += 1.0 float_time = time.time() - start_time print(f"Time taken for integer addition: {int_time:.6f} seconds") print(f"Time taken for float addition: {float_time:.6f} seconds") 

Answer: Python uses integers with no fixed size. Under the hood it's an array of digits (not base-10 digits, but fixed size integers that fit into 1 array element, to which we refer as "digit" in this case) that can be extended if needed. That's why you can have absurdly large integers in Python. CPUs have instructions to perform floating point operations and integer operations, but they usually only work with numbers that are 1 machine word long or less. Python integers do not satisfy this condition, but floats do. Without diving too much into long number arithmetic, integers operations in Python are computed digit by digit. The longer the integers are, the longer the computation takes. In your code however, the ints do not use more than 1 digit. Still, these are arrays, that require some overhead to be processed.
Question: Where can I find the implementation of the default class representation in Python/REPL? Example: >>> class Foo: ... pass >>> Foo <class '__main__.Foo'>  Where is that whole string <class '__main__.Foo'> coming from? Is it defined in any PEP? I've tried greping the source of https://github.com/python/cpython for <class but it gives me no useful results, and greping for class is pointless.

Answer: The string <class '__main__.Foo'> comes from the type.__repr__ method, which is implemented with the type_repr function in Objects/typeobject.c of CPython: PyObject *result; if (mod != NULL && !_PyUnicode_Equal(mod, &_Py_ID(builtins))) { result = PyUnicode_FromFormat("<class '%U.%U'>", mod, name); }
Question: How can I annotate an ndarray so that Pyright/Mypy Intellisense can deduce the type of an entry? What can I fill in for ??? in x: ??? = np.array([1, 2, 3], dtype=int)  so that y = x[0]  is identified as an integer as rather than Any?

Answer: This is not currently possible. As defined in the type stubs, ndarray.__getitem__ has 5 @overloads. When the type checker evaluates x[0], it uses the second one, which returns Any: @overload def __getitem__(self, key: SupportsIndex | tuple[SupportsIndex, ...]) -> Any: ...  The other four all return ndarray/NDArray: @overload def __getitem__(self, key: ( NDArray[integer[Any]] | NDArray[np.bool] | tuple[NDArray[integer[Any]] | NDArray[np.bool], ...] )) -> ndarray[_Shape, _DType_co]: ... @overload def __getitem__(self, key: ( None | slice | EllipsisType | SupportsIndex | _ArrayLikeInt_co | tuple[None | slice | EllipsisType | _ArrayLikeInt_co | SupportsIndex, ...] )) -> ndarray[_Shape, _DType_co]: ... @overload def __getitem__(self: NDArray[void], key: str) -> NDArray[Any]: ... @overload def __getitem__(self: NDArray[void], key: list[str]) -> ndarray[_ShapeType_co, _dtype[void]]: ...  This means that, as long as x is determined to be of type ndarray, x[0] cannot be inferred as int. As for type checkers' support:  Pyright only reads information out of type stubs, so it's not possible to modify its behaviour. Mypy allows for plugins, and Numpy bundles its own Mypy plugin, but this plugin doesn't seem to have the inference capabilities you are looking for.  It is thus necessary to manually cast the values: from typing import cast y = cast(int, x[0]) reveal_type(y) # int
Question: It seems possible to execute this without using a loop, in python:  check = -arr[i] while check >= mi: check -= k  arr[i] and mi are constants at this point. I want to minimise the value of check such that it obeys the following: check = (x * k) - arr[i] for some x AND: check >= mi Everything here is an integer. Obviously using a loop here is very inefficient, so I am wondering how to do it in O(1) time. The while loop code is correct for all cases, just slow. I tried this, but it is wrong.  x = math.ceil((mi + arr[i]) / k) check = x * k - arr[i] 

Answer: The problem is the ">=". That gives your code an off-by-one. Plus, you have addition and subtraction swapped. We need to REDUCE -arr[i] by the x*k value. x = math.ceil((-arr[i]-mi+1)/k) check = -arr[i] - x * k  Consider an example, where arr[i] = -14, mi = 5, k = 3. If we do this by hand, we go 14, 11, 8, 5, 2. So, 4 steps. If arr[i] = -13, then there are only 3 steps: 13, 10, 7, 4. Test app: import math i = 0 def f1(arr,mi,k): check = -arr[i] while check >= mi: check -= k return check def f2(arr,mi,k): x = math.ceil((-arr[i]-mi+1)/k) check = -arr[i] - x * k return check print(f1([-14],5,3)) print(f2([-14],5,3)) print(f1([-13],5,3)) print(f2([-13],5,3))  Output: 2 2 4 4
Question: For my given multi-indexed DataFrame: df = pd.DataFrame( np.random.randn(12), index=[ [1,1,2,3,4,4,5,5,6,6,7,8], [1,2,1,1,1,2,1,2,1,2,2,2], ] )   0 1 1 1.667692 2 0.274428 2 1 0.216911 3 1 -0.513463 4 1 -0.642277 2 -2.563876 5 1 2.301943 2 1.455494 6 1 -1.539390 2 -1.344079 7 2 0.300735 8 2 0.089269  I would like to slice it such that I keep only rows where second index level contains BOTH 1 and 2  0 1 1 1.667692 2 0.274428 4 1 -0.642277 2 -2.563876 5 1 2.301943 2 1.455494 6 1 -1.539390 2 -1.344079  How can I do this?

Answer: Another possible solution, which is based on the following:  df.groupby(level=0) groups the dataframe by the first level of the index.  filter(lambda x: set(x.index.get_level_values(1)) == {1, 2}) checks if the second level of the index for each group contains both 1 and 2, and retains only the groups that meet this condition.   df.groupby(level=0).filter(lambda x: set(x.index.get_level_values(1)) == {1, 2})  Output:  0 1 1 -1.085631 2 0.997345 4 1 -0.578600 2 1.651437 5 1 -2.426679 2 -0.428913 6 1 1.265936 2 -0.866740
Question: How do I update a package on conda-forge? I'm trying to install the newest version of neuralprophet, but it is outdated on the conda-forge channel. There are pull requests on the feedstock github page, but how do I merge these? Can I merge these, or are special permissions needed? As a side note, I've also tried to get this package from the anaconda channel but the windows version is very outdated. As a last resort I could install the package via pip but this creates all kinds of issues.

Answer: You cannot merge the PRs unless you are a maintainer of the github repo. For situations where the package you need is not available or outdated on the anaconda repo, you can install all of the dependencies that are required using conda, and then install the package via PIP. This puts the package as the only part that is not managed by conda. Looking at the dependencies, you can install most of them via conda: # Windows conda install -c conda-forge ` "numpy>=1.25.0,<2.0.0" ` "pandas>=2.0.0" ` "pytorch>=2.0.0,<2.4.0" ` "pytorch-lightning>=2.0.0,<2.4.0" ` "tensorboard>=2.11.2" ` "torchmetrics>=1.0.0" ` "typing-extensions>=4.5.0" ` "holidays>=0.41,<1.0" ` "captum>=0.6.0" ` "matplotlib>=3.5.3" ` "plotly>=5.13.1" ` "python-kaleido==0.2.1"  # Linux conda install -c conda-forge \ "numpy>=1.25.0,<2.0.0" \ "pandas>=2.0.0" \ "pytorch>=2.0.0,<2.4.0" \ "pytorch-lightning>=2.0.0,<2.4.0" \ "tensorboard>=2.11.2" \ "torchmetrics>=1.0.0" \ "typing-extensions>=4.5.0" \ "holidays>=0.41,<1.0" \ "captum>=0.6.0" \ "matplotlib>=3.5.3" \ "plotly>=5.13.1" \ "python-kaleido==0.2.1"  And then install neuralprophet using pip: pip install neuralprophet
Question: I have data like this, df = pl.DataFrame({'a': ["['b', 'c', 'd']"]})  I want to convert the string to a list I use, df = df.with_columns(a=pl.col('a').str.json_decode())  it gives me, ComputeError: error inferring JSON: InternalError(TapeError) at character 1 (''')  then I use this function, import ast def safe_literal_eval(val): try: return ast.literal_eval(val) except (ValueError, SyntaxError): return val df = df.with_columns(a=pl.col('a').map_elements(safe_literal_eval, return_dtype=pl.List(pl.String)))  and get the expected output, but is there a pure polars way to achieve the same?

Answer: A general ast eval is not yet available. The problem with json_decode is that the list representation uses single quotes (instead of double quotes as used in JSON). In your example, this issue can be circumvented by replacing the single quotes using pl.Expr.str.replace_all as follows. df.with_columns( pl.col("a").str.replace_all("'", '"').str.json_decode() )  shape: (1, 1) ┌─────────────────┐ │ a │ │ --- │ │ list[str] │ ╞═════════════════╡ │ ["b", "c", "d"] │ └─────────────────┘