eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Python script that attempts to run a subprocess that runs the same interpreter as the currently-running interpreter. The subprocess's interpreter needs to be the same executable as the currently running interpreter's. Specifically, it needs to be the correct version even when multiple versions of Python are installed. Here are some solutions that almost work and the reasons they aren't quite complete:  Using sys.executable: The expected result of sys.executable differs from it's actual implementation. It relies on potentially-incorrect paths from sys.argv and the PATH environment variable. For example, if sys.argv == [''] and PATH contains a bin directory which has python3 in it, sys.executable will return the path to that python3 interpreter even if it is NOT the same as the current interpreter (IE it refers to a newer version than the current interpreter). Deducing the path from os.__file__: This approach works fine for Linux systems where the path to the interpreter can (typically) be safely deduced from this path (os.__file__ might be something like /usr/lib/python3.8/os.py so we may deduce that the correct interpreter is /usr/bin/python3.8). However, this deduction is only safe when the installation process for the current Python interpreter was "normal" (IE this does not work for embedded installations, custom Python builds installed to nonstandard directories, etc). In addition, I am unsure that it is possible to implement this deduction in a way that is OS-agnostic.  In theory, an ideal implementation would leverage OS-dependent utilities to deduce the path of the currently-running executable. This would likely mean reading /proc/self/exe or /proc/{pid}/exe on Linux or GetModuleFileName(GetCurrentModule()) on Windows. Is there an easier way to do this?

Answer: psutil package claims to be portable across several platforms Can be tested with a little recursive script import psutil import subprocess,os proc = psutil.Process().cmdline() print(f"{os.getpid()} {psutil.Process().exe()} {proc}") # set next command interpreter to the one found proc[0]=psutil.Process().exe() # run that command, forever :-p subprocess.run(proc)  Running it as python3.9 proc.py 2>/dev/null 13148 /usr/bin/python3.9 ['python3.9', 'proc.py'] 13149 /usr/bin/python3.9 ['/usr/bin/python3.9', 'proc.py'] 13150 /usr/bin/python3.9 ['/usr/bin/python3.9', 'proc.py'] 13151 /usr/bin/python3.9 ['/usr/bin/python3.9', 'proc.py'] 13152 /usr/bin/python3.9 ['/usr/bin/python3.9', 'proc.py'] ^C
Question: I have large datasets (ranging from 100k - 4 million rows) where I am looking for different relevant codes across multiple columns. For example, if I wanted to identify each row which has some start to a string '302' I would do: import polars as pl df = pl.DataFrame({ 'Codes_1': ['302E513', '301E513', '302E512'], 'Codes_2': ['303E513', '306E510', '302E512']}).lazy() conditions = ['302E513', '306E510'] column_names = ['Codes_1', 'Codes_2'] #create new column df = df.with_columns( pl.when(pl.any_horizontal( pl.col(column_names).str.starts_with(conditions[0]), pl.col(column_names).str.starts_with(conditions[1]))) .then(1.0) .otherwise(0.0) .alias('Column_name') )  It is really annoying when I am looking for say 4 codes instead of 2 to have to type in each of the codes to form my new column: import polars as pl df = pl.DataFrame({ 'Codes_1': ['302E513', '301E513', '302E512'], 'Codes_2': ['303E513', '306E510', '302E512']}).lazy() conditions = ['302E513', '306E510', '5164E23', '302E514'] column_names = ['Codes_1', 'Codes_2'] #create new column df = df.with_columns( pl.when(pl.any_horizontal( #Tedious part pl.col(column_names).str.starts_with(conditions[0]), pl.col(column_names).str.starts_with(conditions[1]), pl.col(column_names).str.starts_with(conditions[2]), pl.col(column_names).str.starts_with(conditions[3]) )) .then(1.0) .otherwise(0.0) .alias('Column_name') )  I know that this can be done with pandas by updating a mask with a for loop import pandas as pd df = pd.DataFrame({ 'Codes_1': ['302E513', '301E513', '302E512'], 'Codes_2': ['303E513', '306E510', '302E512']}) conditions = ['302E513', '306E510'] column_names = ['Codes_1', 'Codes_2'] #loop to create new column mask = False for code in conditions: mask |= df[column_names].eq(code).any(axis=1) df['Column_name'] = 0.0 df.loc[mask, 'Column_name'] = 1.0 print(df['Column_name'])  And I could change the number of conditions to any number and this code would execute. However, I would much rather use polars as it is faster and does not overflow the RAM on my machine for larger datasets. Any help is appreciated.

Answer: You could replace the multiple str.starts_with with a single regex and str.contains: df.with_columns( pl.when(pl.any_horizontal( pl.col(column_names).str.contains(f"^({'|'.join(conditions)})"), )) .then(1.0) .otherwise(0.0) .alias('Column_name') )  Or use a loop: df.with_columns( pl.when(pl.any_horizontal( pl.col(column_names).str.starts_with(c) for c in conditions )) .then(1.0) .otherwise(0.0) .alias('Column_name') )  Intermediate: # f"^({'|'.join(conditions)})" '^(302E513|306E510|5164E23|302E514)'  Output (non-lazy): ┌─────────┬─────────┬─────────────┐ │ Codes_1 ┆ Codes_2 ┆ Column_name │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 │ ╞═════════╪═════════╪═════════════╡ │ 302E513 ┆ 303E513 ┆ 1.0 │ │ 301E513 ┆ 306E510 ┆ 1.0 │ │ 302E512 ┆ 302E512 ┆ 0.0 │ └─────────┴─────────┴─────────────┘
Question: I have a dataframe: Email | Col1 | Col2 | Col3 | Name -------------------------------------------------------------------- john.cena@gmail.com | CellStr11 | 1.4 | CellStr13 | John Cena damian.doe@gmail.com | CellStr11 | 1.2 | CellStr13 | Matt Smith john.smith@gmail.com | CellStr21 | 1.2 | CellStr23 | John Cena  I need to aggregate the values of cells of Col1, Col2 and Col3 by Name. If row[Name] == row[Name] then: Email | Col1 | Col2 | Col3 | Name ------------------------------------------------------------------------------------------ john.cena@gmail.com | CellStr11 / CellStr21 | 1.4 / 1.2 | CellStr13 / CellStr23 | John Cena damian.doe@gmail.com | CellStr11 | 1.2 | CellStr13 | Matt Smith  I don't have to care about any other columns, their data can be lost. First I though to about this apporach:  Split the dataframe in such a way that I get the other Name matching row (learnt how to do that here Append the columns to the end of the original dataframe  The issue here was that not all rows had the corresponding row with matching Name, which would result in some rows having the additional columns, some not (correct me if I'm wrong, if this apporach is ok that would also be fine, even better). Is there a simple way to do it with df.apply and lambda function? Or any other more complex way? I could use: g2[['B', 'C']].apply(lambda x: x / x.sum()) but instead of summing just join as strings, but how can I differentiate between x.B and x.C?

Answer: If your dataframe is df: df['Col2'] = df['Col2'].astype('str') # all the columns must be strings gp_col = df.groupby(["Name"])[['Col1', 'Col2', 'Col3']] \ .agg(lambda x: " / ".join(x)) \ .reset_index()  display(gp_col) gives:
Question: I have a dataframe: Email | ... | Name -------------------------------------- john.cena@gmail.com | ... | John Cena john.smith@gmail.com | ... | John Cena  I need to find a row, that matches Name column with Email column when email_cell.split("@")[0] == name_cell.lower().replace(" ", ".")  I tried dataframe.loc[dataframe["Email"].str.contains(dataframe["Name"].replace(" ", "."))]  and other ways but I cannot use Series as if it was a string. Is there a way to do this?

Answer: Take email usernames by splitting on '@' and Convert names to lowercase and replaces spaces with dots df[df['Email'].str.split('@').str[0] == df['Name'].str.lower().str.replace(' ', '.')]  Output Matched rows: Email Name 0 john.cena@gmail.com John Cena 2 randy.orton@gmail.com Randy Orton  You can also try apply with lambda df[df.apply(lambda x: x['Email'].split('@')[0] == x['Name'].lower().replace(' ', '.'), axis=1)]
Question: How do I make a contourf plot where the areas are supposed to be discrete (integer array instead of float)? The values should discretely mapped to color indices. Instead matplotlib just scales the result across the whole set of colors. Example: import numpy as np from matplotlib import pyplot as plt axes = (np.linspace(-2, 2, 100), np.linspace(-2, 2, 100)) xx, yy = np.meshgrid(*axes, indexing="xy") fig, ax = plt.subplots() z = np.abs(xx * yy).astype(int) # values 0, 1, 2, 3, 4 z[z==0] = 4 ax.contourf(xx, yy, z, cmap="Set1")   

Answer: Now I got it :) Thanks @jared, pcolormesh was the right function, but I have to explicitly map the colors as the plotted variable: import numpy as np from matplotlib import pyplot as plt axes = (np.linspace(-2, 2, 100), np.linspace(-2, 2, 100)) xx, yy = np.meshgrid(*axes, indexing="xy") fig, ax = plt.subplots() z = np.abs(xx * yy).astype(int) # values 0, 1, 2, 3, 4 z[z==0] = 4 cmap = plt.get_cmap("Set1") z_color = cmap(z) # shape (100, 100, 4) with `z` as index ax.pcolormesh(xx, yy, z_color)
Question: I have the following PLSQL code which I am processing with Antlr4 in Python. I ma having trouble extracting the function parameter name and related details. CREATE OR REPLACE FUNCTION getcost ( p_prod_id IN VARCHAR2 , p_date IN DATE) RETURN number AS  The ParseTree output for this: ╚═ sql_script ╠═ unit_statement ║ ╚═ create_function_body ║ ╠═ "CREATE" (CREATE) ║ ╠═ "OR" (OR) ║ ╠═ "REPLACE" (REPLACE) ║ ╠═ "FUNCTION" (FUNCTION) ║ ╠═ function_name ║ ║ ╚═ identifier ║ ║ ╚═ id_expression ║ ║ ╚═ regular_id ║ ║ ╚═ "getcost" (REGULAR_ID) ║ ╠═ "(" (LEFT_PAREN) ║ ╠═ parameter ║ ║ ╠═ parameter_name ║ ║ ║ ╚═ identifier ║ ║ ║ ╚═ id_expression ║ ║ ║ ╚═ regular_id ║ ║ ║ ╚═ "p_prod_id" (REGULAR_ID) ║ ║ ╠═ "IN" (IN) ║ ║ ╚═ type_spec ║ ║ ╚═ datatype ║ ║ ╚═ native_datatype_element ║ ║ ╚═ "VARCHAR2" (VARCHAR2) ║ ╠═ "," (COMMA) ║ ╠═ parameter ║ ║ ╠═ parameter_name ║ ║ ║ ╚═ identifier ║ ║ ║ ╚═ id_expression ║ ║ ║ ╚═ regular_id ║ ║ ║ ╚═ "p_date" (REGULAR_ID) ║ ║ ╠═ "IN" (IN) ║ ║ ╚═ type_spec ║ ║ ╚═ datatype ║ ║ ╚═ native_datatype_element ║ ║ ╚═ "DATE" (DATE) ║ ╠═ ")" (RIGHT_PAREN) ║ ╠═ "RETURN" (RETURN) ║ ╠═ type_spec ║ ║ ╚═ datatype ║ ║ ╚═ native_datatype_element ║ ║ ╚═ "number" (NUMBER) ║  The python code is:  def enterParameter(self, ctx:PlSqlParser.ParameterContext): print(ctx.toStringTree(recog=parser)) param_name = None if ctx.parameter_name: if ctx.parameter_name.identifier() and ctx.parameter_name.identifier().id_expression(): param_name = ctx.parameter_name.identifier().id_expression().regular_id().getText() param_type = None if ctx.type_spec and ctx.type_spec.datatype: if ctx.type_spec.datatype.native_datatype_element(): param_type = ctx.type_spec.datatype.native_datatype_element().getText() default_value = None print(f"Parameter: {param_name}, Type: {param_type}, Def: {default_value}")  The output from the print statement is: (parameter (parameter_name (identifier (id_expression (regular_id p_prod_id)))) IN (type_spec (datatype (native_datatype_element VARCHAR2))))  But I get the following error:  if ctx.parameter_name.identifier() and ctx.parameter_name.identifier().id_expression(): AttributeError: 'function' object has no attribute 'identifier'  If I use ctx.getChild(0).getText() to get the parameter name it works, but I don't want to rely on hardcoded indices, and I also do not understand why this isn't working.

Answer: The parameter_name in the context is a fuction, beacuse if the context can contain more than one parameter the integer parameter defines the index of it. so please change you code to this: ctx.parameter_name().identifier()  And I wonder why your function has no return value, so for multiple parameters the param_name variable will always change. Of course, this could be intentional.
Question: Suppose I have urls, a list of s3 Parquet urls (on S3). I observe that this collect_all runs in O(urls). Is there a better way to parallelize this task? import polars as pl pl.collect_all(( pl.scan_parquet(url).filter(expr) for url in urls) )) 

Answer: Depending on what your expr is and what you're doing next, you might be better off with pl.concat([ pl.scan_parquet(url) for url in urls ]).filter(expr).collect()  One difference is that instead of getting a list of distinct dfs this one assumes you want them all combined into one and that they have the same schema. Another approach is to use asyncio import asyncio await asyncio.gather(*[pl.scan_parquet(url).filter(expr).collect_async() for url in urls])  I've see where the asyncio.gather approach is slightly faster than the alternative.
Question: I wanted to draw samples from a Dirichlet-multinomial distribution using SciPy. Unfortunately it seems that scipy.stats.dirichlet_multinomial does not define the rvs method that other distributions use to generate random samples. I think this would be equivalent to the following for a single sample: import scipy.stats as sps def dirichlet_multinomial_sample(alpha, n, **kwargs): kwargs['size'] = 1 # force size to 1 for simplicity p = sps.dirichlet.rvs( alpha=alpha, **kwargs ) return sps.multinomial.rvs( n=n, p=p.ravel(), **kwargs )  Multiple samples (i.e. size > 1) could be drawn similarly with a little bit more work to make it efficient. This seems easy enough to implement. My two questions are:  Is the above implementation correct? If it is, how can I suggest this enhancement to SciPy developers? 

Answer:  Is the above implementation correct?  This looks correct to me. Based on the discussion in the PR implementing multinomial, SciPy did implement a bit of code to generate samples from a multinomial Dirichlet, but the code is only part of a test, not a public API. One of the reviewers briefly touches on what you mention:  optional, probably follow-up PR add RVS method (as demonstrated in test_moments)  Looking up the code from the test-case they're referencing, here's what it's doing. https://github.com/scipy/scipy/pull/17211/files#diff-a998a313f078eba79aeb6347a65117e0a0c4542a4d778a4cfd398f1737380a71R3030  rng = np.random.default_rng(28469824356873456) n = rng.integers(1, 100) alpha = rng.random(size=5) * 10 dist = dirichlet_multinomial(alpha, n) # Generate a random sample from the distribution using NumPy m = 100000 p = rng.dirichlet(alpha, size=m) x = rng.multinomial(n, p, size=m)  That would appear to be essentially the same thing you're doing, only using the equivalent NumPy API rather than the SciPy API. See 1 2. Another sticking point that was discussed was whether this would require SciPy to bump the minimum version of NumPy.  The problem is that it will be slow and cumbersome without NumPy 1.22 (vectorization over multinomial shape parameters), but that should be the minimum supported version before the next release of SciPy, so it should be OK.  Since this message, SciPy has changed to a minimum version of 1.23.5, which means this is no longer a problem.  If it is, how can I suggest this enhancement to SciPy developers?  You can open an issue, and ask for them to fix this. You could also try fixing it yourself. If you do this, I would still recommend that you open an issue first. This will ensure that you and the maintainer are on the same page. If you decide to fix it yourself, I would recommend that you read the following things:  The contributor quickstart guide Running SciPy Tests Locally The implementation of another multivariate rvs(), such as multivariate_normal.rvs(). The comments in this thread which specifically mention rvs().
Question: Having a list of points (x, y), the function ConvexHull() of SciPy.spatial is great to calculate the points that form the hull. In my case, every point (x, y) also has a string as attribute. Is it possible to calculate the hull and returning its points including their respective attribute (x, y, str)? I checked both https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.ConvexHull.html as well as http://www.qhull.org/html/qh-optq.htm, but could not find a solution. Here, an example where I would like to run the commented lines with points_with_attribute instead of points_no_attribute: from scipy.spatial import ConvexHull points_no_attribute = [(0, 0), (2, 6), (6, 1), (2, 4), (3, 2)] points_with_attribute = [(0, 0, "point1"), (2, 6, "point2"), (6, 1, "point3"), (2, 4, "point4"), (3, 2, "point5")] hull = ConvexHull(points_no_attribute) hull_points = [points_no_attribute[i] for i in hull.vertices] # hull = ConvexHull(points_with_attribute) # hull_points = [points_with_attribute[i] for i in hull.vertices] print(hull_points)  Thanks for a hint!

Answer: I would suggest: hull = ConvexHull([(p[0], p[1]) for p in points_with_attribute]) hull_points = [points_with_attribute[i] for i in hull.vertices]  This has the advantage of avoiding an O(N^2) loop, by using the index of the points that has already been found.
Question: What puzzles me is that when running class Cat(pydantic.BaseModel): name: str age: int cats = [Cat(name="a", age=1), Cat(name="b", age=2)] df = pl.DataFrame({"cats": cats}) df = df.with_columns(pl.lit(0).alias("acq_num")) def wrap(batch): return Cat(name="c", age=3) df = df.group_by("acq_num").agg(pl.col("cats").map_batches(wrap, return_dtype=pl.Struct).alias("cats")) type(df["cats"][0][0]) # dict  the resulting entries are dicts, even though the function "wrap" returns a Cat. So polars automatically converts it to a dict, calling model_dump of pydantic? Changing to df = df.group_by("acq_num").agg(pl.col("cats").map_batches(wrap, return_dtype=pl.Object).alias("cats"))  results in the error: SchemaError: expected output type 'Object("object", None)', got 'Struct([Field { name: "name", dtype: String }, Field { name: "age", dtype: Int64 }])'; set `return_dtype` to the proper datatype  I am confused by this conversion happening. How can I prevent it?

Answer: Polars already infers the dtype at pl.DataFrame: cats = [Cat(name="a", age=1), Cat(name="b", age=2)] df = pl.DataFrame({"cats": cats})  df shape: (2, 1) ┌───────────┐ │ cats │ │ --- │ │ struct[2] │ ╞═══════════╡ │ {"a",1} │ │ {"b",2} │ └───────────┘  df.schema Schema([('cats', Struct({'name': String, 'age': Int64}))])  Rather than accessing model_dump, it looks at the dict storing the object's attributes and converts that into a struct: Cat(name="a", age=1).__dict__ {'name': 'a', 'age': 1}  To avoid this behaviour, set pl.Object for schema: df = pl.DataFrame({"cats": cats}, schema={'cats': pl.Object})  df shape: (2, 1) ┌────────────────┐ │ cats │ │ --- │ │ object │ ╞════════════════╡ │ name='a' age=1 │ │ name='b' age=2 │ └────────────────┘  type(df.item(0, 'cats')) __main__.Cat   Now, with map_batches, the expected output is a pl.Series (or a np.array, which it converts). So, here too you need to specify the dtype at construction: # import numpy as np def wrap_batch(batch): return pl.Series([Cat(name="c", age=3)], dtype=pl.Object) # return np.array([Cat(name="c", age=3)], dtype=object) will also work df = df.group_by("acq_num").agg( pl.col("cats").map_batches(wrap_batch, return_dtype=pl.Object).alias("cats") )  (Note that the above also works without return_dtype=pl.Object, "but [this] is considered a bug in the user's query".) type(df.explode('cats').item(0, 'cats')) __main__.Cat # you could add `returns_scalar=True` in this trivial example to avoid `explode`   With map_elements, you would only need to worry about the dtype in this case via return_dtype: def wrap_element(element): return Cat(name="c", age=3) df = df.group_by("acq_num").agg( pl.col("cats").map_elements(wrap_element, return_dtype=pl.Object) .alias("cats") )  type(df.item(0, 'cats')) __main__.Cat
Question: I'm using the autoflake tool to remove unused imports and variables in a Python file, but while it prints that unused imports/variables are detected, it doesn't actually remove them from the file. Here's the command I'm running: autoflake --in-place --remove-unused-variables portal/reports.py  Printed Output: portal/reports.py: Unused imports/variables detected  Despite the message indicating unused imports and variables, the file remains unchanged. I've confirmed that the file has unused imports and variables that should be removed. Versions:  Django: 5.0.7 autoflake: 2.3.1 Python: 3.11 Ubuntu: 24.04  I also noticed this issue: when I try to install autoflake after turning off .venv in my Django app folder or anywhere in the OS, I get the following error. error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed. See /usr/share/doc/python3.12/README.venv for more information. note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.  Has anyone encountered this issue? What could be causing it, and how can I fix it to properly remove unused imports and variables?

Answer: A similar issue was raised here and the OP was able to solve the issue by removing [tool.autoflake] check = true  in the pyproject.toml. See Remove unused imports not working in place.
Question: I am trying to use pyarrow to convert a csv to an apache arrow ipc with dictionary encoding turned on. The following appears to convert the csv to an arrow ipc file: file = "./in.csv" arrowFile = "./out.arrow" with pa.OSFile(arrowFile, 'wb') as arrow: with pa.csv.open_csv(file) as reader: with pa.RecordBatchFileWriter(arrow, reader.schema) as writer: for batch in reader: writer.write_batch(batch)  I tried the following to use dictionary encoding: convert_options = pa.csv.ConvertOptions(auto_dict_encode = True) with pa.OSFile(arrowFile, 'wb') as arrow: with pa.csv.open_csv(file, convert_options=convert_options) as reader: with pa.RecordBatchFileWriter(arrow, reader.schema) as writer: for batch in reader: writer.write_batch(batch)  But I get the following error: File "pyarrow/ipc.pxi", line 507, in pyarrow.lib._CRecordBatchWriter.write_batch File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Dictionary replacement detected when writing IPC file format. Arrow IPC files only support a single non-delta dictionary for a given field across all batches.  How do I fix the code to use dictionary encoding?

Answer: It looks like the IPC protocol only supports unified dictionary encoding. In your example each batch has got different dictionary encoding, which IPC doesn't support. You'd have to load the whole table and call unify_dicionaries. import pyarrow as pa schema = pa.schema([pa.field("col1", pa.dictionary(pa.int32(), pa.string()))]) table = pa.Table.from_batches( [ pa.record_batch({"col1": ["a", "b", "b"]}, schema), pa.record_batch({"col1": ["b", "b", "c"]}, schema), ], ) with pa.OSFile("data.arrow", "wb") as arrow: with pa.RecordBatchFileWriter(arrow, schema) as writer: for batch in table.unify_dictionaries().to_batches(): writer.write_batch(batch)