eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am testing of running a Python thread in a c program with a simple example like the below # demo.py import time for i in range(1, 101): print(i) time.sleep(0.1)  // demo.c #include <Python.h> #include <pthread.h> #include <stdio.h> void *run_python_script(void *arg) { Py_Initialize(); if (!Py_IsInitialized()) { fprintf(stderr, "Python initialization failed\n"); return NULL; } FILE *fp = fopen("demo.py", "r"); if (fp == NULL) { fprintf(stderr, "Failed to open demo.py\n"); Py_Finalize(); return NULL; } PyRun_SimpleFile(fp, "demo.py"); fclose(fp); Py_Finalize(); return NULL; } int main() { pthread_t python_thread; if (pthread_create(&python_thread, NULL, run_python_script, NULL) != 0) { fprintf(stderr, "Failed to create thread\n"); return 1; } pthread_join(python_thread, NULL); printf("Python thread has finished. Exiting program.\n"); return 0; }  Then I build the above code with the following command gcc demo.c -o demo -lpthread -I$(python3-config --includes) $(python3-config --ldflags) $(python3-config --cflags)  Then I get the following error: /usr/bin/ld: /tmp/ccsHQpZ3.o: in function `run_python_script': demo.c:(.text.run_python_script+0x7): undefined reference to `Py_Initialize' /usr/bin/ld: demo.c:(.text.run_python_script+0xd): undefined reference to `Py_IsInitialized' /usr/bin/ld: demo.c:(.text.run_python_script+0x41): undefined reference to `PyRun_SimpleFileExFlags' /usr/bin/ld: demo.c:(.text.run_python_script+0x50): undefined reference to `Py_Finalize' /usr/bin/ld: demo.c:(.text.run_python_script+0xab): undefined reference to `Py_Finalize' collect2: error: ld returned 1 exit status  The python library do exists,  python3-config --ldflags -L/home/henry/anaconda3/lib/python3.9/config-3.9-x86_64-linux-gnu -L/home/henry/anaconda3/lib -lcrypt -lpthread -ldl -lutil -lm -lm ls -1 ~/anaconda3/lib | grep python libpython3.9.so libpython3.9.so.1.0 libpython3.so python3.9  I have no idea about is link error.

Answer: You need to pass --embed to python3-config because you are embedding a Python interpreter in your program. Observe the difference: $ python3-config --ldflags -L/usr/lib/python3.10/config-3.10-x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -lcrypt -ldl -lm -lm $ python3-config --embed --ldflags -L/usr/lib/python3.10/config-3.10-x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -lpython3.10 -lcrypt -ldl -lm -lm
Question: I'm working on a Snakemake workflow where I need to combine multiple CSV files into a single Pandas DataFrame. The list of CSV files is dynamic—it depends on upstream rules and wildcard patterns. Here's a simplified version of what I have in my Snakefile: rule combine_tables: input: expand("results/{sample}/data.csv", sample=SAMPLES) output: "results/combined/all_data.csv" run: import pandas as pd dfs = [pd.read_csv(f) for f in input] combined = pd.concat(dfs) combined.to_csv(output[0], index=False)  This works when the files exist, but I’d like to know: What's the best practice for handling missing or corrupt files in this context? Is there a more "Snakemake-idiomatic" way to dynamically list and read input files for Pandas operations? How do I ensure proper file ordering or handle metadata like sample names if not all CSVs are structured identically?

Answer: rule combine_tables: input: # Static sample list (use checkpoints if dynamically generated) expand("results/{sample}/data.csv", sample=SAMPLES) output: "results/combined/all_data.csv" run: import pandas as pd dfs = [] missing_files = [] corrupt_files = [] # Process files in consistent order for file_path in sorted(input, key=lambda x: x.split("/")[1]): # Sort by sample # Handle missing files (shouldn't occur if Snakemake workflow is correct) if not os.path.exists(file_path): missing_files.append(file_path) continue # Handle corrupt/unreadable files try: df = pd.read_csv(file_path) # Add sample metadata column sample_id = file_path.split("/")[1] df.insert(0, "sample", sample_id) # Add sample column at start dfs.append(df) except Exception as e: corrupt_files.append((file_path, str(e))) # Validation reporting if missing_files: raise FileNotFoundError(f"Missing {len(missing_files)} files: {missing_files}") if corrupt_files: raise ValueError(f"Corrupt files detected:\n" + "\n".join( [f"{f[0]}: {f[1]}" for f in corrupt_files])) if not dfs: raise ValueError("No valid dataframes to concatenate") # Concatenate and save combined = pd.concat(dfs, ignore_index=True) combined.to_csv(output[0], index=False)
Question: Question: Does scipy.optimize have minimizing functions that can divide their workload among multiple processes to save time? If so, where can I find the documentation? I've looked a fair amount online, including here, for answers:  Scipy's optimization incompatible with Multiprocessing? Parallel optimizations in SciPy Multiprocessing Scipy optimization in Python  I could be misunderstanding, but I don't see a clear indication in any of the above posts that the scipy library is informed of the fact that there are multiple processes that it can utilize simultaneously while also providing the minimization functions with all of the arguments needed to determine the minimum. I also don't see multiprocessing discussed in detail in the scipy docs that I read and I haven't had any luck finding real world examples of optimization gains to justify optimizing versus a parallel brute force effort. Here's a fictional example of what I'd like the scipy.optimize library to do (I know that the differential_evolution function doesn't have a multiprocessing argument): import multiprocessing as mp from scipy.optimize import differential_evolution def objective_function(x): return x[0] * 2 pool = mp.Pool(processes=16) # Perform differential evolution optimization result = differential_evolution(objective_function, multiprocessing = pool) 

Answer: With respect to scipy.optimize.differential_evolution, it does seem to offer multiprocessing through multiprocessing.Pool via the optional "workers" call parameter, according to the official documentation at https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution This may also be offered for other optimization methods but the API documents would need to be examined. The docs also say that the objective function must be pickleable. The official docs also have some general remarks on parallel execution with SciPy at https://docs.scipy.org/doc/scipy/tutorial/parallel_execution.html The call would look like this for differential_evolution: import multiprocessing as mp from scipy.optimize import differential_evolution def objective_function(x): return x[0] * 2 my_workers = 16 # Perform differential evolution optimization result = differential_evolution(objective_function, workers = my_workers)
Question: As we know input makes the inputs to string. How can I take matrix like this ([[1,2],[3,4]]) from input() by user and have it like a normal 2D list to do some thing on it. It should be like that data = input([[1,2],[3,4]]) print(data)  output : [[1,2],[3,4]] I tried this data = list(input()) but it was so wrong.

Answer: Using AST You can use ast Literals to parse your input string into a list. import ast raw_input = input("Enter the matrix (e.g., [[1,2],[3,4]]): ") # Parse the input string as a list matrix = ast.literal_eval(raw_input)  Using numpy In order to use numpy you would have to enter the matrix in a slightly different format: import numpy as np raw_input = input("Enter the matrix (e.g., 1,2;3,4): ") matrix = np.matrix(raw_input, dtype=int)
Question: I have a spacy nlp function nlp(<string>).vector that I need to apply to a string column in a dataframe. This function takes on average 13 milliseconds to return. The function returns a ndarray that contains 300 Float64s. I need to expand these Floats to their own columns. This is the sketchy way I've done this: import spacy import polars as pl nlp = spacy.load('en_core_web_lg') full = pl.LazyFrame([["apple", "banana", "orange"]], schema=['keyword']) VECTOR_FIELD_NAMES = ['dim_' + str(x) for x in range(300)] full = full.with_columns( pl.col('keyword').map_elements( lambda x: tuple(nlp(x).vector), return_dtype=pl.List(pl.Float64) ).list.to_struct(fields=VECTOR_FIELD_NAMES).struct.unnest() ) full.collect()  This takes 11.5s to complete, which is >100 times slower than doing the computation outside of Polars. Looking at the query plan, it reveals this: naive plan: (run LazyFrame.explain(optimized=True) to see the optimized plan) WITH_COLUMNS: [col("keyword").map_list().list.to_struct().struct.field_by_name(dim_0)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_1)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_2)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_3)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_4)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_5)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_6)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_7)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_8)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_9)(), col("keyword").map_list().list.to_struct().struct.field_by_name(dim_10)(), ...  It carries on like this for all 300 dims. I believe it might be computing nlp(<keyword>) for every cell of the output. Why might this be? How do I restructure my statements to avoid this?

Answer: It's due to how expression expansion works. The expression level unnest expands into multiple expressions (one for each field) pl.col("x").struct.unnest()  Would turn into pl.col("x").struct.field("a") pl.col("x").struct.field("b") pl.col("x").struct.field("c")  Normally you don't notice as Polars caches expressions (CSE), but UDFs are not eligible for caching.  https://github.com/pola-rs/polars/issues/20260  def udf(x): print("Hello") return x df = pl.DataFrame({"x": [[1, 2, 3], [4, 5, 6]]}) df.with_columns( pl.col.x.map_elements(udf, return_dtype=pl.List(pl.Int64)) .list.to_struct(fields=['a', 'b', 'c']) .struct.unnest() )  It calls the UDF for each element. Hello Hello Hello Hello Hello Hello  You can use the unnest frame method instead. df.with_columns( pl.col.x.map_elements(udf, return_dtype=pl.List(pl.Int64)) .list.to_struct(fields=['a', 'b', 'c']) .alias('y') ).unnest('y')  Hello Hello
Question: I used to have a working python function to fetch files from a specific Slack channel, but that stopped working a few months ago. I tested the same request to the slack API (files.list) using Postman which does give me an array with a number of files. The following code used to work but no longer does: import requests import json apiBase = "https://slack.com/api/" accesToken = "Bearer xoxb-<secret>" requestData = { "channel": "<obscured>" } r = requests.post(apiBase + "files.list", headers={'Authorization': accesToken, 'Content-Type': 'application/json; charset=utf-8'}, data=requestData) try: response = json.loads(r.text) except: print("Read error") isError = True if(not 'files' in response): if('error' in response): print(response['error']) if('warning' in response): print(response['warning']) isError = True files = response['files'] files.sort(key=lambda x:x['timestamp']) count = len(files) print(str(r)) print(str(r.request.body)) print(str(r.request.headers['Content-Type'])) print(str(r.text))  The result is: <Response [200]> channel=<secret> application/json; charset=utf-8 {"ok":true,"files":[],"paging":{"count":100,"total":0,"page":1,"pages":0}} Process finished with exit code 0  Postman also returns a 200 OK, but the array contains 3 files for this channel. So why is Python not getting the 3 files...? I know that an app needs to be given access to the channel in Slack which is the case here. (The channel and credentials are identical in both scenario's (Python and Postman). Please advise me ...

Answer: I think it has something to do with the content-type you send with the requests.post Have you tried using json=requestData  instead of data=requestData  Even though the content-type is correctly set in your headers the requests.post might still send "data" as a dictionary, this is maybe why the slack api is ignoring your data.  Update: The solution was to put the requestData into the url as a parameter, this can be elegantly done using the "params" argument of the requests.post() function like so: requestData = { "channel": channel_id } requests.post(params=requestData)  This is the way the Slack api expects the data.
Question: I do generate that figure with seaborn.distplot(). My problem is that the ticks on the X-axis do not fit to the bars, in all cases. I would expect a relationship between bars and ticks like you can see at 11 and 15.  This is the MWE import numpy as np import pandas as pd import seaborn as sns # Data np.random.seed(42) n = 5000 df = pd.DataFrame({ 'PERSON': np.random.randint(100000, 999999, n), 'Fruit': np.random.choice(['Banana', 'Strawberry'], n), 'Age': np.random.randint(9, 18, n) }) fig = sns.displot( data=df, x='Age', hue='Fruit', multiple='dodge').figure fig.show() 

Answer: You need discrete=True to tell seaborn that the x values are discrete. Adding shrink=0.8 will leave some space between the bars. import numpy as np import pandas as pd import seaborn as sns from matplotlib import pyplot as plt # Data np.random.seed(42) n = 5000 df = pd.DataFrame({ 'PERSON': np.random.randint(100000, 999999, n), 'Fruit': np.random.choice(['Banana', 'Strawberry'], n), 'Age': np.random.randint(9, 18, n) }) sns.displot( data=df, x='Age', hue='Fruit', multiple='dodge', discrete=True, shrink=0.8) plt.show()  . Note that sns.displot() is a figure-level function that creates a grid of one or more subplots, with a common legend outside. sns.countplot() is an axes-level function, that creates a single subplot with a legend inside. An alternative is creating a countplot: sns.countplot( data=df, x='Age', hue='Fruit', dodge=True )
Question: I have a short example python script that I'm calling glbltest.py: a = [] def fun(): global a a = [20,30,40] print("before ",a) fun() print("after ",a)  If I run it from the command line, I get what I expect: $ python glbltest.py before [] after [20, 30, 40]  I open a python shell and run it by importing, and I get basically the same thing: >>> from glbltest import * before [] after [20, 30, 4]  So far so good. Now I comment out those last three lines and do everything "by hand": >>> from glbltest import * >>> a [] >>> fun() # I run fun() myself >>> a # I look at a again. Surely I will get the same result as before! [] # No! I don't!  What is the difference between fun() being run "automatically" by the importing of the script, and me running fun() "by hand"?

Answer: global a refers to the name a in the glbltest module's namespace. When you set a by hand, it refers to the name a in the __main__ module's namespace. When you use from glbltest import * the names in the module are imported into the __main__ module's namespace. Those are different names but refer to the same objects. When you use global a and a = [20,30,40] in the glbltest module, assignment makes a new object that a in glbltest module's namespace now refers to. The name a in the __main__ module still refers to the original object (the empty list). As a simple example, print the id() of a in the fun() function, and print(id(a)) "by hand" after you set it: a = [] def fun(): global a print(a, id(a)) a = [20,30,40] print(a, id(a)) # To view the global a object id again def show(): print(a, id(a))  "by hand", with comments: >>> from glbltest import * >>> a # the imported name [] >>> id(a) # its object ID 2056911113280 >>> fun() [] 2056911113280 # start of fun() the object is the same ID [20, 30, 40] 2056902829312 # but assignment changes to new object (different ID) >>> a [] # main a still refers to original object >>> id(a) 2056911113280 >>> show() # glbltest module still sees *its* global a [20, 30, 40] 2056902829312  Note that if you use mutation vs. assignment to change the existing object. You'll see the change: a = [] def fun(): global a print(a, id(a)) a.extend([20,30,40]) # modify existing object, not assigning a new object. print(a, id(a)) # To view the global a object id again def show(): print(a, id(a))  Now the object IDs remain the same. >>> from glbltest import * >>> a, id(a) # import object ([], 1408887112064) >>> fun() [] 1408887112064 # before change still the same object [20, 30, 40] 1408887112064 # mutated the *existing* list >>> a, id(a) ([20, 30, 40], 1408887112064) # main's 'a' refers to the same object, same ID >>> show() [20, 30, 40] 1408887112064 # glbltest refers to the same object, same ID  It's a bit more obvious that the names are different if you just import the module and the module's a can be referred to directly as glbltest.a. a = [] def fun(): global a a = [20,30,40]  >>> import glbltest >>> glbltest.a [] >>> a = 5 # main's a >>> a 5 >>> glbltest.a # module's a [] >>> glbltest.fun() >>> a # main's a doesn't change 5 >>> glbltest.a # module's a does. [20, 30, 40]
Question: I’m building a small Flask 3.0 / Python 3.12 micro-service that calls an external REST API on almost every request Right now each route makes a new requests.Session which is slow and leaks sockets under load from flask import Flask, jsonify import requests app = Flask(__name__) @app.get("/info") def info(): with requests.Session() as s: r = s.get("https://api.example.com/info") return jsonify(r.json())  What I tried  Global Variable  session = requests.Session()  I get a resource warning through the above. How can I re-use one requests.Session for all incoming requests and close it exactly once when the application exits?

Answer: Use serving-lifecycle hooks  @app.before_serving – runs once per worker, right before the first request is accepted.  @app.after_serving – runs once on a clean shutdown   Create the requests.Session in the first hook, stash it on the application object and close it in the second.
Question: I have an existing dataframe: data = [[5011025, 234], [5012025, 937], [5013025, 625]] df = pd.DataFrame(data)  output:  0 1 0 5011025 234 1 5012025 937 2 5013025 625  What I need to do is insert a new column at 0 (the same # of rows) that contains 3 spaces. Recreating the dataframe, from scratch, it would be something like this: data = [[' ',5011025, 234], [' ',5012025, 937], [' ',5013025, 625]] df = pd.DataFrame(data)  desired output:  0 1 2 0 5011025 234 1 5012025 937 2 5013025 625  What is the best way to insert() this new column into an existing dataframe, that may be hundreds of rows? Ultimately, i'm trying to figure out how to write a function that will shift all columns of a dataframe x number of spaces to the right.

Answer: Based on your comment, you could shift all cols up one and add a col 0 like this:  import pandas as pd data = [[5011025, 234], [5012025, 937], [5013025, 625]] df = pd.DataFrame(data) df.columns = df.columns + 1 df[0] = ' ' df = df.sort_index(axis=1)
Question: I’m building a small internal micro-service in Python 3.12 / Flask 3.0. The service accepts POST /upload requests that insert a record into PostgreSQL. Problem Mobile clients sometimes retry the request when the network is flaky, so I end up with duplicate rows: @app.post("/upload") def upload(): payload = request.get_json() db.execute( "INSERT INTO photos (user_id, filename, uploaded_at) VALUES (%s, %s, NOW())", (payload["user_id"], payload["filename"]), ) return jsonify({"status": "ok"}), 201  What I tried  Added a UNIQUE (user_id, filename) constraint – works, but clients now get a raw SQL error on duplicate inserts.  Wrapped the insert in ON CONFLICT DO NOTHING – avoids the error but I can’t tell whether the row was really inserted.  Googled for “Flask idempotent POST” and found libraries like Flask-Idem, but they feel heavyweight for a single route.   Question: What’s the simplest, idiomatic way in Flask to make this endpoint idempotent so that:  POSTing the same photo twice is harmless;  the client still gets a clear 201 Created the first time and 200 OK for retries; and  I don’t have to introduce extra infrastructure (Kafka, Redis, etc.)?  

Answer:  Give the table a uniqueness guarantee so duplicates physically can’t happen.  Use an UPSERT (INSERT … ON CONFLICT) with RETURNING so you know whether the row was really inserted.  Map that to HTTP status codes.
Question: In the below dataframe I would like to filter the columns based on a list called 'animals' to select all the columns that include the list elements. animal_data = { "date": ["2023-01-22","2023-11-16","2024-06-30","2024-08-16","2025-01-22"], "cats_fostered": [1,2,3,4,5], "cats_adopted":[1,2,3,4,5], "dogs_fostered":[1,2,3,4,5], "dogs_adopted":[1,2,3,4,5], "rabbits_fostered":[1,2,3,4,5], "rabbits_adopted":[1,2,3,4,5] } animals = ["date","cat","rabbit"] animal_data = { "date": ["2023-01-22","2023-11-16","2024-06-30","2024-08-16","2025-01-22"], "cats_fostered": [1,2,3,4,5], "cats_adopted":[1,2,3,4,5], "rabbits_fostered":[1,2,3,4,5], "rabbits_adopted":[1,2,3,4,5] }  I have tried some approaches below but they either don't work with lists or return no columns as it is looking for an exact match with 'cats' or 'rabbits' and not just columns that contain the strings animal_data[animal_data.columns.intersection(animals)] # returns an empty df animal_data.filter(regex=animals) # returns an error: not able to use regex with a list 

Answer: The issue with both attempts is that you are looking for a substring of the columns name. Except for the date column there is not a full match between the strings in the animals list and the actual column names. One possibility is to use filter with .join to construct the regex if using .filter, or a "manual" list comprehension with strings operations (for example in or .startswith). You can also "hardcode" "date" so the animals list only contains animals. >>> animals = ["cat"] >>> df.filter(regex="date|" + "|".join(animals)) date cats_fostered cats_adopted 0 2023-01-22 1 1 1 2023-11-16 2 2 2 2024-06-30 3 3 3 2024-08-16 4 4 4 2025-01-22 5 5 >>> animals = ["cat", "rabbit"] >>> df.filter(regex="date|" + "|".join(animals)) date cats_fostered cats_adopted rabbits_fostered rabbits_adopted 0 2023-01-22 1 1 1 1 1 2023-11-16 2 2 2 2 2 2024-06-30 3 3 3 3 3 2024-08-16 4 4 4 4 4 2025-01-22 5 5 5 5
Question:  I have a 1d numpy array It's mostly decreasing, but it increases in a few places I'm interested in the places where it increases in several consecutive elements, and how many consecutive elements it increases for in each case In other words, I'm interested in the lengths of increasing contiguous sub-arrays I'd like to compute and store this information in an array with the same shape as the input (EG that I could use for plotting) This could be achieved using cumsum on a binary mask, except I want to reset the accumulation every time the array starts decreasing again See example input and expected output below How do I do that?  import numpy as np def count_consecutive_increases(y: np.ndarray) -> np.ndarray: ... y = np.array([9, 8, 7, 9, 6, 5, 6, 7, 8, 4, 3, 1, 2, 3, 0]) c = count_consecutive_increases(y) print(y) print(c) # >>> [9 8 7 9 6 5 6 7 8 4 3 1 2 3 0] # >>> [0 0 0 1 0 0 1 2 3 0 0 0 1 2 0] 

Answer: Here is another solution: import numpy as np def count_consecutive_increases(y: np.ndarray) -> np.ndarray: increases = np.diff(y, prepend=y[0]) > 0 all_summed = np.cumsum(increases) return all_summed - np.maximum.accumulate(all_summed * ~increases) y = np.array([9, 8, 7, 9, 6, 5, 6, 7, 8, 4, 3, 1, 2, 3, 0]) c = count_consecutive_increases(y) print(y) # >>> [9 8 7 9 6 5 6 7 8 4 3 1 2 3 0] print(c) # >>> [0 0 0 1 0 0 1 2 3 0 0 0 1 2 0]  The idea is the same as with the solution proposed by OP, albeit a bit shorter: Naively count (by summing over) all indices that have been marked as increasing, then subtract, for each consecutive increasing segment, the count right before its start (the value of which we get by a cumulative maximum over the naive counts at the positions marked as not increasing).