eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm currently learning Pandas and stuck with a problem. I have the following data: labels = [ 'date', 'name', 'opponent', 'gf', 'ga'] data = [ [ '2023-08-5', 'Liverpool', 'Man Utd', 5, 0 ], [ '2023-08-10', 'Liverpool', 'Everton', 0, 0 ], [ '2023-08-14', 'Liverpool', 'Tottenham', 3, 2 ], [ '2023-08-18', 'Liverpool', 'Arsenal', 4, 4 ], [ '2023-08-27', 'Liverpool', 'Man City', 0, 0 ], ] df = pd.DataFrame(data, columns=labels)  The games / rows are sorted by date. for each row / game I would like to count the column values of 'goals_for' and 'goals_against' in the previous rows / games (excluding the current row or any after the date). So I would like the data to be like this: labels = [ 'date', 'name', 'opponent', 'gf', 'ga', 'total_gf', 'total_ga' ] data = [ [ '2023-08-5', 'Liverpool', 'Man Utd', 5, 0, 0, 0 ], [ '2023-08-10', 'Liverpool', 'Everton', 0, 0, 5, 0 ], [ '2023-08-14', 'Liverpool', 'Tottenham', 3, 2, 5, 0 ], [ '2023-08-18', 'Liverpool', 'Arsenal', 4, 4, 8, 2 ], [ '2023-08-27', 'Liverpool', 'Man City', 0, 0, 12, 6 ], ]  I tried expanding() but it seems to include the current row. rolling has a parameter closed='left' but others don't have it. Any help or tips or links to similar solutions would be appreciated.

Answer: You can shift with fill_value=0, then cumsum: df['total_gf'] = df['gf'].shift(fill_value=0).cumsum() df['total_ga'] = df['ga'].shift(fill_value=0).cumsum()  Alternatively, processing all columns at once: df[['total_gf', 'total_ga']] = df[['gf', 'ga']].shift(fill_value=0).cumsum()  Or, create a new DataFrame: out = df.join(df[['gf', 'ga']].shift(fill_value=0).cumsum().add_prefix('total_'))  Output:  date name opponent gf ga total_gf total_ga 0 2023-08-5 Liverpool Man Utd 5 0 0 0 1 2023-08-10 Liverpool Everton 0 0 5 0 2 2023-08-14 Liverpool Tottenham 3 2 5 0 3 2023-08-18 Liverpool Arsenal 4 4 8 2 4 2023-08-27 Liverpool Man City 0 0 12 6
Question: My data frame is similar to below: Name Col1 Col2 Col3 Col4 Col5 Col6 A Y Y Y B Y Y Y Y C Y Y Y D Y  I want to add a column that counts the other columns containing a value similar to: Name Col1 Col2 Col3 Col4 Col5 Col6 Score A Y Y Y 3 B Y Y Y Y 4 C Y Y Y 3 D Y 1  I have tried the following but with no success: df['Score'] = df.count(df[['Col1','Col2','Col3','Col4','Col5','Col6']], axis='columns') 

Answer: If there are missing values use DataFrame.count with subset of columns: df['Score'] = df[['Col1','Col2','Col3','Col4','Col5','Col6']].count(axis=1) print (df) Name Col1 Col2 Col3 Col4 Col5 Col6 Score 0 A Y Y NaN Y NaN NaN 3 1 B Y NaN Y Y NaN Y 4 2 C NaN Y Y NaN Y NaN 3 3 D Y NaN NaN NaN NaN NaN 1
Question: I have a multi index dataframe, and I want to combine rows based on certain conditions and I want to combine rows per index. import pandas as pd # data data = { 'date': ['01/01/17', '02/01/17', '03/01/17', '01/01/17', '02/01/17', '03/01/17'], 'language': ['python', 'python', 'python', 'r', 'r', 'r'], 'ex_complete': [6, 5, 10, 8, 8, 8] } # Convert to DataFrame df = pd.DataFrame(data) # Convert DataFrame to JSON json_data = df.to_json(orient='records') # Convert JSON data back to DataFrame df_from_json = pd.read_json(json_data, orient='records') # Set date and language as multi-index df_from_json.set_index(['date', 'language'], inplace=True) df_from_json.sort_index(inplace= True) df_from_json  1st Problem: I want to combine the dates '01/01/17', '02/01/17' and rename as '1_2', this should give me 4 rows: 2 rows for '1_2' - (Python and R) and 2 rows for '03/01/17' (Python and R) 2nd Problem: I want to combine Python and R rows and rename as Python_R, this should give 3 rows for 3 dates. Any guidance or pointer will be hugely appreciated.

Answer: IIUC use DataFrame.rename with aggregate, e.g. sum: out = (df_from_json.rename({pd.Timestamp('01/01/17'):'1_2', pd.Timestamp('02/01/17'):'1_2'}, level=0) .groupby(level=[0,1]).sum()) print (out) ex_complete date language 2017-03-01 00:00:00 python 10 r 8 1_2 python 11 r 16   out = (df_from_json.rename({'python':'Python_R', 'r':'Python_R'}, level=1) .groupby(level=[0,1]).sum()) print (out) ex_complete date language 2017-01-01 Python_R 14 2017-02-01 Python_R 13 2017-03-01 Python_R 18
Question: I have the following dataframe:  df = pl.DataFrame({ "Column A": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "Column B": [2, 3, 1, 4, 1, 7, 3, 2, 12, 0] })  I want to create a new column C that holds the distance, in rows, between the B value of the current row and the next value in column B that is greater than or equal to B + 50%. The end result should look like this:  df = pl.DataFrame({ "Column A": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "Column B": [2, 3, 1, 4, 1, 7, 3, 2, 12, 0], "Column C": [1, 4, 1, 2, 1, 3, 2, 1, None, None] })  How can I efficiently achieve this using Polars, especially since I'm working with a large DataFrame?

Answer: Ok, so first I should say - this one looks like it requires join on inequality on multiple columns and from what I've found pure polars is not great with it. It's probably possible to do it with join_asof but I couldn't make it pretty. I'd probably use duckdb integration with polars to achieve the results: import duckdb duckdb.sql(""" select d."Column A", d."Column B", ( select tt."Column A" from df as tt where tt."Column A" > d."Column A" and tt."Column B" >= d."Column B" * 1.5 order by tt."Column A" asc limit 1 ) - d."Column A" as "Column C" from df as d """).pl() ┌──────────┬──────────┬──────────┐ │ Column A ┆ Column B ┆ Column C │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞══════════╪══════════╪══════════╡ │ 1 ┆ 2 ┆ 1 │ │ 2 ┆ 3 ┆ 4 │ │ 3 ┆ 1 ┆ 1 │ │ 4 ┆ 4 ┆ 2 │ │ 5 ┆ 1 ┆ 1 │ │ 6 ┆ 7 ┆ 3 │ │ 7 ┆ 3 ┆ 2 │ │ 8 ┆ 2 ┆ 1 │ │ 9 ┆ 12 ┆ null │ │ 10 ┆ 0 ┆ null │ └──────────┴──────────┴──────────┘
Question: So, I have to match everything before the last open-close parenthesis and get that in group. And after the last open-close parenthesis try to get the value with some pattern, again in group. This is my sample example: ID pqr () name:123.  And this is my regex: ^(?P<JUNK>.*?)(?!\(.\))(\(.*\))?\sname\:(?P<name>\d+)\.$  And now with this I am getting ID pqr  for JUNK key and 123 for name key which is perfect. And now, with this regex, it's working fine for below strings:  ID pqr (a) () name:123. ID pqr (a) (b) () name:123. ID pqr (a) (b) () name:123. ID pqr (a) (b) (XX) name:123.  in returns I am getting these outputs:  {'JUNK': 'ID pqr ', 'name': '123'} {'JUNK': 'ID pqr (a) ', 'name': '123'} {'JUNK': 'ID pqr (a) (b) ', 'name': '123'} {'JUNK': 'ID pqr (a) (b) ', 'name': '123'}  So far, it's working fine with above strings, but for the below ones I am having some trouble  ID pqr (a) (b) (X) name:123. ID pqr (aa) (b) (X) name:123. ID pqr (a) (bb) (X) name:123.  For these strings, I am getting output like this:  {'JUNK': 'ID pqr (a) (b) (X)', 'name': '123'} {'JUNK': 'ID pqr ', 'name': '123'} {'JUNK': 'ID pqr (a) ', 'name': '123'}  But basically I want like this:  {'JUNK': 'ID pqr (a) (b) ', 'name': '123'} {'JUNK': 'ID pqr (aa) (b) ', 'name': '123'} {'JUNK': 'ID pqr (a) (bb) ', 'name': '123'}  This is my regex101 attempt: https://regex101.com/r/AV8WlB/4 Can anyone point out where I am going wrong?

Answer: Try: ^(?P<JUNK>.*?)(\([^)]*\)\s)?name:(?P<name>\d+)\.$  See: regex101  Explanation  ^: Start of string... (?P<JUNK>.*?): followed by JUNK... (\([^)]*\)\s)?: then, if applicable the last pair of brackets... name:: before literal "name:"... (?P<name>\d+): and collecting the number before... \.$: final "." at the end of the string.
Question: I made a Tkinter app using Python. I want it to auto-resize and fill the whole window no matter how I resize it. The problem is that when I expand the window size (using my cursor), grey space appears on the borders to fill the gaps (see the image below). Grey space filling Here is the code minus the unnecessary details : import tkinter as tk from tkinter import ttk import openpyxl root = tk.Tk() frame = ttk.Frame(root) frame.pack() widgets_frame = ttk.LabelFrame(frame, text = "Insert Data") widgets_frame.grid(row = 0, column = 0, padx = 20, pady = 10) day_label = ttk.Label(widgets_frame, text = "Date") day_label.grid(row = 1, column = 0, padx = 5, pady = (0, 5), sticky = "ew") day_entry = ttk.Entry(widgets_frame) day_entry.grid(row = 2, column = 0, padx = 5, pady = (0, 5), sticky = "ew") walking_entry = ttk.Entry(widgets_frame) walking_entry.insert(0, "Walking duration (min)") walking_entry.bind("<FocusIn>", lambda e: walking_entry.delete('0', 'end')) walking_entry.grid(row = 3, column = 0, padx = 5, pady = (0, 5), sticky = "ew") button = ttk.Button(widgets_frame, text = "Insert") button.grid(row = 4, column = 0, padx = 5, pady = 5, sticky = "nsew") treeFrame = ttk.Frame(frame) treeFrame.grid(row = 0, column = 1, pady = 10) treeScroll = ttk.Scrollbar(treeFrame) treeScroll.pack(side = "right", fill = "y") cols = ("Day", "Walking") treeview = ttk.Treeview(treeFrame, show = "headings", yscrollcommand = treeScroll.set, columns = cols, height = 13) treeview.column("Day", width = 100) treeview.column("Walking", width = 50) treeview.pack() treeScroll.config(command = treeview.yview) root.mainloop()  Does anybody have a suggestion that would help me solve this issue ? Thanks. I added this line of code because, in another thread, they said it helps with expanding the widgets equally when the window size increases : root.grid_columnconfigure(0,weight = 1) root.grid_columnconfigure(1,weight = 1) root.grid_rowconfigure(0, weight = 1) root.grid_rowconfigure(1, weight = 1) root.grid_rowconfigure(2, weight = 1) root.grid_rowconfigure(3, weight = 1) root.grid_rowconfigure(4, weight = 1)  Unfortunately, even after doing so, I didn't notice any changes.

Answer: You could use expand = True, fill = BOTH as an attribute for the required widgets to expand them to full size.
Question: ts = pd.Series({'a' : 1, 'b' : 2}) ids = ['a','c'] # 'c' is not in the index # the result I want np.array([ts.get(k, np.nan) for k in ids])  Is there a pandas native way to achieve this?

Answer: reindex and access the underlying numpy array: out = ts.reindex(ids).values # or # out = ts.reindex(ids).to_numpy()  Output: array([ 1., nan]) Note that the fill_value in reindex is NaN by default, you can change it if desired: ts.reindex(ids, fill_value=0).values # array([1, 0])
Question: I have the following dataframe in:  a 1 3 2 2 3 Nan 4 3 5 Nan  I need to recode this column so it looks like this:  df_miss_a 1 0 2 0 3 1 4 0 5 1  I've tried: df_miss_a = np.where(df['a'] == 'Nan', 1, 0)  and df_miss_a = np.where(df['a'] == Nan, 1, 0)  The above outputs only 0s. The format of the output is unimportant.

Answer: If you have NaNs in your column you can use pd.Series.isna(): df_miss_a = df["a"].isna().astype(int) print(df_miss_a)  Prints: 1 0 2 0 3 1 4 0 5 1 Name: a, dtype: int64
Question: I'm new to JAX and reading the docs i found that jitted functions should not contain iterators (section on pure functions) and they bring this example: import jax.numpy as jnp import jax.lax as lax from jax import jit # lax.fori_loop array = jnp.arange(10) print(lax.fori_loop(0, 10, lambda i,x: x+array[i], 0)) # expected result 45 iterator = iter(range(10)) print(lax.fori_loop(0, 10, lambda i,x: x+next(iterator), 0)) # unexpected result 0  trying to fiddling with it a little bit in order to see if i can get directly an error instead of undefined behaviour i wrote @jit def f(x, arr): for i in range(10): x += arr[i] return x @jit def f1(x, arr): it = iter(arr) for i in range(10): x += next(it) return x print(f(0,array)) # 45 as expected print(f1(0,array)) # still 45  Is it a "chance" that the jitted function f1() now shows the correct behaviour?

Answer: Your code works because of the way that JAX's tracing model works. When JAX's tracing encounters Python control flow, like for loops, the loop is fully evaluated at trace-time (There's some exploration of this in JAX Sharp Bits: Control Flow). Because of this, your use of an iterator in this context is fine, because every iteration is evaluated at trace-time, and so next(it) is re-evaluated at every iteration. In contrast, when using lax.fori_loop, next(iterator) is only executed a single time and its output is treated as a trace-time constant that will not change during the runtime iterations.
Question: I'm working on matlab, but I noticed that the image I was loading becomes black-an-white, when the original image has grays in it. I decided to check the image using a imshow on a new file, but it really makes it black and white. My code is simply: image_path = 'no.png'; img = imread(image_path); imshow(img); axis off;  Then I tried to make it grayscale, but the problem was not solved. Then I tried to use python to show the image, and it showed the image correctly. import matplotlib.pyplot as plt import matplotlib.image as mpimg def show_image(image_path): img = mpimg.imread(image_path) plt.imshow(img) plt.axis('off') plt.show() # Example usage image_path = "no.png" show_image(image_path)  I don't know what's wrong. This is the original image:  This is the image matlab shows:  This is the image python shows: 

Answer: Actually, the problem comes from the fact that your image is not grayscale, but is indexed. image_path = 'no.png'; [img,map] = imread(image_path); % Returns a non-empty map --> indexed image  Now, you can get the expected output by using the map: imshow(img,map); axis off;   You can look at this answer for how to check what the type of your image is, in particular the following part (trimmed for clarity): [imageArray, colorMap] = imread(fullImageFileName); [rows, columns, numberOfColorChannels] = size(imageArray); % % % % Tell user the type of image it is. if numberOfColorChannels == 3 % This is a true color, RGB image. % elseif numberOfColorChannels == 1 && isempty(colorMap) % This is a gray scale image, with no stored color map. % elseif numberOfColorChannels == 1 && ~isempty(colorMap) % This is an indexed image. It has one "value" channel with a % stored color map that is used to pseudocolor it.
Question: I am interacting with the openai assistant API (python). so far, it works well. but sometimes the api returns the message i sent to the assistant instead of the assistants reply. from openai import OpenAI from os import environ OPEN_API_KEY = environ.get('OPEN_API_KEY') assistant_id = "xxxxxxx" client = OpenAI(api_key=OPEN_API_KEY) assistant = client.beta.assistants.retrieve(assistant_id) def ask_assistant(message_text): print(f'received message: {message_text}') thread = client.beta.threads.create() message = client.beta.threads.messages.create( thread_id=thread.id, role="user", content=message_text ) run = client.beta.threads.runs.create( thread_id=thread.id, assistant_id=assistant_id ) run_retrieve = client.beta.threads.runs.retrieve( thread_id=thread.id, run_id=run.id ) messages = client.beta.threads.messages.list(thread.id) final_text = messages.data[0].content[0].text.value try: final_text = messages.data[0].content[0].text.value print(final_text) except Exception as e: print(e) final_text = '' return final_text if __name__ == "__main__": ask_assistant('How are you?')  as a hot fix, i implemented a sleep function that waits for 3 seconds before retrieving the reply. this works, but i don't expect it to be the best solution (or reliable at all). any ideas how to wait until the assistant REALLY replied? thank you very much. Update: This works: from openai import OpenAI from os import environ OPEN_API_KEY = environ.get('xxxxx') assistant_id = "xxxxx" client = OpenAI(api_key=OPEN_API_KEY) assistant = client.beta.assistants.retrieve(assistant_id) def get_answer(run, thread): while not run.status == "completed": print("Waiting for answer...") run = client.beta.threads.runs.retrieve( thread_id=thread.id, run_id=run.id ) messages = client.beta.threads.messages.list(thread.id) answer = messages.data[0].content[0].text.value try: answer = messages.data[0].content[0].text.value except Exception as e: print(e) answer = '' return answer def ask_assistant(message_text): print(f'sending to assistant: {message_text}') thread = client.beta.threads.create() message = client.beta.threads.messages.create( thread_id=thread.id, role="user", content=message_text ) run = client.beta.threads.runs.create( thread_id=thread.id, assistant_id=assistant_id ) answer = get_answer(run, thread) print(f'assistant response: {answer}') return answer if __name__ == "__main__": ask_assistant('Hello') 

Answer: This should work: run = openai.beta.threads.runs.create( thread_id=thread.id, assistant_id=assistant.id ) print(run) while run.status !="completed": run = openai.beta.threads.runs.retrieve( thread_id=thread.id, run_id=run.id ) print(run.status) messages = openai.beta.threads.messages.list( thread_id=thread.id ) print(messages.data[0].content[0].text.value)  Thanks to @joyasree78
Question: Sometime, DataFrame.plot() inside a for loop produces multiple charts. import pandas as pd data = {'Str': ['A', 'A', 'B', 'B'], 'Num': [i for i in range(4)]} df = pd.DataFrame(data) for n in ['A', 'B']: df[df.Str == n].plot(kind='bar')   But sometimes, it produces a single chart. import pandas as pd data = {'C1': ['A', 'A', 'B', 'B'], 'C2': [i for i in range(4)], 'C3': [1,2,1,2]} df = pd.DataFrame(data) for n in [1,2]: df[df.C3 == n].groupby('C1').C2.sum().plot(kind='bar')   From the previous code, if plt.show() was added at the end of the loop. It will produce multiple charts. import pandas as pd import matplotlib.pyplot as plt data = {'C1': ['A', 'A', 'B', 'B'], 'C2': [i for i in range(4)], 'C3': [1,2,1,2]} df = pd.DataFrame(data) for n in [1,2]: df[df.C3 == n].groupby('C1').C2.sum().plot(kind='bar') plt.show()   I don't want to use plt.show(). Actually I want to import only pandas and create multiple charts using a for loop.

Answer: I'm not sure why matplotlib behave this way, but your code in the second loop products a Series while the first produces a DataFrame. This might force reusing the same axes. You could add as_index=False in the groupby to have an object that is comparable to the one in the first loop: for n in [1,2]: df[df.C3 == n].groupby('C1', as_index=False).C2.sum().plot(kind='bar')  Output:   Also note that (explicitly) importing matplotlib should not be an issue. It is already a dependency of pandas and will be imported by pandas anyway, therefore not importing it won't cause any gain in the code.
Question: According to How the write(), read() and getvalue() methods of Python io.BytesIO work? , it seems io.BytesIO will copy initial bytes provided. I have a buffer, and I want pickle to directly dump object in that buffer, without any copy. So I tried to use f = io.BytesIO(buffer). However, after f.write, my buffer is not modified. Here is an example: a = bytearray(1024) import io b = io.BytesIO(memoryview(a)) b.write(b"1234") a[:4] # a does not change  What I want, is to make io.BytesIO directly writes to my buffer. My ultimate goal is: a = bytearray(1024) import io, pickle with io.BytesIO(memoryview(a)) as f: obj = [1, 2, 3] pickle.dump(obj, f) # a should hold the pickled data of obj 

Answer: IIUC you can try to make your own IO class: import io import pickle class MyIO(io.IOBase): def __init__(self, buf): self.buf = memoryview(buf) self.position = 0 def write(self, b): start = self.position n = len(b) end = start + n self.buf[start: end] = b self.position = end a = bytearray(1024) with MyIO(a) as f: obj = [1, 2, 3] pickle.dump(obj, f) print(a)  Prints: bytearray(b'\x80\x04\x95\x0b\x00\x00\x00\x00\x00\x00\x00]\x94(K\x01K\x02K\x03e.\x00\x00\x00 ...
Question: I am trying to switch some of my pandas code to polars to leverage it's performance. I have found that the .filter operation is much slower than a similar slicing using .loc. import pandas as pd import polars as pl import datetime as dt import numpy as np date_index = pd.date_range(dt.date(2001,1,1), dt.date(2020,1,1),freq='1H') n = date_index.shape[0] test_pd = pd.DataFrame(data = np.random.randint(1,100, n), index=date_index, columns = ['test']) test_pl = pl.DataFrame(test_pd.reset_index()) test_dates = date_index[np.random.randint(0,n,1000)] st = time.perf_counter() for i in test_dates: d = test_pd.loc[i,:] print(f"Pandas {time.perf_counter() - st}") st = time.perf_counter() for i in test_dates: d = test_pl.filter(index=i) print(f"Polars {time.perf_counter() - st}") Pandas 0.1854726000019582 Polars 2.1125728000042727  Is there some other way to speedup the slicing operation in polars?

Answer: Polars doesn't use indexes, so random access of one specific element (if not by row number) will always have to loop over all the data. But you can efficiently get all the dates you are interested in in one go using a left join: test_dates_df = pl.DataFrame({"index": test_dates}) out = test_dates_df.join(test_pl, on="index", how="left")  Then out[0] contains the row where the index column matches test_dates[0], etc. On my machine this gives the following times: Pandas 0.029560166876763105 Polars 0.0009763331618160009
Question: I have the data in the form of table below Name Mas Sce M ( (87) 83 (91) (97) ) T (77) 76 R (60) 32 G (95) 20 M ( (50) 89 (50) (99) )  Some of my data runs through multiple row such as M case. The data is enclosed within the bracket I have tried drop duplicates. It works when its single row. But, now i have a few rows as a group import pandas as pd d = {'Name': ['M', None, None, 'T', 'R', 'G', 'M', '', ''], 'Mas': ['( (87)', '(91)', '(97) )', '(77)', '(60)', '(95)', '( (50)', '(50)', '(99) )'], 'Sce': ['83', '', '', '76', '32', '20', '89', '', '']} df = pd.DataFrame(d) df['Name'] = df['Name'].ffill() print(df) df.drop_duplicates(subset='Name', keep='first', inplace=True) print(df)  I want to remove reoccurence of the data. In this case, the 2nd M Name Mas Sce M ( (87) 83 (91) (97) ) T (77) 76 R (60) 32 G (95) 20 

Answer: A pure pandas one-liner with duplicated, a mask and ffill: out = df[~df['Name'].duplicated() .mask(df['Name'].replace('', None).isna()) .ffill().infer_objects(copy=False)]  Output:  Name Mas Sce 0 M ( (87) 83 1 None (91) 2 None (97) ) 3 T (77) 76 4 R (60) 32 5 G (95) 20
Question: Happy Sunday. I have this code that I want to run using the multi-processing module. But it doesn't just work for some reason. with ProcessPoolExecutor() as executor: while True: if LOCAL_RUN: print("ALERT: Doing a local run of the automation with limited capabilities.") list_of_clients = db_manager.get_clients() random.shuffle(list_of_clients) list_of_not_attempted_clients_domains = db_manager.tags() group_of_clients_handlers = {} # no matches if not list_of_not_attempted_clients_domains: sleep = 60 * 10 pretty_print(f'No matches found. Sleeping for {sleep}s') time.sleep(sleep) continue for client in list_of_clients: client_id = client[0] client_name = client[1] group_of_clients_handlers[client_id] = [ClientsHandler(db_manager), client_name] # MULTI-PROCESSING CODE try: print('running function...') executor.map( partial(run, group_of_clients_handlers=group_of_clients_handlers), list_of_not_attempted_clients_domains ) except Exception as err: print(err)  Despite all my attempts to debug this, I have no idea why this doesn't work, although I feel it relates to processes taking time to start up or scheduling task etc but I am not certain. The while loop just keeps running and I see all the print statements like running function... but the run function never executes. The run function is a very large function with nested large functions. The except block doesn't print out any error either. Would love to hear what you think...

Answer: ProcessPoolExecutor.map creates an iterator, you must consume the iterator to get the exception, otherwise the exception will be discarded. from concurrent.futures import ProcessPoolExecutor def raising_func(val): raise ValueError(val) with ProcessPoolExecutor(4) as pool: pool.map(raising_func, [1,2,3,4,5]) with ProcessPoolExecutor(4) as pool: list(pool.map(raising_func, [1,2,3,4,5])) # < ---- exception is thrown from here