eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My input dataframe (or 2D numpy array) shape is 11kx12k For each element, I check how many values are higher in the given row and column. For example: array([[1, 3, 1, 5, 3], [3, 7, 3, 2, 1], [2, 3, 1, 8, 9]])  row-wise: 3 1 3 0 1 1 0 1 3 4 3 2 4 1 0  column-wise: 2 1 1 1 1 0 0 0 2 2 1 1 1 0 0  total higher values for that element: 5 2 4 1 2 1 0 1 5 6 4 3 5 1 0  this code works good but for this shape of matrix It takes ~1 hour. dfT = df.T rows = df.apply(lambda x: np.sum(df>x),axis=1) cols = dfT.apply(lambda x: np.sum(dfT>x),axis=1).T output = rows+cols  I wonder is there any way to do this more efficient way? I also tried numpy but for this I have splited my 2D to 12kx100 or 12kx200 shapes and merged all arrays again, in the end runtime was close to eachother so couldn't get any progress. np.sum(matrix > matrix[:,None], axis=1) 

Answer: I came up with the following: from scipy.stats import rankdata def element_wise_bigger_than(x, axis): return x.shape[axis] - rankdata(x, method='max', axis=axis)  What you want is similar to a ranking of elements, which basically tells you how many items are smaller than a given element (maybe with subtracting 1 if your ranking is 1, 2, 3, ...). rankdata does this ranking, and can do it row-wise or column-wise. We can convert the "smaller than" to "bigger than" by inverting the ranks, relative to the number of items being counted (in each row/column). I.e., subtract the ranks from the number of rows/columns. For rows/columns without ties (e.g., the third row in your example), this works as expected. There is a slight complication with ties and how they are ranked. Through trial and error, I found that using method='max' gave the desired behavior. To be frank, I am still wrapping my head on why/how this works, so take the answer with a grain of salt. Still, it gives the desired output on the input data: >>> x = np.array([[1, 3, 1, 5, 3], [3, 7, 3, 2, 1], [2, 3, 1, 8, 9]]) >>> element_wise_bigger_than(x, 1) array([[3, 1, 3, 0, 1], [1, 0, 1, 3, 4], [3, 2, 4, 1, 0]]) >>> element_wise_bigger_than(x, 0) array([[2, 1, 1, 1, 1], [0, 0, 0, 2, 2], [1, 1, 1, 0, 0]])  And it can handle your size of data in far less than an hour: def experiment(): x = np.random.randint(0, 10, size=(11000, 12000)) rows = element_wise_bigger_than(x, 1) cols = element_wise_bigger_than(x, 0) output = rows + cols return output %%time experiment() # CPU times: user 8.68 s, sys: 805 ms, total: 9.49 s # Wall time: 9.47 s  Takes about 10 seconds. My understanding is the ranking should be similar to a sorting operation. My answer is relying on a scipy/numpy implementation of the rank/sort, rather than a user-written loop/"apply" (which tend to be much slower).
Question: Imagine I have two threads, each modifying a different variable. Can I pass the same lock object to them, or shall I use two separate locks? In general, when shall I use multiple locks? Here is a toy example: from threading import Thread, Lock from time import sleep def task(lock, var): with lock: var = 1 sleep(5) lock = Lock() var1 = [] var2 = [] Thread(target=task, args=(lock, var1)).start() Thread(target=task, args=(lock, var2)).start()  or is it better lock1 = Lock() lock2 = Lock() var1 = [] var2 = [] Thread(target=task, args=(lock1, var1)).start() Thread(target=task, args=(lock2, var2)).start() 

Answer: Only one thread can hold a lock at a time, and a lock is used to provide mutually-exclusive access to a piece of data (or variable). Based on your example, where each thread accesses separate variables, you want two separate locks—one per variable. Note that if only one thread accesses a variable, though, you do not need to use a lock for it. If you were to only use one lock, only one thread would be able to perform "work" at a time—even though their work is completely independent.
Question: I want to write something shaped like this this except the invalid_task and work functions are of course written inline. unfinished_tasks = 0 async def worker(task): nonlocal unfinished_tasks unfinished_tasks += 1 try: if is_invalid(task): return None result = work(task) if is_error(result): raise Exception() return result else: unfinished_tasks -= 1  But you can't write just: try: ... else: ...  You need an except to write an else (try-else-finally also doesn't work). The use case for an else without an except is to run some code on a non exception exit of the block. There are a few ways to exit a block, like return, break, or reaching the end of one of the code paths inside it. Should python allow try-else without except? Is there a better way to write this? My current solution is: try: ... except None: raise else: ... 

Answer: For this case specifically you can use: unfinished_tasks = 0 async def worker(task): nonlocal unfinished_tasks try: if is_invalid(task): return None result = work(task) if is_error(result): raise Exception() return result except: unfinished_tasks += 1  For general cases pass can be used: try: ... except: pass else: ...
Question: I have dataframe like this. But there are about ten thousand rows. import pandas as pd import numpy as np data = {'gameId': [1, 1, 1, 1, 1, 2, 2, 2, 2, 2], 'eventId': [1, 2, 3, 4, 5, 1, 2, 3, 4, 5], 'player': ['A', 'B', 'C', 'D', 'E', 'A', 'B', 'C', 'D', 'E'], 'related_eventId': [2, 1, 4, 3, np.nan, 2, 1, 4, 3, np.nan]}  So I need to create column "related_player" based on the player from row which eventId is equal related_eventId. If I would not have column gameId I can do it by merging result = df.merge(df[['eventId', 'player']], left_on='related_eventId', right_on='eventId', how='left', suffixes=('', '_related')) result.rename(columns={'player_related': 'related_player', 'eventId_related': 'related_eventId'}, inplace=True) result = result[['eventId', 'player', 'related_eventId', 'related_player']]  But output is not correct because I need to group by gameId. In R it is pretty simple, but I don't understand how to correctly do it in Python. My expected output should be like this     gameId eventId player related_eventId related_player     1 1 A 2 B   1 2 B 1 A   1 3 C 4 D   1 4 D 3 C   1 5 E NaN NaN   2 1 A 2 B   2 2 B 1 A   2 3 C 4 D   2 4 D 3 C   2 5 E NaN NaN    

Answer: Add column to list ['eventId', 'player','gameId'] and to parameters left_on and right_on: result = df.merge(df[['eventId', 'player','gameId']], left_on=['gameId','related_eventId'], right_on=['gameId','eventId'], how='left', suffixes=('', '_related')) result.rename(columns={'player_related': 'related_player', 'eventId_related': 'related_eventId'}, inplace=True) result = result[['eventId', 'player', 'related_eventId', 'related_player']] print (result) eventId player related_eventId related_eventId related_player 0 1 A 2.0 2.0 B 1 2 B 1.0 1.0 A 2 3 C 4.0 4.0 D 3 4 D 3.0 3.0 C 4 5 E NaN NaN NaN 5 1 A 2.0 2.0 B 6 2 B 1.0 1.0 A 7 3 C 4.0 4.0 D 8 4 D 3.0 3.0 C 9 5 E NaN NaN NaN
Question: I have VSCode in Windows 11. I have WSL (Ubuntu 22.04) and launch VSCode from the terminal like code . from the project folder. When I open the built-in terminal it shows two conda (Anaconda) environments in parentheses, so I have no idea which one is active, if any. On subsequent conda deactivate you can see in the attached screenshot that the prompt and the active env changing but there is surely something messed up here. Also, in VSCode, when I set Python Interpreter to a conda env, in a few seconds the built-in terminal prompt picks up the change and the env name in the first parens changes to the new value. Any idea how to fix it? (The prompt should obviously show just one (the active) conda env and that one should change whenever the Python interpreter is updated in the command palette.)  I looked into my ~/.bashrc file but there is just the seemingly normal >>> conda initialize block at the bottom that was added when installing Anaconda

Answer: It turns out that I had to delete ~/.vscode-server directory and let it being autogenerated again on the next code . run. I also had to click the "Inherit Env" setting of the built-in Terminal (I forgot, it may not be the default). With these steps, a newly launched VSCode behaves almost exactly like I expected:  If there was no built-in terminal open in the previous VSCode session, opening a new terminal looks like as in the bottom right tab: prompt starts with (base) env but immediately picks up correct end and returns correct prompt if there was a built-in terminal open when killing the previous session, on the next launch the terminal will open automatically with the incorrect (base) env, see bottom left tab, but any subsequent new terminal added will look like the right window, which is now correct. This is still a bit inconvenient, having to either do a manual conda activate <correct_env> or close the terminal and open a fresh one, now with the correct conda env  anyways, I think I can live with this
Question: I want to construct a pandas dataframe where columns have elements from specified arrays with unique elements only. I want to find the most efficient pythonic way of doing this. For example, I have the following numpy arrays as input: a = np.array(["a1"]) b = np.array(["b1", "b2"]) c = np.array(["c1", "c2", "c3"])  Below is how I want my Pandas dataframe to be like  a b c 0 a1 b1 c1 1 a1 b1 c2 2 a1 b1 c3 3 a1 b2 c1 4 a1 b2 c2 5 a1 b2 c3  Below is the code that I am using: import pandas as pd hashmap = {"a":[], "b":[], "c":[]} for a_elem in a: for b_elem in b: for c_elem in c: hashmap["a"] += [a_elem] hashmap["b"] += [b_elem] hashmap["c"] += [c_elem] df = pd.DataFrame.from_dict(hashmap)  How can I make this code more efficient?

Answer: Use itertools.product: from itertools import product df = pd.DataFrame(product(a, b, c), columns=['a', 'b', 'c'])  Output:  a b c 0 a1 b1 c1 1 a1 b1 c2 2 a1 b1 c3 3 a1 b2 c1 4 a1 b2 c2 5 a1 b2 c3  Alternative with MultiIndex.from_product: df = pd.MultiIndex.from_product([a, b, c], names=['a', 'b', 'c'] ).to_frame(index=False)
Question: Given a dataframe like the one below, for each date in Enrol Date, how can I count the number of values in preceding rows in Close Date that are earlier? Ideally I would like to add the results as a new column.    Class Enrol Date Close Date     A 30/10/2003 05/12/2003   A 22/12/2003 23/09/2005   A 06/09/2005 29/09/2005   A 15/11/2005 07/12/2005   A 27/02/2006 28/03/2006    Desired result:    Class Enrol Date Close Date Prior Dates     A 30/10/2003 05/12/2003 0   A 22/12/2003 23/09/2005 1   A 06/09/2005 29/09/2005 1   A 15/11/2005 07/12/2005 3   A 27/02/2006 28/03/2006 4   

Answer: A possible option using triu : cols = ["Enrol Date", "Close Date"] df[cols] = df[cols].apply(pd.to_datetime, dayfirst=True) enrol = df["Enrol Date"].to_numpy() close = df["Close Date"].to_numpy()[:, None] df["Prior Dates"] = np.triu(enrol>close).sum(0)  Output : print(df) Class Enrol Date Close Date Prior Dates 0 A 2003-10-30 2003-12-05 0 1 A 2003-12-22 2005-09-23 1 2 A 2005-09-06 2005-09-29 1 3 A 2005-11-15 2005-12-07 3 4 A 2006-02-27 2006-03-28 4
Question: I'm trying to select all shapes within an existing Visio-file that are i. e. in layer "help" to delete them. Is there a way to achive that using python package "vsdx"?

Answer: You can use CreateSelection method with parameter visSelTypeByLayer = 3. Right now I haven't Python environment there, my simple VBA code Sub SOF_77409035() Dim Sh As Shape Dim sl As Selection ActiveWindow.DeselectAll ' deselect all Set sl = ActivePage.CreateSelection(3, 0, "help") ' create selection in memory For Each Sh In sl ' iterate all shapes in selection ' ActiveWindow.Select Sh, 2 ' select shape visSelect = 2 sh.delete Next End Sub   UPDATE This python-code can print a list of shapes belonging to a layer. But it cannot select these shapes. import win32com.client as w32 visio = w32.Dispatch("visio.Application") visio.Visible = 1 win = visio.activewindow doc = visio.activedocument pag = doc.pages(1) sl = pag.createselection(3, 0,"help") for i in range (1, sl.count+1): sh = sl.item(i) sh.delete  PS vsdx python package have poor documentation, I suspect that's not possible.
Question: I have the following piece of code: import pandas as pd import streamlit as st cashflow = pd.DataFrame(data=[ (2023, 10000), (2022, 95000), (2021, 90000), (2020, 80000), ], columns=['year', 'cashflow']) st.dataframe( data=cashflow, column_config={ 'year': st.column_config.NumberColumn(format='%d'), 'cashflow': st.column_config.NumberColumn(label='cashflow', format='$%.0f'), } )  But, this outputs the cashflow column as $95000 but I want it output $95,000

Answer: Prefer use the Styler instead of the DataFrame: import pandas as pd import streamlit as st cashflow = pd.DataFrame(data=[ (2023, 10000), (2022, 95000), (2021, 90000), (2020, 80000), ], columns=['year', 'cashflow']) st.dataframe( data=cashflow.style.format({'cashflow': '${:,}'}), # <- HERE )  Output:
Question: Recently I came across PEP 683 – Immortal Objects, Using a Fixed Refcount https://peps.python.org/pep-0683/ What I figured out that objects like None, True, False will be shared across interpreter instead of creating copies. These were already immutable, weren't they? Can someone please explain in easy terms. Thanks.

Answer: That PEP is about CPython, which is the reference implementation of Python. As such, it talks about things on a very different level than what most of us are thinking about. CPython keeps track of how many references there are to any given object (the refcount). If you do something like a = some_expression(), the object that is assigned to a gets its refcount increased. Same with something like my_list.append(something). If you clear a list, reassign or delete a variable, or something else like that, CPython decreases the refcount of the objects involved by one. When the refcount reaches 0, it knows it can clean up that object. Now, certain values like None, True, and False are never going to get to 0 as long as the program is running, so they are effectively immortal. But up until 3.12, CPython didn't know that it didn't need to bother changing the refcount for these objects, so it still updated it every time they got a new reference or a reference disappeared. So while the Python-facing part of these objects were immutable, on the C-side, None and objects like it actually have a bit of memory attached to it that was mutable. And that has impacts on performance (outlined in the "Motivation" section of the PEP). This is all very nitty-gritty, and I don't understand the full implications of all this myself, but unless you're writing a C extension, you probably don't need to worry about it.
Question: When I do a curl request for GET endpoint of a REST API, I would go it like this curl -X 'GET' \ 'http://localhost:8000/user/?limit=10' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '[ { "item_type": "user", "skip": 0 } ]'  Directly translated into requests, I would get import requests headers = { 'accept': 'application/json', 'Content-Type': 'application/json', } params = { 'limit': '10' } json_data = [ { 'item_type': 'user', 'skip': 0, }, ] response = requests.get('http://localhost:8000/user/', params=params, headers=headers, json=json_data) # Note: json_data will not be serialized by requests # exactly as it was in the original request. #data = '[\n {\n "item_type": "document",\n "skip": 0\n }\n]' #response = requests.get('http://localhost:8000/user/', params=params, headers=headers, data=data)  Now for my test environment I am going to use httpx's AsyncClient, but I cannot find a way to use the data part, i.e., [{"item_type": "user", "skip": 0}], in a get request. It would be terrific, if you know how to address this.

Answer: According to the httpx docs, .get doesn't support request bodies. It suggests you use the more generic .request function instead.  The HTTP GET, DELETE, HEAD, and OPTIONS methods are specified as not supporting a request body. To stay in line with this, the .get, .delete, .head and .options functions do not support content, files, data, or json arguments.  # possible alternative httpx.request( method="GET", url="https://www.example.com/", content=b'A request body on a GET request.' )  Important note: the HTTP spec states clients should not use body content with GET requests and if they do, there's no guarantee that it will show up at the server (as intermediaries are allowed to strip it off):  Although request message framing is independent of the method used, content received in a GET request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [HTTP/1.1]). A client SHOULD NOT generate content in a GET request unless it is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported. An origin server SHOULD NOT rely on private agreements to receive content, since participants in HTTP communication are often unaware of intermediaries along the request chain
Question: I have a dataframe: [{'Date': Timestamp('2023-01-01 00:00:00'),'Sex':'M', 'Value':11, 'Target':5, 'A':48}, {'Date': Timestamp('2023-01-01 00:00:00'),'Sex':'F', 'Value':25, 'Target':7, 'A':20}, {'Date': Timestamp('2023-01-10 00:00:00'),'Sex':'M', 'Value':45, 'Target':6, 'A':20}, {'Date': Timestamp('2023-01-10 00:00:00'),'Sex':'F', 'Value':5, 'Target':2, 'A':16}, {'Date': Timestamp('2023-01-20 00:00:00'),'Sex':'M', 'Value':10, 'Target':8, 'A':30}] {'Date': Timestamp('2023-01-20 00:00:00'),'Sex':'M', 'Value':1, 'Target':18, 'A':3}]  Date Sex Value Target A 0 2023-01-01 M 11 5 48 1 2023-01-01 F 25 7 20 2 2023-01-10 M 45 6 20 3 2023-01-10 F 5 2 16 4 2023-01-20 M 10 8 30 5 2023-01-20 M 1 18 3  And like to fill up the missing Date: 2023-01-20 Sex: F as 0 to Value, Target and A. Result: Date Sex Value Target A 0 2023-01-01 M 11 5 48 1 2023-01-01 F 25 7 20 2 2023-01-10 M 45 6 20 3 2023-01-10 F 5 2 16 4 2023-01-20 M 10 8 30 5 2023-01-20 M 1 18 3 6 2023-01-20 F 0 0 0 

Answer: You can perform a double merge, once to combine the dates and M/F, then to add the missing combinations to the original data. out = (df[['Date']] .drop_duplicates() .merge(pd.Series(['M', 'F'], name='Sex'), how='cross') .merge(df, how='left').fillna(0).convert_dtypes() )  Alternatively, using janitor's complete: import janitor out = df.complete('Date', {'Sex': ['M', 'F']}, fill_value=0)  Output:  Date Sex Value Target A 0 2023-01-01 M 11 5 48 1 2023-01-01 F 25 7 20 2 2023-01-10 M 45 6 20 3 2023-01-10 F 5 2 16 4 2023-01-20 M 10 8 30 5 2023-01-20 M 1 18 3 6 2023-01-20 F 0 0 0  Another option is to build a new dataframe and run an outer/left merge (similar to what complete does internally) : index = pd.MultiIndex.from_product([df.Date.unique(), df.Sex.unique()], names = ['Date', 'Sex']) index = pd.DataFrame([], index = index) index.merge(df, on=['Date','Sex'],how='left').fillna(0) Date Sex Value Target A 0 2023-01-01 M 11.0 5.0 48.0 1 2023-01-01 F 25.0 7.0 20.0 2 2023-01-10 M 45.0 6.0 20.0 3 2023-01-10 F 5.0 2.0 16.0 4 2023-01-20 M 10.0 8.0 30.0 5 2023-01-20 M 1.0 18.0 3.0 6 2023-01-20 F 0.0 0.0 0.0
Question: I'm suddenly getting flake8 errors for syntax within strings. For example, for the following line of code:  tags.append(f'error_type:{error.get("name")}')  I'm getting this error: E231 missing whitespace after ':'. I don't want to ignore all E231 errors, because I care about them when they don't refer to text within strings. I also don't want to have to go and add # noqa comments to each of my strings. I've tried to pin my flake8 version to 6.0.0 (which is the version that previously didn't raise these errors before). I'm running flake8 with pre-commit (if that is relevant). Why am I suddenly getting these errors for strings and how can I turn them off? I should also mention that this is happening in Github Actions, specifically.

Answer: This issue seems limited to Python 3.12+ and it was fixed by flake8 version 6.1.0. The errors stopped appearing when I upgrade to flake8 6.1.0.
Question: I have the following polars dataframe import polars as pl data = { "product_id": ["1", "2", "3", "4", "5", "6", "7", "8", "9"], "col1": ["a", "a", "a", "a", "a", "a", "a", "a", "a",], "col2": ["b", None, "b", None, "b", None, "b", None, "b"], "col3": ["c", None, "c", None, None, None, None, None, None], "col4": [None, None, None, None, None, "d", None, None, "d"] } df = pl.DataFrame(data)  ┌────────────┬──────┬──────┬──────┬──────┐ │ product_id ┆ col1 ┆ col2 ┆ col3 ┆ col4 │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ str │ ╞════════════╪══════╪══════╪══════╪══════╡ │ 1 ┆ a ┆ b ┆ c ┆ null │ │ 2 ┆ a ┆ null ┆ null ┆ null │ │ 3 ┆ a ┆ b ┆ c ┆ null │ │ 4 ┆ a ┆ null ┆ null ┆ null │ │ 5 ┆ a ┆ b ┆ null ┆ null │ │ 6 ┆ a ┆ null ┆ null ┆ d │ │ 7 ┆ a ┆ b ┆ null ┆ null │ │ 8 ┆ a ┆ null ┆ null ┆ null │ │ 9 ┆ a ┆ b ┆ null ┆ d │ └────────────┴──────┴──────┴──────┴──────┘  I need to fill null values in column col4 with the closest non-null value from the columns to the left (not considering product_id column) and create a new column based on those values. The desired output should be the following data = { "product_id": ["1", "2", "3", "4", "5", "6", "7", "8", "9"], "col1": ["a", "a", "a", "a", "a", "a", "a", "a", "a",], "col2": ["b", None, "b", None, "b", None, "b", None, "b"], "col3": ["c", None, "c", None, None, None, None, None, None], "col4": [None, None, None, None, None, "d", None, None, "d"], "desired_column": ["c", "a", "c", "a", "b", "d", "b", "a", "d"] } df = pl.DataFrame(data)  shape: (9, 6) ┌────────────┬──────┬──────┬──────┬──────┬────────────────┐ │ product_id ┆ col1 ┆ col2 ┆ col3 ┆ col4 ┆ desired_column │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ str ┆ str │ ╞════════════╪══════╪══════╪══════╪══════╪════════════════╡ │ 1 ┆ a ┆ b ┆ c ┆ null ┆ c │ │ 2 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 3 ┆ a ┆ b ┆ c ┆ null ┆ c │ │ 4 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 5 ┆ a ┆ b ┆ null ┆ null ┆ b │ │ 6 ┆ a ┆ null ┆ null ┆ d ┆ d │ │ 7 ┆ a ┆ b ┆ null ┆ null ┆ b │ │ 8 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 9 ┆ a ┆ b ┆ null ┆ d ┆ d │ └────────────┴──────┴──────┴──────┴──────┴────────────────┘  I tried different approaches but didn't have any success. Is there a way to achieve this behaviour within polars?

Answer: It looks like "closest" means to favor the higher numbered columns ie 4,3,2,1. You can do that with just a coalesce df.with_columns( desired_column=pl.coalesce( 'col4', 'col3', 'col2', 'col1' ) ) shape: (9, 6) ┌────────────┬──────┬──────┬──────┬──────┬────────────────┐ │ product_id ┆ col1 ┆ col2 ┆ col3 ┆ col4 ┆ desired_column │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ str ┆ str │ ╞════════════╪══════╪══════╪══════╪══════╪════════════════╡ │ 1 ┆ a ┆ b ┆ c ┆ null ┆ c │ │ 2 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 3 ┆ a ┆ b ┆ c ┆ null ┆ c │ │ 4 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 5 ┆ a ┆ b ┆ null ┆ null ┆ b │ │ 6 ┆ a ┆ null ┆ null ┆ d ┆ d │ │ 7 ┆ a ┆ b ┆ null ┆ null ┆ b │ │ 8 ┆ a ┆ null ┆ null ┆ null ┆ a │ │ 9 ┆ a ┆ b ┆ null ┆ d ┆ d │ └────────────┴──────┴──────┴──────┴──────┴────────────────┘  You can do it more dynamically with a generator: pl.coalesce(x for x in df.columns[::-1] if x!='product_id')