eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Given a Polars dataframe like below, how can I call explode() on both columns while expanding the null entry to the correct length to match up with its row? shape: (3, 2) ┌───────────┬─────────────────────┐ │ x ┆ y │ │ --- ┆ --- │ │ list[i64] ┆ list[bool] │ ╞═══════════╪═════════════════════╡ │ [1] ┆ [true] │ │ [1, 2] ┆ null │ │ [1, 2, 3] ┆ [true, false, true] │ └───────────┴─────────────────────┘  Currently calling df.explode(["x", "y"]) will result in this error. polars.exceptions.ShapeError: exploded columns must have matching element counts  I'm assuming there's not a built-in way. But I can't find/think of a way to convert that null into a list of correct length, such that the explode will work. Here, the required length is not known statically upfront. I looked into passing list.len() expressions into repeat_by(), but repeat_by() doesn't support null.

Answer: You were on the right track, trying to fill the missing values with a list of null values of correct length. To make pl.Expr.repeat_by work with null, we need to ensure that the base expression is of a non-null type. This can be achieved by setting the dtype argument of pl.lit explicity. Then, the list column of (lists of) nulls can be used to fill the null values in y. From there, exploding x and y simultaneously works as usually. ( df .with_columns( pl.col("y").fill_null( pl.lit(None, dtype=pl.Boolean).repeat_by(pl.col("x").list.len()) ) ) )  shape: (3, 2) ┌───────────┬─────────────────────┐ │ x ┆ y │ │ --- ┆ --- │ │ list[i64] ┆ list[bool] │ ╞═══════════╪═════════════════════╡ │ [1] ┆ [true] │ │ [1, 2] ┆ [null, null] │ │ [1, 2, 3] ┆ [true, false, true] │ └───────────┴─────────────────────┘  From here, df.explode("x", "y") should work as expected. Note. If there are more than two columns, which all might contain null values, one can combine the answer above with this answer to have a valid solution. Note.
Question: I have the following conditions:  The number u(0) = 1 is the first one in u. For each x in u, then y = 2 * x + 1 and z = 3 * x + 1 must be in u also. There are no other numbers in u. No duplicates should be present. The numbers must be in ascending sequential order  Ex: u = [1, 3, 4, 7, 9, 10, 13, 15, 19, 21, 22, 27, ...] The program wants me to give the value of a member at a given index. I've already found ways to solve this using insort and I hacked together a minimal binary search tree, as well. Unfortunately, this process needs to be faster than what I have and I'm at a loss as to what to do next. I thought the BST would do it, but it is not fast enough. Here's my BST code: class BSTNode: def __init__(self, val=None): self.left = None self.right = None self.val = val def insert(self, val): if not self.val: self.val = val return if self.val == val: return if val < self.val: if self.left: self.left.insert(val) return self.left = BSTNode(val) return if self.right: self.right.insert(val) return self.right = BSTNode(val) def inorder(self, vals): if self.left is not None: self.left.inorder(vals) if self.val is not None: vals.append(self.val) if self.right is not None: self.right.inorder(vals) return vals  and here's my function: from sys import setrecursionlimit def twotimeslinear(n): #setrecursionlimit(2000) i = 0 u = [1] ended = False bst = BSTNode() bst.insert(1) while i < n and not ended: for j in range(2, 4): k = 1 cur = j * bst.inorder([])[i] + 1 bst.insert(cur) if len(u) == n: ended = True break i+= 1 return bst.inorder([])[n]  I just need directions as to what I could do to make the process faster. I can solve the problem if I only knew what I was missing. I'm probably overlooking some data structure that would work better, but I don't even know what to look for. Thank you for any help.

Answer: Generating and merging the ys and zs: from heapq import merge from itertools import groupby def twotimeslinear(n): u = [1] ys = (2*x+1 for x in u) zs = (3*x+1 for x in u) for x, _ in groupby(merge(ys, zs)): u.append(x) if n < len(u): return u[n] print(*map(twotimeslinear, range(20)))  Attempt This Online! Takes ~0.05 seconds for your limit index 60,000 and ~0.7 seconds for index 1,000,000. Alternative basic implementation: def twotimeslinear(n): u = [1] i = j = 0 while n >= len(u): y = 2*u[i] + 1 z = 3*u[j] + 1 m = min(y, z) u.append(m) if y == m: i += 1 if z == m: j += 1 return u[n] print(*map(twotimeslinear, range(20)))  Attempt This Online! And another version, throwing tons of batteries at it :-) from heapq import merge from itertools import groupby, tee, chain, islice, repeat from operator import itemgetter, mul, add def twotimeslinear(n): parts = [[1]] whole = chain.from_iterable(parts) output, feedback1, feedback2 = tee(whole, 3) ys = map(1 .__add__, map(2 .__mul__, feedback1)) zs = map(add, map(mul, repeat(3), feedback2), repeat(1)) # different way just for fun merged = map(itemgetter(0), groupby(merge(ys, zs))) parts.append(merged) return next(islice(output, n, None)) print(*map(twotimeslinear, range(20)))  After setting it up with Python code, this runs entirely in C code. It's a silly hobby of mine. Attempt This Online!
Question: EDIT: Edited original post replacing my code with an MRE from user @no comment. I noticed a seemingly non-intuitive behaviour using a seeded random.sample(population, k) call to sample from a list of files in a directory. I would expect that sampling 100 items with k=100 while using a seed, guarantees that when I subsequently sample with k=800, the 100 sampled items are guaranteed to be within the list of 800 sampled items. Below is an example for a number of population sizes and seeds: import random for n in 1000, 2000, 5000: print(end=f'{n=}: ') for i in range(10): random.seed(i) x = random.sample(range(n), 100) random.seed(i) y = random.sample(range(n), 800) print(sum(i==j for i,j in zip(x,y[:100])), end=' ') print()  The output shows the expected perfect match when sampling from a population of 1000 or 5000, but somehow randomly only a partial match when sampling from 2000: n=1000: 100 100 100 100 100 100 100 100 100 100 n=2000: 93 61 38 44 68 36 39 33 71 86 n=5000: 100 100 100 100 100 100 100 100 100 100  Similarly, after running my sampling script in some directories, I get mixed results. From some directories, the 100 are within the 800, but for others there are differences for a few files, some are here and not there and vice versa. Are there any sources of randomness that I am not aware of? I tried sorting the os.listdir() call that lists directory contents but this didn't change anything. I know I can refactor the script to work by first sampling a greater value and slicing the sample list, but I would expect my original script to work in the same way too.

Answer: The exact algorithm isn't documented/guaranteed. You're likely using CPython, which chooses one of two slightly different algorithms for efficiency, depending on whether you want a small or large percentage of the population. Both algorithms select one element at a time and append it to the sample, but they differ in how to select the next element and how to keep track of the selected or remaining elements. Each of the two algorithms actually has the property you desire. But what can happen is that you sample 100 with one algorithm and 800 with the other, and then they don't match.  With n=1000, sampling 100 or 800 both use the first algorithm (the one for large percentages). Hence the perfect match of the first 100. With n=2000, sampling 100 or 800 use different algorithms (sampling 100 is now a small percentage). Hence the difference. With n=5000, sampling 100 or 800 both use the second algorithm (the one for small percentages). Hence the perfect match of the first 100.  (If you noticed and are wondering why 100 out of 1000 is a "large percentage" while 800 out of 5000 is a "small percentage": It's not a fixed percentage, see the implementation for the formula.)
Question: I have a Pandas Series that has these unique values: array(['17', '19', '21', '20', '22', '23', '12', '13', '15', '24', '25', '18', '16', '14', '26', '11', '10', '12/16', '27', '10/14', '16/22', '16/21', '13/17', '14/19', '11/15', '10/15', '15/21', '13/19', '13/18', '32', '28', '12/15', '29', '42', '30', '31', '34', '46', '11/14', '18/25', '19/26', '17/24', '19/24', '17/23', '13/16', '11/16', '15/20', '36', '17/25', '19/25', '17/22', '18/26', '39', '41', '35', '50', '9/13', '33', '10/13', '9/12', '93/37', '14/20', '10/16', '14/18', '16/23', '37', '9/11', '37/94', '20/54', '22/31', '22/30', '23/33', '44', '40', '50/95', '38', '16/24', '15/23', '15/22', '18/23', '16/20', '37/98', '19/27', '38/88', '23/31', '14/22', '45', '39/117', '28/76', '33/82', '15/19', '23/30', '47', '46/115', '14/21', '17/18', '25/50', '12/18', '12/17', '21/28', '20/27', '26/58', '22/67', '22/47', '25/51', '35/83', '39/86', '31/72', '24/56', '30/80', '32/85', '42/106', '40/99', '30/51', '21/43', '52', '56', '25/53', '34/83', '30/71', '27/64', '35/111', '26/62', '32/84', '39/95', '18/24', '22/29', '42/97', '48', '55', '58', '39/99', '49', '43', '40/103', '22/46', '54/133', '25/54', '36/83', '29/72', '28/67', '35/109', '25/62', '14/17', '42/110', '52/119', '20/60', '46/105', '25/56', '27/65', '25/74', '21/49', '29/71', '26/59', '27/62'], dtype=object)  The ones that have the '/', I want to split these into arrays and then average their values. One simpler but a flawed approach is to simply extract the first value: master_data["Cmb MPG"].str.split('/').str[0].astype('int8') However, what I truly require is the two values being averaged. I have tried several commands and this one: np.array(master_data["Cmb MPG"].str.split('/')).astype('int8').mean()  Should ideally do the job, but I get a ValueError followed by a TypeError: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list' The above exception was the direct cause of the following exception: ValueError Traceback (most recent call last) Cell In[88], line 1 ----> 1 np.array(master_data["Cmb MPG"].str.split('/')).astype('int8') ValueError: setting an array element with a sequence.  The slice() method returns a Series but it won't proceed either with the splitting of strings. What is required is: '18/25' ---> [18, 25] ---> 22 (rounded) 

Answer: I would use extractall and groupby.mean: s = pd.Series(['10', '12/16', '27', '10/14', '16/22', '16/21', '13/17']) out = (s.str.extractall(r'(\d+)')[0].astype(int).groupby(level=0).mean() .round().astype(int) )  You could also go with split and mean, but this generates a more expensive intermediate and will not scale as well if you have many items (1/2/3/4/5): out = (s.str.split('/', expand=True).astype(float).mean(axis=1) .round().astype(int) )  Output: 0 10 1 14 2 27 3 12 4 19 5 18 6 15 dtype: int64
Question: Recently, I discovered the really weird behaviour of my an AI model. I wanted to build AI model that would try and guess the implicit functions based on the data I gave it. For example, the equation of the flower is:  And this is how I wrote it in the Numpy array: K = 0.75 a = 9 def f(x): return x - 2 - 2*np.floor((x - 1)/2) def r1(x): return abs(6 * f(a * K * x / 2 / np.pi)) - a/2 t = np.linspace(0, 17 * math.pi, 1000) x = np.cos(t) * r1(t) y = np.sin(t) * r1(t) points = np.vstack((x, y)).T  After that, I tried to experiment a bit and allowed my AI to actually try and guess the shape of this flower! At first try, it actually got it written. Here it is:  Well, I got a great example. After that, I tried to experiment and checked what would happen if I shuffled the point array, and I completely got devasting results!  And I couldn't explain why the order of cartesian coordinates mattered to the approximate flower implicit function. Can anyone explain it? Here is the code for AI. # Define the neural network model model = Sequential() model.add(Dense(128, input_dim=2, activation='relu')) model.add(Dense(64, activation='relu')) model.add(Dense(2, activation='linear')) # Compile the model model.compile(optimizer='adam', loss='mean_squared_error') # Train the model model.fit(points, points, epochs=100, batch_size=32) 

Answer: Actually, it's a plotting issue. The order in a plot matters while the one of a scatter don't :  import matplotlib.pyplot as plt shuffled_points = np.random.permutation(points) plt.scatter(*points.T) # in .scatter()-method POINT-s are depicted plt.scatter(*shuffled_points.T) # in .scatter()-method order does NOT matter plt.plot(*points.T, color="r") # in .plot()-method interpolator follows the order plt.plot(*shuffled_points.T, color="r") # method creates a SEQUENCE of LINEs
Question: I want to find all 5-digit strings, for which  the first three digits are in my first list, the second trough fourth are in my second and the third to fifth are in my last list:  l0=["123","567","451"] l1=["234","239","881"] l2=["348","551","399"]  should thus yield: ['12348', '12399']. I have therefore written a function is_successor(a,b) that tests if a and b overlap: def is_successor(a:str,b:str)->bool: """tests if strings a and b overlap""" return a[1:]==b[:2]  I can then achieve my goal by writing this nested loop/check structure, that basically appends back to front strings and resulting in all valid strings: pres=[] for c in l2: for b in l1: if is_successor(b,c): for a in l0: if is_successor(a,b): pres.append(a[0]+b[0]+c) pres   I know I could write it as a list comprehension, but for my original data i have more nested lists and i loose readability even in a list comprehension. I start from l2 -> l0 because in my original data the lists become longer, the lower the index and thus I can filter out more cases early this way. A single loop trough all combinations of l0,l1,l2 and checking the succession of all items a,b,c simultaneously would work but it test way more unnecessary combinations than my current construct does.  Question: How can this nested loop and conditional checking call be abstracted? Is there a pythonic way to capture the repetition of for -> is_successor()? A larger input might be: primes = [2, 3, 5, 7, 11, 13, 17] lsts=[ [ str(j).zfill(3) for j in range(12,988) if not j%prime ] for prime in primes ] 

Answer: I would use itertools product to define a function that will perform the check annd append the data: import itertools def get_list(l1,l2): #THIS FUNCTION WILL TEST ALL THE PRODUCTS AND RETURN ONLY THE CORRECT ONES return [a+b[2:] for a,b in list(itertools.product(l1,l2)) if a[1:] == b[:2]]  then you can nest the function like this: get_list(l0,get_list(l1,l2))  you will get the expected result: ['12348', '12399'] EDIT: for a long list use reduce. let s call l the list of list of inputs. In your specific case l = list([l0,l1,l2]) Reverse the list to apply from the end till the beginning l.reverse() and the final code: from functools import reduce reduce(lambda x,y:get_list(x,y),l)  Note: in this case you want to change the inputs of the "get_list" function as it will go l2 then l1 then l0.Just change the def of get_list like this: def get_list(l2,l1)
Question: Examples:  For 0123123123, 1 should be matched since the 2nd 1 appears before the repetition of any other digit. For 01234554321, 5 should be matched since the 2nd 5 appears before the repetition of any other digit.  Some regexes that I have tried:  The below works for the 1st but not the 2nd example. It matches 1 instead because 1 is the first digit that appears in the string which is subsequently repeated.  import re m = re.search(r"(\d).*?\1", string) print(m.group(1))   The below works for the 2nd but not the 1st example. It matches 3 instead - in particular the 2nd and 3rd occurrence of the digit. I do not know why it behaves that way.  import re m = re.search(r"(\d)(?!(\d).*?\2).*?\1", string) print(m.group(1)) 

Answer: One idea: capture the end of the string and add it in the negative lookahead (group 2 here): (\d)(?=.*?\1(.*))(?!.*?(\d).*?\3.+?\2$)  This way you can control where the subpattern .*?(\d).*?\3 in the negative lookahead ends. If .+?\2$ succeeds, that means there's an other digit that is repeated before the one in group 1. I anchored the pattern for the regex101 demo with ^.*?, but you don't need to do that with the re.search method.  Other way: reverse the string and find the last repeated digit: re.search(r'^.*(\d).*?\1', string[::-1]).group(1)
Question: I have a shiny app that displays a number of layers on a map using folium. I want to give the user the possibility to download one of the layers (a linestring geodataframe) as a geopackage file. Here is the code I have so far: # relevant imported packages from shiny import App, render, ui, reactive import pandas as pd import geopandas as gpd # code in the ui for the download button ui.download_button(id="downloadRoute", label="Get the route file"), # code in the server function @output @render.download def downloadRoute(): result_value = result() route = result_value['route_detailed'] with io.BytesIO() as buffer: route.to_file(buffer, driver='GPKG') buffer.seek(0) return ui.download(filename="my_route.gpkg", file=buffer.read(), content_type="application/geopackage+sqlite3")  I have verified that route is actually a valid geodataframe. If I download it outside shiny, it is a valid geopackage. In shiny, clicking the download button doesn't do anything on the UI. It only prints this error in the console: 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "pyogrio\_io.pyx", line 1268, in pyogrio._io.ogr_create File "pyogrio\_err.pyx", line 177, in pyogrio._err.exc_wrap_pointer pyogrio._err.CPLE_OpenFailedError: sqlite3_open(<_io.BytesIO object at 0x000002494BFCE3E0>) failed: unable to open database file  What could be the thing I am doing wrong? Are there other ways I can achieve this?

Answer: As a side note, the @output decorator is no longer necessary since v0.6.0. And regarding the download issue, you (must?) yield the value of the buffer in the render.download, something like below : from io import BytesIO from shiny import App, render, ui import geopandas as gpd from shapely import LineString app_ui = ui.page_fluid( ui.download_button(id="downloadRoute", label="Download as GPKG") ) route = gpd.GeoDataFrame(geometry=[LineString([(0, 0), (1, 0)])]) def server(input): @render.download( filename="file.gpkg", # media_type="application/geopackage+sqlite3", # optional ) def downloadRoute(): with BytesIO() as buff: route.to_file(buff, driver="GPKG") yield buff.getvalue() app = App(app_ui, server)  Demo at shinylive:
Question: I have a dataframe (df1) with unique IDs (col1) but also duplicated IDs for each row (col2). On another dataframe (df2) I have unique IDs of col2. I want to somehow link the two dataframes together without copying the data of df2 (col3) (linking instead of copying). Here is an simplified example of what I try to achieve : df1 = pd.read_clipboard() col1 col2 0 a A 1 b A 2 c B 3 d C 4 e B df2 = pd.read_clipboard() col2 col3 0 A Albert is the best 1 B Bernard is cool 2 C Conrad is ok df3 # col3 being the values of df2, not copied col1 col2 col3 0 a A Albert is the best 1 b A Albert is the best 2 c B Bernard is cool 3 d C Conrad is ok 4 e B Bernard is cool  I've been using pd.merge and it creates the df3 I want but uses too much memory, I think that it copies the values to the new tables but it's useless as values are only in a few states. df3 = pd.merge(df1, df2, on='col2', how='left')  There is a copy=False attribute I can add but doesn't change anything, it stills seems to copy the data.

Answer: I would suggest to convert the non-key columns to Categorical, which should save a lot of memory if you have many duplicates: out = pd.merge(df1.astype({'col1': 'category'}), df2.astype({'col3': 'category'}), on='col2', how='left')  Or, for simplicity: out = pd.merge(df1.astype('category'), df2.astype('category'), on='col2', how='left')  But col2 won't be kept as a categorical. Output:  col1 col2 col3 0 a A Albert is the best 1 b A Albert is the best 2 c B Bernard is cool 3 d C Conrad is ok 4 e B Bernard is cool  If you want to convert all object columns automatically and leave the numeric ones unchanged: # example let's add numeric columns df1['num1'] = 1 df2['num2'] = 2 out = pd.merge(df1.astype(dict.fromkeys(df1.select_dtypes('object'), 'category')), df2.astype(dict.fromkeys(df2.select_dtypes('object'), 'category')), on='col2', how='left') out.dtypes # col1 category # col2 object # num1 int64 # col3 category # num2 float64 # dtype: object
Question: In numpy.unique there is an option return_index=True - which returns positions of unique elements (first occurrence - if several). Unfortunately, there is no such option in torch.unique ! Question: What are the fast and torch-style ways to get indexes of the unique elements ? ===================== More generally my issue is the following: I have two vectors v1, v2, and I want to get positions of those elements in v2, which are not in v1 and also for repeated elements I need only one position. Numpy's unique with return_index = True immediately gives the solution. How to do it in torch ? If we know that vector v1 is sorted, can it be used to speed up the process ?

Answer: You can achieve this in PyTorch with the following approach: def get_unique_elements_first_idx(tensor): # sort tensor sorted_tensor, indices = torch.sort(tensor) # find position of jumps unique_mask = torch.cat((torch.tensor([True]), sorted_tensor[1:] != sorted_tensor[:-1])) return indices[unique_mask]  Example usage: v1 = torch.tensor([2, 3, 3]) v2 = torch.tensor([1, 2, 6, 2, 3, 10, 4, 6, 4]) # Mask to find elements in v2 that are not in v1 mask = ~torch.isin(v2, v1) v2_without_v1 = v2[mask] # Get unique elements and their first indices unique_indices = get_unique_elements_first_idx(v2_without_v1) print(unique_indices) #[0, 3, 1, 2] print(v2[mask][unique_indices]) #[1, 4, 6, 10]  P.S. On my computer, the function processes a vector of size 10 million in about (1.1 ± 0.1)s.
Question: I seem to be experiencing an issue with the following Python Scheduler tasks. I have the below code which has a function to refresh all workbooks in a set bunch of directories. I have attempted to test the code only refreshing the one directory before moving onto the others and I am receiving the titled error. Is anyone able to help? import time import schedule import win32com.client as win32 import os OD = 'C:/DirectoryString/' # Sub Directory Dir1 = 'STR - Doc/Mer Reps/' Dir2 = 'UK & IE ST - ICS/DB/' Dir3 = 'UK & IE ST - Docs/DB/2024/' Dir4 = 'UK & IE RDF - Docs/Rep/DB/Aut/' Dir5 = 'STR - Doc/RMS/' Dir6 = 'STR - Doc/Int DB/' directory_1 = OD + Dir1 directory_2 = OD + Dir2 directory_3 = OD + Dir3 directory_4 = OD + Dir4 directory_5 = OD + Dir5 directory_6 = OD + Dir6 def excel_update(directory): for filename in os.listdir(directory): if filename.endswith('.xlsx') or filename.endswith('.xlsm'): print(f"{directory} {filename}") file = filename xlapp = win32.DispatchEx('Excel.Application') xlapp.DisplayAlerts = False xlapp.Visible = True xlbook = xlapp.Workbooks.Open(directory + file) xlbook.RefreshAll() xlbook.Save() xlbook.Close() del xlbook xlapp.Quit() del xlapp schedule.every(30).minutes.do(excel_update(directory_2)) while True: schedule.run_pending() time.sleep(1)  The initial job runs, but, when it comes to the first repetition, I get the following error. Traceback (most recent call last): File "C:\Users\gat_mbr\OneDrive - PurmoGroup\Projects\pythonProject\MerchantReports\LoopFunction.py", line 45, in <module> schedule.every(30).minutes.do(excel_update(directory_2)) File "C:\Users\gat_mbr\OneDrive - PurmoGroup\Projects\pythonProject\MerchantReports\Lib\site-packages\schedule\__init__.py", line 655, in do self.job_func = functools.partial(job_func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: the first argument must be callable 

Answer: do() accepts a function object and its arguments. Your code calls it with the result of excel_update(directory_2) call, which is not a function, but a good old None. Here's how to fix it: schedule.every(30).minutes.do(excel_update, directory_2))  See Pass arguments to a job