eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working in databricks where I have a dataframe as follows: dummy_df names items Ash [c1,c2,c2,c3] Bob [c1,c2] May [] Amy [c2,c3,c3]  Where names column contains strings for values and items is a column of arrays. I would like to count how many times each item appears for each name and make each count into a column. So the desired output will look something like so: names items c1_count c2_count c3_count Ash [c1,c2,c2,c3] 1 2 1 Bob [c1,c2] 1 1 0 May [] 0 0 0 Amy [c2,c3,c3] 0 1 2  So far my approach is to count each item separately using aggregate as so: c1_count = dummy_df.select("*", explode("items").alias("exploded"))\ .where(col("exploded").isin(['c1']))\ .groupBy("names", "items")\ .agg(count("exploded").alias("c1_count")) c2_count = ... c3_count = ...  And then I concatenate a new df by taking count columns from my new 3 dataframes and adding them to my dummy_df. But that is very inefficient and if I were to have many items in my arrays (say 50-100), it may even be impractical. I wonder if there is a better way? Can I somehow calculate the count of all items and make such counts into the columns without needing to count them individually and do massive concatenation as the end?

Answer: Explode the column items and then use crosstab to get the number of items for each name. exploded_df = dummy_df.explode('items') crosstab_df = (pd.crosstab(exploded_df['names'], exploded_df['items'], dropna=False) .drop(columns=np.nan) .add_prefix('Count_') .reset_index() ) new_df = df.merge(crosstab_df, on='names')  Output:  names items Count_c1 Count_c2 Count_c3 0 Ash [c1, c2, c2, c3] 1 2 1 1 Bob [c1, c2] 1 1 0 2 May [] 0 0 0 3 Amy [c2, c3, c3] 0 1 2  This is the data I used: data = {'names': ['Ash', 'Bob', 'May', 'Amy'], 'items': [['c1', 'c2', 'c2', 'c3'], ['c1', 'c2'], [], ['c2', 'c3', 'c3']]} dummy_df = pd.DataFrame(data)
Question: I have a DataFrame of events (Event Name - Time) and a DataFrame of time windows (Start Time - End Time). I want to get a DataFrame containing only the events not in any of the time windows. I am looking for a "pythonic" way to filter the DataFrame. Example: Events DataFrame:    Event Name Event Time     Event1 02/01/2000 00:00:00   Event2 05/01/2000 10:00:00   Event3 07/01/2000 09:00:00   Event4 10/01/2000 02:00:00    Time Windows DataFrame:    Time Window Name Start Time End Time     Window1 01/01/2000 00:00:00 06/01/2000 00:00:00   Window2 10/01/2000 01:00:00 10/01/2000 04:00:00    Result: Filtered Events DataFrame:    Event Name Event Time     Event3 07/01/2000 09:00:00    Setup: import pandas as pd events_data = { 'Event Name': ['Event1', 'Event2', 'Event3', 'Event4'], 'Event Time': ['02/01/2000 00:00:00', '05/01/2000 10:00:00', '07/01/2000 09:00:00', '10/01/2000 02:00:00'] } time_windows_data = { 'Time Window Name': ['Window1', 'Window2'], 'Start Time': ['01/01/2000 00:00:00', '10/01/2000 01:00:00'], 'End Time': ['06/01/2000 00:00:00', '10/01/2000 04:00:00'] } events_df = pd.DataFrame(events_data) time_windows_df = pd.DataFrame(time_windows_data) events_df['Event Time'] = pd.to_datetime(events_df['Event Time'], format='%d/%m/%Y %H:%M:%S') time_windows_df['Start Time'] = pd.to_datetime(time_windows_df['Start Time'], format='%d/%m/%Y %H:%M:%S') time_windows_df['End Time'] = pd.to_datetime(time_windows_df['End Time'], format='%d/%m/%Y %H:%M:%S') 

Answer: You can build an IntervalIndex then create a boolean mask with reindex: # build IntervalIndex idx = pd.IntervalIndex.from_arrays(df_time['Start Time'], df_time['End Time']) # build boolean mask m = (pd.Series(False, index=idx) .reindex(df_events['Event Time'],fill_value=True) .to_numpy() ) # select non-matching rows out = df_events[m]  Alternative to build m: m = idx.reindex(df_events['Event Time'])[1] == -1  Output:  Event Name Event Time 2 Event3 2000-01-07 09:00:00  Intermediates: # idx IntervalIndex([(2000-01-01 00:00:00, 2000-01-06 00:00:00], (2000-01-10 01:00:00, 2000-01-10 04:00:00]], dtype='interval[datetime64[ns], right]') # m array([False, False, True, False])  Reproducible inputs: import pandas as pd from pandas import Timestamp df_events = pd.DataFrame({'Event Name': ['Event1', 'Event2', 'Event3', 'Event4'], 'Event Time': [Timestamp('2000-01-02 00:00:00'), Timestamp('2000-01-05 10:00:00'), Timestamp('2000-01-07 09:00:00'), Timestamp('2000-01-10 02:00:00')]}) df_time = pd.DataFrame({'Time Window Name': ['Window1', 'Window2'], 'Start Time': [Timestamp('2000-01-01 00:00:00'), Timestamp('2000-01-10 01:00:00')], 'End Time': [Timestamp('2000-01-06 00:00:00'), Timestamp('2000-01-10 04:00:00')]})
Question: I have a large library with many doctests. All doctests pass on my computer. When I push changes to GitHub, GitHub Actions runs the same tests in Python 3.8, 3.9, 3.10 and 3.11. All tests run correctly on on Python 3.8; however, on Python 3.9, 3.10 and 3.11, I get many errors of the following type: Expected: [13.0, 12.0, 7.0] Got: [np.float64(13.0), np.float64(12.0), np.float64(7.0)]  I.e., the results are correct, but for some reason, they are displayed inside "np.float64". In my code, I do not use np.float64 at all, so I do not know why this happens. Also, as the tests pass on my computer, I do not know how to debug the error, and it is hard to produce a minimal working example. Is there a way I can make the doctests pass again, without changing each individual test?

Answer: This is due to a change in how scalars are printed in numpy 2: numpy 1.x.x: >>> repr(np.array([1.0])[0]) '1.0'  numpy 2.x.x: >>> repr(np.array([1.0])[0]) 'np.float64(1.0)'  You should restrict the version of numpy to be 1.x.x in your requirements file to make sure you don't end up installing numpy 2.x.x: numpy ~> 1.26  (same as numpy >= 1.26, == 1.*, see this answer) or update your code to work with numpy 2 and change it to numpy ~> 2.0.
Question: Hi I have a question related to model inheritance and accessing the fields in a Django template. My Model: class Security(models.Model): name = models.CharField(max_length=100, blank=False) class Stock(Security): wkn = models.CharField(max_length=15) isin = models.CharField(max_length=25) class Asset(models.Model): security = models.ForeignKey(Security, on_delete=models.CASCADE,blank=False)  My views.py context["assets"] = Asset.objects.all()  My template: {% for asset in assets %} {{ asset.security.wkn }} ...  This gives me an error as wkn is no field of security. Any idea how I can solve this? Thanks in advance!

Answer: Django does not support polymorphism out of the box: even if you use inheritance and you retrieve an item from the database that is a Stock, if you query through Security, you retrieve it as a Security object, so without the specific Stock fields. Inheritance and polymorphism are often a pain in relational databases, and therefore it is often something you should not do, unless there are very good reasons to do so. In this case, you can fetch it with: {{ asset.security.stock.wkn }} But this will thus make extra queries, and generate an N+1 problem. You can work with django-polymorphic [readthedocs.io] where you can subclass from PolymorphicModel. Queries will then walk down the class hierarchy and add a lot of extra LEFT OUTER JOINs to the database, and then based on the data it receives generate a Stock or a Security object. But as you probably understand, this querying generates a lot of extra work for the database and for Python. Furthermore it makes behavior less predictable, and constraints less easy to enfroce.
Question: I'm trying to transform an image into a matrix of it's rbg values in c++, i really like the simplicity of PIL on handling different images extensions, so i currently have two codes from PIL import Image img=Image.open("oi.png") pixel=img.load() width,height=img.size print(height) print(width) def rgb(r, g, b): return ((r & 0xff) << 16) + ((g & 0xff) << 8) + (b & 0xff) for x in range(width): for y in range(height): print(rgb(pixel[x,y][0],pixel[x,y][1],pixel[x,y][2]))  and to recieve in C++ #include <bits/stdc++.h> using namespace std; #define __ ios_base::sync_with_stdio(false);cin.tie(NULL); int main(){__ long long height,width;cin>>height>>width; unsigned long img[width][height]; for(long long j=0; j<height;j++) for (long long i=0; i<width;i++){ cin>>img[i][j]; } return 0;}  and i am connecting both trough terminal withpython3 code.py | ./code it works for really small images, but for bigger ones it returns BrokenPipeError: [Errno 32] Broken pipe What should i do? is there a better way to achieve what i am trying to achieve? I want to connect python output to c++ input even with big outputs without Broken pipe error

Answer: "Broken pipe" means that a program tried to write to a pipe that no longer had any programs reading from it. This means that your C++ program is exiting before it should. For a 1000x1000 image, assuming you're running this on x86_64 Linux, img is 8MB. I suspect that's too big for the stack, which is causing your C++ program to crash. You can fix that by allocating img on the heap instead.
Question: I have a pandas dataframe with 25 rows, and also a list with 5 elements. How do I: - assign 1st element of the list to first row of the dataframe - 2nd element of the list to second row - ... - 1st element to 6th row of dataframe etc  Eg: Need to assign a doctor to each patient sequentially df: | Name | Gender | | -------- | -------------- | | First | Male | | Second | Female | | Third | Male | | Fourth | Male | | Fifth | Male |  list_doctor = ['andrea','anup'] Required Output: | Name | Gender | Doctor | | -------- | -------------- |---------| | First | Male |andrea | | Second | Female |anup | | Third | Male |andrea | | Fourth | Male |anup | | Fifth | Male |andrea |  Tried with iterrows, but all rows are being assigned to the same name for index, row in df.iterrows(): for doctor in l_doctor: print(index, doctor) df.loc[index,'Assignee'] = doctor 

Answer: Approach To avoid Python for loop we can use itertools functions cycle and islice as follows:  cycle to make an iterator that will indefinitely loop through a list islice to create an iterator that returns selected elements from the iterable list to instantiate the iterator elements  Code import pandas as pd from itertools import cycle, islice df = pd.DataFrame({ 'Name': ['First', 'Second', 'Third', 'Fourth', 'Fifth'], 'Gender': ['Male', 'Female', 'Male', 'Male', 'Male'] }) docs = ['andrea', 'anup'] df['Doctor'] = list(islice(cycle(docs), 0, len(df))) print(df)  Output  Name Gender Doctor 0 First Male andrea 1 Second Female anup 2 Third Male andrea 3 Fourth Male anup 4 Fifth Male andrea
Question: I have a somewhat peculiar structure of python list of lists that I need to convert to a numpy array, so far I have managed to simply get by using np.array(myarray, dtype = object), however a seemingly insignificant change to the structure of myarray has caused me to get an error. I have managed to reduce my issue down into two lines of code, the following is what I was using previously and works exactly how I want it to: import numpy as np myarray = [np.array([[1,2,3,4],[5,6,7,8]]), np.array([[9,10],[11,12]]), np.array([[13,14],[15,16],[17,18]])] np.array(myarray,dtype = object)  However, simply removing the last [17,18] array we have import numpy as np myarray = [np.array([[1,2,3,4],[5,6,7,8]]), np.array([[9,10],[11,12]]), np.array([[13,14],[15,16]])] np.array(myarray,dtype = object)  Which gives "ValueError: could not broadcast input array from shape (2,4) into shape (2,)" when it attempts to run the second line. It seems to me that this only happens when the arrays all have the same length but the underlying lists have different lengths, what I don't understand is why setting dtype = object doesnt cover this especially considering it handles the more complicated list of lists shape.

Answer: np.array tries, as first priority, to make a n-d numeric array - one where all elements are numeric, and the shape is consistent in all dimensions. i.e. no 'ragged' array. In [36]: alist = [np.array([[1,2,3,4],[5,6,7,8]]), np.array([[9,10],[11,12]]), np.array([[13,14],[15,16],[17,18]])] In [38]: [a.shape for a in alist] Out[38]: [(2, 4), (2, 2), (3, 2)]  alist works making a 3 element array of arrays. Your problem case: In [39]: blist = [np.array([[1,2,3,4],[5,6,7,8]]), np.array([[9,10],[11,12]]), np.array([[13,14],[15,16]])] In [40]: [a.shape for a in blist] Out[40]: [(2, 4), (2, 2), (2, 2)]  Note that all subarrays have the same first dimension. That's what's giving the problem. The safe way to make such an array is to start with a 'dummy' of the right shape, and fill it: In [41]: res = np.empty(3,object); res[:] = blist; res Out[41]: array([array([[1, 2, 3, 4], [5, 6, 7, 8]]), array([[ 9, 10], [11, 12]]), array([[13, 14], [15, 16]])], dtype=object) In [42]: res = np.empty(3,object); res[:] = alist; res Out[42]: array([array([[1, 2, 3, 4], [5, 6, 7, 8]]), array([[ 9, 10], [11, 12]]), array([[13, 14], [15, 16], [17, 18]])], dtype=object)  It also works when all subarrays/lists have the same shape In [43]: clist = [np.array([[1,2],[7,8]]), np.array([[9,10],[11,12]]), np.array([[13,14],[15,16]])] In [44]: res = np.empty(3,object); res[:] = clist; res Out[44]: array([array([[1, 2], [7, 8]]), array([[ 9, 10], [11, 12]]), array([[13, 14], [15, 16]])], dtype=object)  Without that clist produces a (3,2,2) array of number objects: In [45]: np.array(clist, object) Out[45]: array([[[1, 2], [7, 8]], [[9, 10], [11, 12]], [[13, 14], [15, 16]]], dtype=object)  One way to think of it, np.array does not give you a way of specifying the 'depth' or 'shape' of object array. It has to 'guess', and in some cases guesses wrong.
Question: When I click on the buttons to change the graph they become invisible, but are still usable. How do you keep them always displayed on the window? I know it's plt.clf() that clears the content, but how to place buttons that will always stay under the graph? import matplotlib.pyplot as plt from matplotlib.widgets import Button def plot1(event): plt.clf() plt.scatter(liste_x,liste_graphe_1,s=5,c="red") plt.draw() def plot2(event): plt.clf() plt.scatter(liste_x,liste_graphe_2,s=5,c="green") plt.draw() liste_x=[1,2,3] liste_graphe_1=[1,2,3] liste_graphe_2=[3,2,1] fig,ax=plt.subplots() plt.scatter(liste_x,liste_graphe_1,s=5,c="red") axBtn1=plt.axes([0.0,0.0,0.2,0.05]) Btn1=Button(axBtn1,label="graphe_1",color="grey",hovercolor="yellow") Btn1.on_clicked(plot1) axBtn2=plt.axes([0.2,0.0,0.2,0.05]) Btn2=Button(axBtn2,label="graphe_2",color="grey",hovercolor="yellow") Btn2.on_clicked(plot2) plt.show()  Buttons are placed, they change the graph as expected. I tried to take off plt.clf(), but the graphs are not replaced without this function.

Answer: You have to use ax. instead of plt.. First you have to use it to create plot ax.scatter(...) Next you have to use it to remove plot ax.clear() # PEP8: use space after `,` around `=`, etc. import matplotlib.pyplot as plt from matplotlib.widgets import Button def plot1(event): ax.clear() ax.scatter(liste_x, liste_graphe_1, s=5, c="red") plt.draw() def plot2(event): ax.clear() ax.scatter(liste_x, liste_graphe_2, s=5, c="green") plt.draw() liste_x = [1, 2, 3] liste_graphe_1 = [1, 2, 3] liste_graphe_2 = [3, 2, 1] fig, ax = plt.subplots() ax.scatter(liste_x, liste_graphe_1, s=5, c="red") ax_btn1 = plt.axes([0.0, 0.0, 0.2, 0.05]) # PEP8: `lower_case_names` for variables btn1 = Button(ax_btn1, label="graphe_1", color="grey", hovercolor="yellow") btn1.on_clicked(plot1) ax_btn2 = plt.axes([0.2, 0.0, 0.2, 0.05]) btn2 = Button(ax_btn2, label="graphe_2", color="grey", hovercolor="yellow") btn2.on_clicked(plot2) plt.show()  PEP 8 -- Style Guide for Python Code
Question: I'm looking for suggestions on how I could rewrite this so that the code isn't repeating. It's suppose to separate the float(digits) from the string(alphabetical characters) within a dictionary's value, and subtract or add a user's numerical input to the float. Afterwards, it rejoins the digits and the letters, converts them back into a string and stores the value. An example of what it should do is take "19.0: chicken" from the dictionary, split it, add 3 from the user's input, and return "22.0 chicken" to the dictionary. if modify_options == "1": quantity_value = float(re.findall(r"[0-9]+(?:\.[0-9]*)?", inventory_list[key]) [0]) + quantity quantity_measurement = ''.join(re.findall(r'(?i)[A-Z]', str(inventory_list[key]))) inventory_list[key] = str(quantity_value) + str(quantity_measurement) return inventory_list elif modify_options == "2": quantity_value = float(re.findall(r"[0-9]+(?:\.[0-9]*)?", inventory_list[key])[0]) - quantity quantity_measurement = ''.join(re.findall(r'(?i)[A-Z]', str(inventory_list[key]))) inventory_list[key] = str(quantity_value) + str(quantity_measurement) return inventory_list 

Answer: The best practice for this would be wrapping up the code within a function. Infact whenever you find some part of code is repeating multiple times, you are supposed to put it into a function. Here's how to do it: def update_inventory(key, option, quantity): quantity_value = float(re.findall(r"[0-9]+(?:\.[0-9]*)?", inventory_list[key]) [0]) quantity_measurement = ''.join(re.findall(r'(?i)[A-Z]', str(inventory_list[key]))) if option=="1": quantity_value += quantity elif option=="2": quantity_value -= quantity inventory_list[key] = str(quantity_value) + str(quantity_measurement) return inventory_list #now you can update the inventory_list by calling the function update_inventory(2, "1", 9) update_inventory(1, "2", 4)
Question: I have a dataframe like this lstvals = [30.81,27.16,82.15,31.00,9.13,11.77,25.58,7.57,7.98,7.98] lstdates = ['2021-01-01', '2021-01-05', '2021-01-09', '2021-01-13', '2021-01-17', '2021-01-21', '2021-01-25', '2021-01-29', '2021-02-02', '2021-02-06'] data = { "Dates": lstdates, "Market Value": lstvals } df = pd.DataFrame(data) df.set_index('Dates', inplace = True) df  I want to forecast the values which are out of this sample, for example, from '2021-02-10' to '2022-04-23' (in my dataset, I have data from '2021-01-01' to '2023-11-09', and want to forecast for next year, from '2024-01-01' to '2023-11-09) https://www.statsmodels.org/devel/examples/notebooks/generated/statespace_forecasting.html I have defined and fitted my model as follows, which predicts the test data:  train = df['Market Value'].iloc[:1187] test = df['Market Value'].iloc[-200:] ... ARMAmodel = SARIMAX(y, order = (2,1,2)) ARMAResults = ARMAmodel.fit() ... y_pred = ARMAResults.get_forecast(len(test.index)) y_pred_df = y_pred.conf_int(alpha = 0.05) y_pred_df["Predictions"] = ARMAResults.predict(start = y_pred_df.index[0], end = y_pred_df.index[-1]) y_pred_df.index = test.index y_pred_out = y_pred_df["Predictions"] ... plt.plot(train, color = "black") plt.plot(test, color = "red") plt.ylabel('Market Value ($M)') plt.xlabel('Date') plt.xticks(rotation=45) plt.title("Train/Test/Prediction for Market Data") plt.plot(y_pred_out, color='green', label = 'Predictions') plt.legend() plt.show()   How can I make predictions for future dates? I have just tried to input future dates with the forecast method, and apparently, it is not working for me ARMAResults.forecast(start = '2024-01-01', end = '2024-11-09') TypeError: statsmodels.tsa.statespace.mlemodel.MLEResults.predict() got multiple values for keyword argument 'start'  https://www.statsmodels.org/devel/examples/notebooks/generated/statespace_forecasting.html

Answer: Issues:  Specify the frequency for the dates index: df = pd.DataFrame(data) df['Dates'] = pd.to_datetime(df['Dates']) df.set_index('Dates', inplace=True) df = df.asfreq('4D')   forecast is strictly for out-of-sample forecasts, and has no start or end parameters. (Note that its steps parameter can be passed a string or datetime type.) Either use predict or get_prediction, which support both in-sample and out-of-sample results. ARMAResults.predict(start='2024-01-01', end='2024-11-09')  or # mean ARMAResults.get_prediction(start='2024-01-01', end='2024-11-09').predicted_mean # mean, standard error, prediction interval ARMAResults.get_prediction(start='2024-01-01', end='2024-11-09').summary_frame()
Question: I am using the Python numexpr module to evaluate user inputs (numeric or formula). Numbers can be complex, and this works as long as I'm avoiding scientific notation: >>> import numexpr as ne >>> ne.evaluate("1000000000000j") array(0.+1.e+12j) >>> ne.evaluate("0.+1.e+12j") ValueError: Expression 0.+1.e+12j has forbidden control characters.  I tried to evaluate complex numbers in scientific notation with some variations and was expecting numexpr to process these numbers. However, I always ended up with the ValueError described above. Is this a bug or am I doing something wrong?

Answer: The errors you see with numexpr when trying to evaluate the expression "0.+1.e+12j" are due to the fact that numexpr parses complex numbers differently than standard Python. It does not accept "0.+1.e+12j" as valid because it prefers expressions where operations between numerical and logical units are explicitly defined. However, if you reformify the expression to "1.e+12 * 1j", numexpr handles it correctly because this option explicitly separates the scalar and imaginary units by the multiplication operation, which numexpr can be handled effectively: >>> ne.evaluate("1.e+12 * 1j") array(0.+1.e+12j)
Question: Assuming the following pandas data frame:  data lookup_val 0 [1.3, 4.5, 6.4, 7.3, 8.9, 10.3] 5 1 [2.5, 4.7, 6.4, 6.6, 8.5, 9.3, 17.4] 3 2 [3.3, 4.2, 5.1, 7.8, 9.2, 11.5] 6  I need to look up the value within the list of each 'data' column at the position of the value in the 'lookup_val' column. Expected output would be a new column like this:  data lookup_val output 0 [1.3, 4.5, 6.4, 7.3, 8.9, 10.3] 5 8.9 1 [2.5, 4.7, 6.4, 6.6, 8.5, 9.3, 17.4] 3 6.4 2 [3.3, 4.2, 5.1, 7.8, 9.2, 11.5] 6 11.5  What is the most efficient way to do so, assuming the data frame has millions of rows like this with each list having a different length, but no longer than 50 values? Iterating over the data frame or using apply with a simple indexing takes literally hours and a more performant structure is needed. Code to generate the above sample: import pandas as pd df = pd.DataFrame( [ {'data': [1.3, 4.5, 6.4, 7.3, 8.9, 10.3], 'lookup_val': 5}, {'data': [2.5, 4.7, 6.4, 6.6, 8.5, 9.3, 17.4], 'lookup_val': 3}, {'data': [3.3, 4.2, 5.1, 7.8, 9.2, 11.5], 'lookup_val': 6}, ] ) 

Answer: There is no efficient vectorial way to perform this. You need to loop. The easiest is most likely to use a list comprehension and zip: df['output'] = [l[i-1] for l, i in zip(df['data'], df['lookup_val'])]  Since your lookup values use a one-based indexing, you must subtract 1. Output:  data lookup_val output 0 [1.3, 4.5, 6.4, 7.3, 8.9, 10.3] 5 8.9 1 [2.5, 4.7, 6.4, 6.6, 8.5, 9.3, 17.4] 3 6.4 2 [3.3, 4.2, 5.1, 7.8, 9.2, 11.5] 6 11.5  If there is a chance that lookup values are incorrect, you should add a manual check. For example checking for the upper bound: df['output'] = [l[i-1] if i<= len(l) else None for l, i in zip(df['data'], df['lookup_val'])]  Or with a custom function: def get_val(lst, i): try: return lst[i-1] except IndexError: return None df['output'] = [get_val(*x) for x in zip(df['data'], df['lookup_val'])]  Output:  data lookup_val output 0 [1.3, 4.5, 6.4, 7.3, 8.9, 10.3] 5 8.9 1 [2.5, 4.7, 6.4, 6.6, 8.5, 9.3, 17.4] 3 6.4 2 [3.3, 4.2, 5.1, 7.8, 9.2, 11.5] 7 NaN  timings Using up to 1M rows with a random number of items in each list between 1 and 50:  Code to set up the random example: def init(N): ns = np.random.randint(1, 51, size=N) df = pd.DataFrame({'data': [list(range(n)) for n in ns]}) df['lookup_val'] = np.random.randint(0, ns, size=N) return df  I also tested the implication of incorrect values for the custom function. For this I generated a second lookup column with 90% chance that the value in out of bounds. This seems to have a limited impact on efficiency (the other two approaches still use a valid lookup value in this graph):