Question: I have a number of dataframes from which I have to take a sample. The samples taken from that dataframe, have to be excluded from the next dataframe in order to not have any 'double' samples as there is some overlap. my code is as follows df_list = [df1, df2, df3, df4, df5] samplesizes = [8, 2, 4, 4, 2] sample = [] for df, samplesize in zip(df_list, samplesizes): if sample: #can't drop in the first loop df = df.drop(sample) #I want to drop the taken samples from the current df if max_pop_size < len(df): samplesize = max_pop_size #can't take a sample larger than population sample.append(df.sample(samplesize, random_state=1000))  I get stuck on the dropping after the first loop. I've tried several things and none seem to work. EDIT: the df's are a subset of 1 big population, so the columns are identical. the subsets (in df_list) are sliced based on earlier criteria which may contain duplicates. If a sample is taken, we don't want that specific row to be in the population of the next sample. To make things easier, they all have the same index! Any help would be much appreciated!

Answer: This is an interesting and not straightforward problem. I think you cannot/shouldn't:  pre-filter the duplicates, since it might remove too many rows (it might remove a row in group 2 that was in group 1 even if it was not sampled from group 1) sample then post-filter, since removing rows after sampling would decrease the sample size in an unpredictable way concatenate the sampled DataFrames in the loop(*), this has quadratic complexity and would make the code slow for a large number of inputs  I think a reasonable approach could be to first compute an ID to identify duplicated rows, then to loop over the groups and keep track of the already sampled rows: df1 = pd.DataFrame({'A': [1,2,3], 'B': [10,20,30]}) df2 = pd.DataFrame({'A': [2,3,4], 'B': [20,30,40]}) df3 = pd.DataFrame({'A': [1,4,5], 'B': [10,40,50]}) np.random.seed(1) df_list = [df1, df2, df3] samplesizes = [2, 1, 1] # combine all datasets to be able to identify the duplicates tmp = (pd.concat(df_list, keys=range(len(df_list))) .assign(ID=lambda d: d.groupby(list(tmp)).ngroup()) .set_index('ID', append=True) ) # set to keep track of already sampled unique IDs seen = set() # for each input (group) # first remove the already sampled rows # then sample the expected size sample = [] for k, g in tmp.groupby(level=0, sort=False): selected = g.drop(index=seen, level='ID').sample(samplesizes[k]) seen.update(selected.index.get_level_values('ID')) sample.append(selected.droplevel([0, -1])) print(sample)  Of course, it could be that in some cases there are not enough rows left to sample from, but this is not really avoidable if you want to maintain an unbiased sampling. Example output: [ A B 0 1 10 2 3 30, A B 2 4 40, A B 2 5 50]  (*) concatenating in a loop If your have a small number of DataFrames and rows, concatenating in a loop the total sampled rows (concat+merge) might be acceptable. This gives a shorter code: df1 = pd.DataFrame({'A': [1,2,3], 'B': [10,20,30]}) df2 = pd.DataFrame({'A': [2,3,4], 'B': [20,30,40]}) df3 = pd.DataFrame({'A': [1,4,5], 'B': [10,40,50]}) np.random.seed(1) df_list = [df1, df2, df3] samplesizes = [2, 1, 1] sample = [] total = pd.DataFrame(columns=df_list[0].columns) for df, samplesize in zip(df_list, samplesizes): selected = (df.merge(total, how='left', indicator=True) .loc[lambda x: x.pop('_merge').eq('left_only')] .sample(samplesize) ) total = pd.concat([total, selected]) sample.append(selected) print(sample)  Example output: [ A B 0 1 10 2 3 30, A B 2 4 40, A B 2 5 50]