Q:

Filtering a dataframe using Lambda in Python

I have two data frames in python: df and list:
data1 = [[0, ("a","b")], [1, ("d","e")], [2, ("a","e")],[3,("f", "g")],[4,("c","h")]]
df = pd. DataFrame(data1, columns = ['Row', 'Letters'])
data2 = [[0,"a"],[1,"b"],[2,"c"]]
list = pd. DataFrame(data2, columns = ['Row', 'Letters'])

I now want to filter down df for only rows such that any item in df['Letters'] is found in list['Letters']
The Any function works fine for individual rows:
any(item in df["Letters"][1] for item in list['Letters'])
any(item in df["Letters"][2] for item in list['Letters'])

correctly returns False and True, respectively. 
Now how do I filter down the entire dataframe? 
I tried the following code:
new_df = df[df.apply(lambda x : any(item in x["Letters"] for item in list), axis=1)]

which returns an empty dataframe when I want to return only rows 0, 2 and 4. 
Any help would be appreciated. 

A:

You can use a dataframe constructor with stack then compare using series.isin with any for level=0
df[pd. DataFrame(df['Letters'].tolist()).stack().isin(list_['Letters']).any(level=0)]

  Row Letters
0    0  (a, b)
2    2  (a, e)
4    4  (c, h)

Note: I have change the list variable where you save the list as list_ since you should not have a variable name same as a builtin function
benchmarking for a larger dataframe:
m = pd.concat([df]*10000,ignore_index=True)
%%timeit
m[pd. DataFrame(m['Letters'].tolist()).stack().isin(list_['Letters']).any(level=0)]
#25.3 ms ± 1.08 ms per loop (mean ± std.  dev.  of 7 runs, 10 loops each)

%%timeit
m.loc[~m['Letters'].apply(lambda x: set(x).isdisjoint(set(list_['Letters'])))]
#644 ms ± 8.38 ms per loop (mean ± std.  dev.  of 7 runs, 1 loop each)

%%timeit
m[m. Letters.apply(lambda x : any(item in list_.Letters.to_numpy().tolist() for item in x))]
#665 ms ± 13.2 ms per loop (mean ± std.  dev.  of 7 runs, 1 loop each)

%%timeit
m.loc[m['Letters'].apply(lambda x: len(set(x).intersection(set(list_['Letters']))) > 0)]
#707 ms ± 56.4 ms per loop (mean ± std.  dev.  of 7 runs, 1 loop each)

