Repository: vaexio/vaex

Issue:
[BUG-REPORT] Joining using virtual columns throws an exception
Joining using virtual columns throws an exception. For example:

```python
import vaex

df1 = vaex.from_arrays(numbers1=['t_h_r_e_e', 't_w_o', 'o_n_e'])
df2 = vaex.from_arrays(numbers2=['one', 'one', 'four'])

print('df1')
print(df1)

def transform(text):
    return '_'.join(char for char in text)

df2['numbers2x'] = df2['numbers2'].apply(transform, multiprocessing=False)

print('df2')
print(df2)

joined = df1.join(df2, left_on='numbers1', right_on='numbers2x', how='inner', allow_duplication=True)

print('joined')
print(joined)
```

Produces this output:

```
ERROR:MainThread:vaex:error evaluating: numbers2x at rows 0-2
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/vaex/scopes.py", line 102, in evaluate
    result = self[expression]
  File "/usr/local/lib/python3.9/site-packages/vaex/scopes.py", line 159, in __getitem__
    raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: "Unknown variables or column: 'lambda_function(numbers2)'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 1968, in data_type
    data = self.evaluate(expression, 0, 1, filtered=False, array_type=array_type, parallel=False)
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 2642, in evaluate
    return self._evaluate_implementation(expression, i1=i1, i2=i2, out=out, selection=selection, filtered=filtered, array_type=array_type, parallel=parallel, chunk_size=chunk_size)
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 5676, in _evaluate_implementation
    value = scope.evaluate(expression)
  File "/usr/local/lib/python3.9/site-packages/vaex/scopes.py", line 102, in evaluate
    result = self[expression]
  File "/usr/local/lib/python3.9/site-packages/vaex/scopes.py", line 152, in __getitem__
    values = self.evaluate(expression)
  File "/usr/local/lib/python3.9/site-packages/vaex/scopes.py", line 108, in evaluate
    result = eval(expression, expression_namespace, self)
  File "<string>", line 1, in <module>
NameError: name 'lambda_function' is not defined
```

This is repeated a few times. Then:

```
df1
  #  numbers1
  0  't_h_r_e_e'
  1  't_w_o'
  2  'o_n_e'
df2
  #  numbers2    numbers2x
  0  'one'       o_n_e
  1  'one'       o_n_e
  2  'four'      f_o_u_r
joined
  #  numbers1    numbers2    numbers2x
  0  'o_n_e'     one         error
  1  'o_n_e'     one         error
```

Vaex version: `{'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0', 'vaex-hdf5': '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0', 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.11.1'}`

Installed with Pip
Python 3.9.1 on MacOS 10.15.7



Discussion:
Hi @maxharlow, even I encountered with this issue while trying to join on virtual column. I'm just checking whether you found a way around this or still stuck with the same error? 

If yes, please let me know how did you solve this.

Thanks and Regards,

Vijay
Thanks for the report @maxharlow 
This is fixed in #1298 and we hope to release this soon.
Has this been released? I am also have trouble joining on columns with after a string transformation, untransformed columns are joining without issues. Though my error shows up when I try to convert to pandas.
![image](https://user-images.githubusercontent.com/3199847/124205647-81a5ac00-da96-11eb-8f73-d86db0ff54eb.png)

It is released indeed, this is a different issue, do you want to open a new issue with a way to reproduce this error?
Hi @maartenbreddels I've found that although the issue mentioned in this thread has been fixed for inner joins, it still creates the same error when doing a left join. Below is the exact code used previously, but with how = 'left' in the join condition.

```
import vaex

df1 = vaex.from_arrays(numbers1=['t_h_r_e_e', 't_w_o', 'o_n_e'])
df2 = vaex.from_arrays(numbers2=['one', 'one', 'four'])

print('df1')
print(df1)

def transform(text):
    return '_'.join(char for char in text)

df2['numbers2x'] = df2['numbers2'].apply(transform, multiprocessing=False)

print('df2')
print(df2)

joined = df1.join(df2, left_on='numbers1', right_on='numbers2x', how='left',allow_duplication=True)

print('joined')
print(joined)
```
The resulting output is once again referencing the virtual column that cannot be found. 

```
ERROR:MainThread:vaex:error evaluating: numbers2x at rows 0-4
Traceback (most recent call last):
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/scopes.py", line 113, in evaluate
    result = self[expression]
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/scopes.py", line 198, in __getitem__
    raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: "Unknown variables or column: 'lambda_function(numbers2)'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/dataframe.py", line 2157, in data_type
    data = self.evaluate(expression, 0, 1, filtered=False, array_type=array_type, parallel=False)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/dataframe.py", line 2967, in evaluate
    return self._evaluate_implementation(expression, i1=i1, i2=i2, out=out, selection=selection, filtered=filtered, array_type=array_type, parallel=parallel, chunk_size=chunk_size, progress=progress)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/dataframe.py", line 6352, in _evaluate_implementation
    value = block_scope.evaluate(expression)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/scopes.py", line 113, in evaluate
    result = self[expression]
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/scopes.py", line 188, in __getitem__
    values = self.evaluate(expression)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/scopes.py", line 119, in evaluate
    result = eval(expression, expression_namespace, self)
  File "<string>", line 1, in <module>
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/arrow/numpy_dispatch.py", line 136, in wrapper
    result = f(*args, **kwargs)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/expression.py", line 1618, in __call__
    return vaex.multiprocessing.apply(self._apply, args, kwargs, self.multiprocessing)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/multiprocessing.py", line 32, in apply
    return f(*args, **kwargs)
  File "/home/ramp/base/lib/python3.8/site-packages/vaex/expression.py", line 1633, in _apply
    scalar_result = self.f(*[fix_type(k[i]) for k in args], **{key: value[i] for key, value in kwargs.items()})
  File "<ipython-input-431-0ea11e67461d>", line 10, in transform
    return '_'.join(char for char in text)
TypeError: 'NoneType' object is not iterable
```
and again repeats. 

Has there been any other reports of this happening/ could it be version numbers creating this issue?

Current versions: 
```vaex.__version__
{'vaex': '4.7.0',
 'vaex-core': '4.7.0.post1',
 'vaex-viz': '0.5.1',
 'vaex-hdf5': '0.11.1',
 'vaex-server': '0.8.0',
 'vaex-astro': '0.9.0',
 'vaex-jupyter': '0.7.0',
 'vaex-ml': '0.16.0'}
```

Hey @HvdHorst 

Thanks for bringing this up! I was able to reproduce it and track it to the usage of apply. 
I will create a new issue so we can track it better. 
I don't think this is fixed -- I still see this issue for inner joins when both columns are virtual columns. To adapt my previous example:

```python
import vaex

def transform(text):
    return '_'.join(char for char in text)

df1 = vaex.from_arrays(numbers1=['three', 'two', 'one'])
df2 = vaex.from_arrays(numbers2=['one', 'one', 'four'])

df1['numbers1x'] = df1['numbers1'].apply(transform, multiprocessing=False)

print('df1')
print(df1)

df2['numbers2x'] = df2['numbers2'].apply(transform, multiprocessing=False)

print('df2')
print(df2)

joined = df1.join(df2, left_on='numbers1', right_on='numbers2x', how='inner', allow_duplication=True)

print('joined')
print(joined)
```

Produces a similar error:

```
Traceback (most recent call last):
  File "/Users/maxharlow/Desktop/example.py", line 19, in <module>
    joined = df1.join(df2, left_on='numbers1', right_on='numbers2x', how='inner', allow_duplication=True)
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 6573, in join
    return vaex.join.join(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/vaex/join.py", line 244, in join
    right.rename(name, new_name)
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 3714, in rename
    self._rename(name, new_name, rename_meta_data=True)
  File "/usr/local/lib/python3.9/site-packages/vaex/dataframe.py", line 3742, in _rename
    self.column_names[self.column_names.index(old)] = new
ValueError: 'lambda_function' is not in list
```

I'm using Python 3.9.10 on MacOS 12.2.1. Vaex version: `{'vaex': '4.8.0', 'vaex-core': '4.8.0', 'vaex-viz': '0.5.1', 'vaex-hdf5': '0.12.0', 'vaex-server': '0.8.1', 'vaex-astro': '0.9.0', 'vaex-jupyter': '0.7.0', 'vaex-ml': '0.17.0'}` Installed via Pip.

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -3753,9 +3753,13 @@ def rename(self, name, new_name, unique=False):
 
     def _rename(self, old, new, rename_meta_data=False):
         is_variable = False
+        is_function = False
         if old in self.variables:
             self.variables[new] = self.variables.pop(old)
             is_variable = True
+        if old in self.functions:
+            self.functions[new] = self.functions.pop(old)
+            is_function = True
         elif old in self.virtual_columns:
             # renaming a column should not change the internal order, otherwise virtual
             # columns do not resolve (it will reference an unknown column)
@@ -3773,7 +3777,7 @@ def _rename(self, old, new, rename_meta_data=False):
                     del d[old]
         for key, value in self.selection_histories.items():
             self.selection_histories[key] = list([k if k is None else k._rename(self, old, new) for k in value])
-        if not is_variable:
+        if not (is_variable or is_function):
             if new not in self.virtual_columns:
                 self._renamed_columns.append((old, new))
             self.column_names[self.column_names.index(old)] = new
diff --git a/packages/vaex-core/vaex/expresso.py b/packages/vaex-core/vaex/expresso.py
--- a/packages/vaex-core/vaex/expresso.py
+++ b/packages/vaex-core/vaex/expresso.py
@@ -516,6 +516,7 @@ def __init__(self):
 
     def visit_Call(self, node):
         # we skip visiting node.id
+        self.visit(node.func)
         node.args = [self.visit(k) for k in node.args]
         if hasattr(node, 'keywords'):
             node.keywords = [self.visit(k) for k in node.keywords]


Test patch:
diff --git a/tests/join_test.py b/tests/join_test.py
--- a/tests/join_test.py
+++ b/tests/join_test.py
@@ -372,3 +372,15 @@ def test_join_datetime():
                                       np.datetime64('2009-10-12T11:00:00'),
                                       None,
                                       np.datetime64('2009-12-12T03:00:00')]
+
+def test_apply_function_name_collision():
+    def transform(x):
+        return x + 1 if x is not None else 99
+
+    df1 = vaex.from_arrays(x=[1, 2])
+    df2 = vaex.from_arrays(y=[1, 3])
+
+    df1["x2"] = df1["x"].apply(transform, multiprocessing=False)
+    df2["y2"] = df2["y"].apply(transform, multiprocessing=False)
+    joined = df1.join(df2, left_on="x2", right_on="y2", allow_duplication=True)
+    assert joined.y2.tolist() == [2, 99]
diff --git a/tests/rename_test.py b/tests/rename_test.py
--- a/tests/rename_test.py
+++ b/tests/rename_test.py
@@ -99,3 +99,16 @@ def test_rename_expression():
     assert expr.tolist() == [1, 2]
     expr2 = expr._rename('1', '#')
     assert expr2.tolist() == [2, 3]
+
+
+def test_rename_function():
+    df = vaex.from_arrays(x=[1, 2], y=[2, 4])
+    def add(a, b):
+        return a + b
+    df.add_function('add', add)
+    assert 'add' in df.functions
+    df['z'] = df.func.add(df.x, df.y)
+    assert df.z.tolist() == [3, 6]
+    df.rename('add', 'add_test')
+    assert df.z.tolist() == [3, 6]
+    assert 'add' not in df.functions