Repository: vaexio/vaex

Issue:
[BUG-REPORT] Multiple selection filters broken in latest version
**Description**
I think I've found an issue with the new internal representation of arrays as arrow arrays.  This is in 4.0.0.dev0, so you might have a plan to fix it by the next proper release, but just flagging it anyway as it seems relatively major.

With a simple test table `data`:

```
import pandas as pd
import numpy as np
import vaex

data = pd.DataFrame({
  "A":np.array([1,2,3,4]),
  "B":["A","B","C","D"],
  "C":[np.datetime64('2019-10-07 09:00:00'),np.datetime64('2019-10-07 10:00:00'),np.datetime64('2019-10-07 11:00:00'),np.datetime64('2019-10-07 12:00:00')]
  })

vaex.from_pandas(data).export_arrow('arrow_sample.arrow')
data = vaex.open('arrow_sample.arrow')
```

Normal single clause filters work as you'd expect:

```
>>> data[(data['B'] == 'D')]
  #    A  B    C
  0    4  'D'  Timestamp('2019-10-07 12:00:00')
>>> data[(data['A'] > 2)]
  #    A  B    C
  0    3  'C'  Timestamp('2019-10-07 11:00:00')
  1    4  'D'  Timestamp('2019-10-07 12:00:00')
```

But when you combine multiple clauses the answer is wrong (this should return a single row):

```
>>> data[(data['B'] == 'D') & (data['A'] > 2)]
  #    A  B    C
  0    1  'A'  Timestamp('2019-10-07 09:00:00')
  1    2  'B'  Timestamp('2019-10-07 10:00:00')
  2    3  'C'  Timestamp('2019-10-07 11:00:00')
  3    4  'D'  Timestamp('2019-10-07 12:00:00')
```

I first noticed this using a much larger table, and in that case it seemed to cause a memory leak (process memory continually rose after the query)  I haven't been able to reliably reproduce this case though, so I'm just reporting the simple case above.  It seems likely they're the same bug with two different symptoms.  With vaex==3.0.0 the query above returns a single row as you'd expect, so I suspect maybe [this change](https://github.com/vaexio/vaex/pull/984/files) has caused the issue, although I don't know enough to say how?

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: `{'vaex': '4.0.0.dev0', 'vaex-core': '4.0.0a5', 'vaex-viz': '0.5.0.dev0', 'vaex-hdf5': '0.7.0a2', 'vaex-server': '0.4.0.dev0', 'vaex-astro': '0.8.0.dev0', 'vaex-ui': '0.3.0', 'vaex-jupyter': '0.6.0.dev0', 'vaex-ml': '0.11.0a4', 'vaex-graphql': '0.2.0.dev0'}`
 - Vaex was installed from source
 - OS: Ubuntu 18.04.4 LTS



Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/arrow/numpy_dispatch.py b/packages/vaex-core/vaex/arrow/numpy_dispatch.py
--- a/packages/vaex-core/vaex/arrow/numpy_dispatch.py
+++ b/packages/vaex-core/vaex/arrow/numpy_dispatch.py
@@ -69,15 +69,6 @@ def arrow_array(self):
                 self._arrow_array = vaex.array_types.to_arrow(self._numpy_array)
         return self._arrow_array
 
-    def __eq__(self, rhs):
-        if vaex.array_types.is_string(self.arrow_array):
-            # this does not support scalar input
-            # return pc.equal(self.arrow_array, rhs)
-            return NumpyDispatch(pa.array(vaex.functions.str_equals(self.arrow_array, rhs)))
-        else:
-            if isinstance(rhs, NumpyDispatch):
-                rhs = rhs.numpy_array
-            return NumpyDispatch(pa.array(self.numpy_array == rhs))
 
 for op in _binary_ops:
     def closure(op=op):
@@ -88,7 +79,10 @@ def operator(a, b):
                 a_data = a.numpy_array
             if isinstance(b, NumpyDispatch):
                 b_data = b.numpy_array
-            result_data = op['op'](a_data, b_data)
+            if op['name'] == 'eq' and (vaex.array_types.is_string(a_data) or vaex.array_types.is_string(b_data)):
+                result_data = vaex.functions.str_equals(a_data, b_data)
+            else:
+                result_data = op['op'](a_data, b_data)
             if isinstance(a, NumpyDispatch):
                 result_data = a.add_missing(result_data)
             if isinstance(b, NumpyDispatch):
@@ -96,8 +90,7 @@ def operator(a, b):
             return NumpyDispatch(result_data)
         return operator
     method_name = '__%s__' % op['name']
-    if op['name'] != "eq":
-        setattr(NumpyDispatch, method_name, closure())
+    setattr(NumpyDispatch, method_name, closure())
      # to support e.g. (1 + ...)    # to support e.g. (1 + ...)
     if op['name'] in reversable:
         def closure(op=op):
@@ -154,7 +147,5 @@ def wrapper(*args, **kwargs):
         args = list(map(unwrap, args))
         kwargs = {k: unwrap(v) for k, v, in kwargs.items()}
         result = f(*args, **kwargs)
-        if isinstance(result, vaex.array_types.supported_arrow_array_types):
-            result = NumpyDispatch(result)
-        return result
+        return wrap(result)
     return wrapper


Test patch:
diff --git a/tests/compute_test.py b/tests/compute_test.py
--- a/tests/compute_test.py
+++ b/tests/compute_test.py
@@ -17,6 +17,11 @@ def y(array_factory2):
     return array_factory2([1, 2, None, None])
 
 
+@pytest.fixture(scope='session')
+def s():
+    return pa.array(['a', 'b', None, 'd'])
+
+
 def test_add(x, y):
     df = vaex.from_arrays(x=x, y=y)
     df['z'] = df.x + df.y
@@ -47,7 +52,13 @@ def test_stay_same_type(x):
     assert (df.x.sin()).tolist()[-1] == None
     assert isinstance(df.x.sin().values, type(x))
 
-# def test_values2():
-#     df = vaex.from_arrays(x=, y=y)
-#     df = vaex.from_scalars(x=0, y=1)
-#     assert isinstance(df.evaluate('cos(x)+y'), type(x))
\ No newline at end of file
+
+def test_mix_string_and_numeric(x, s):
+    df = vaex.from_arrays(x=x, s=s)
+    # TODO: Note that this is a seperate bug, it ignored the missing value
+    assert (df.s == 'a').tolist() == [True, False, False, False]
+    assert (df.x == 1).tolist() == [False, True, False, None]
+    assert ((df.s == 'a') | (df.x == 1)).tolist()[0] is True
+    assert ((df.s == 'a') | (df.x == 1)).tolist() == [True, True, False, None]
+    assert (('a' == df.s) | (df.x == 1)).tolist() == [True, True, False, None]
+    assert ((df.x == 1) | (df.s == 'a')).tolist() == [True, True, False, None]