Repository: vaexio/vaex

Issue:
[BUG-REPORT] Empty Unmasked Indices in Masked Columns Throw Numpy Error
**Description**
In some rare edge cases, line 199 of `column.py` in vaex core throws a numpy error `ValueError: zero-size array to reduction operation minimum which has no identity`. This is because the `unmasked_indices` is an empty list.

I ran into this error after attempting a join where there were no overlapping indices.

It's possible that this error is also present in other `column.py` functions, but I didn't check.

Subset of the `__getitem__` function (that contains the offending line):
```
def __getitem__(self, slice):
        start, stop, step = slice.start, slice.stop, slice.step
        start = start or 0
        stop = stop or len(self)
        assert step in [None, 1]
        indices = self.indices[start:stop]
        ar_unfiltered = self.column
        if self.masked:
            mask = indices == -1
        if isinstance(ar_unfiltered, Column):
            # TODO: this is a workaround, since we do not require yet
            # that Column classes know how to deal with indices, we get
            # the minimal slice, and get those (not the most efficient)
            if self.masked:
                unmasked_indices = indices[~mask]  # <-- can return an empty list if all indices are masked
                i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)  # <-- the offending line!
            else:
                i1, i2 = np.min(indices), np.max(indices)
            ar_unfiltered = ar_unfiltered[i1:i2+1]
            if self.masked:
                indices = indices - i1
            else:
                indices = indices - i1
            ...
```

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`:
```
{'vaex': '4.0.0.dev0',
 'vaex-core': '4.0.0a3',
 'vaex-viz': '0.5.0.dev0',
 'vaex-hdf5': '0.7.0a1',
 'vaex-server': '0.4.0.dev0',
 'vaex-astro': '0.8.0.dev0',
 'vaex-ui': '0.3.0',
 'vaex-jupyter': '0.6.0.dev0',
 'vaex-ml': '0.11.0a3',
 'vaex-arrow': '0.5.1',
 'vaex-graphql': '0.2.0.dev0'}
```
 - Vaex was installed via: pip
 - OS: Windows 10

**Additional information**
I've tried to create a reproducible example for over two hours now, and I still can't do it by starting with vaex.from_arrays(). However, I was able to reproduce the error by exporting my actual data to CSV, obfuscating it, and reading it into a vaex df. Please see the attached .zip with a jupyter notebook and data that illustrates the problem: [dt2.zip](https://github.com/vaexio/vaex/files/5330490/dt2.zip).

I'm starting to think that the issue may stem from some underlying dtype issue. Despite both `dt2` and `dt3` in my notebook having the same dtypes, their outputs look different. Note the apostrophes around the strings in `dt2`:
![image](https://user-images.githubusercontent.com/20652336/95139089-8a906980-0720-11eb-919b-0076dc3ddb98.png)


ZeroDivisionError on joining dataframes
**Description**
I get the **ZeroDivisionError** on doing a left join on the data frames. I got the same error when performing inner join too in the past. In the past I have observed the following when doing inner joins.
1. the error occurs when there is no match and the resulting data frame is empty- can be handled in a try catch
2. the error occurs when the size of left dataframe is smaller- can be handled by switching the df positions.

This is the left join code I am using :

`df1 = (df1.join(df2,on='id', lsuffix='_incl', rsuffix='_excl', allow_duplication=True))`

But as I am doing a left join this time I cannot switch the df positions. There are no virtual columns in both the dataframes

To give you some context regarding the use case, I want to do a set difference kind of operation on the dataframes. For example, if the 2 dataframes are df1 and df2- get all rows that are there in df1 but are not there in df2.  I was trying to do this using left join and isnan() filtering.

Since the left join operation is not woking I am trying the following to methods:
1. Use isin on the joining key ` df1[~df1.unq_member_id.isin(df2['id'])]` - this is giving error **len() of unsized object**
2. concat the 2 dataframes->group by with the joining key->filter out the one with count==1

Both my dataframes are quite big. Which approach do you think is best for this kind of problem?

**Software information**
 - Vaex version : 3
 - Vaex was installed via: pip 
 - OS: Ubuntu 18.04.3 LTS



Discussion:
One hacky workaround is to do ```dt2 = vx.from_arrow_table(dt2.to_arrow_table())``` prior to joining. I presume the dtypes under the hood are different?
I am getting the same error when doing left or right joins. But inner join works. The dtypes seem to match too.
Hi, 

Can you maybe try to isolate the issue, maybe send us an reproducible example? I understand the data is big, but issues like these are typically reproducible with small samples if one chooses the right stuff that cause the problem..

Things to try are including / excluding, nans or missing values, inspect the dtypes, along those lines..


I am not able to reproduce the error with dummy data. I am going to try to reproduce the error with some public datasets. Meanwhile, I ran the same code on a different machine  (below are the specs) on which I did not get the error with the same code and same data. But now for some of the cases, I get: 
 `ValueError: zero-size array to reduction operation minimum which has no identity`

```
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/vaex/execution.py", line 157, in execute_async
    cancel=lambda: self._cancel(run), unpack=True, run=run):
  File "/opt/conda/lib/python3.7/site-packages/vaex/multithreading.py", line 74, in map_async
    yield await value
  File "/opt/conda/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.7/site-packages/vaex/multithreading.py", line 64, in <lambda>
    iterator = [loop.run_in_executor(self, lambda value=value: wrapped(value)) for value in values]
  File "/opt/conda/lib/python3.7/site-packages/vaex/multithreading.py", line 54, in wrapped
    callable(self.local.index, *args, **kwargs, **kwargs_extra)
  File "/opt/conda/lib/python3.7/site-packages/vaex/execution.py", line 206, in process_part
    block_dict = {expression: block_scope.evaluate(expression) for expression in run.expressions}
  File "/opt/conda/lib/python3.7/site-packages/vaex/execution.py", line 206, in <dictcomp>
    block_dict = {expression: block_scope.evaluate(expression) for expression in run.expressions}
  File "/opt/conda/lib/python3.7/site-packages/vaex/scopes.py", line 97, in evaluate
    result = self[expression]
  File "/opt/conda/lib/python3.7/site-packages/vaex/scopes.py", line 128, in __getitem__
    self.values[variable] = self.df.columns[variable][offset+self.i1:offset+self.i2]
  File "/opt/conda/lib/python3.7/site-packages/vaex/column.py", line 149, in __getitem__
    i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)
  File "<__array_function__ internals>", line 6, in amin
  File "/home/s1967611/.local/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2831, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/home/s1967611/.local/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 87, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity
```

New Machine Information

 Debian GNU/Linux 9.13 (stretch) (GNU/Linux 4.9.0-13-amd64 x86_64\n)


This might be related to https://github.com/vaexio/vaex/issues/1001

Resolving patch:
diff --git a/packages/vaex-core/vaex/column.py b/packages/vaex-core/vaex/column.py
--- a/packages/vaex-core/vaex/column.py
+++ b/packages/vaex-core/vaex/column.py
@@ -88,6 +88,7 @@ class ColumnNumpyLike(Column):
     """Wraps a numpy like object (like hdf5 dataset) into a column vaex understands"""
     def __init__(self, ar):
         self.ar = ar  # this should behave like a numpy array
+        self.dtype = ar.dtype
 
     def __len__(self):
         return len(self.ar)
@@ -201,7 +202,10 @@ def __getitem__(self, slice):
             # the minimal slice, and get those (not the most efficient)
             if self.masked:
                 unmasked_indices = indices[~mask]
-                i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)
+                if len(unmasked_indices) > 0:
+                    i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)
+                else:
+                    i1, i2 = 0, 0
             else:
                 i1, i2 = np.min(indices), np.max(indices)
             ar_unfiltered = ar_unfiltered[i1:i2+1]


Test patch:
diff --git a/tests/column_test.py b/tests/column_test.py
--- a/tests/column_test.py
+++ b/tests/column_test.py
@@ -111,6 +111,14 @@ def test_column_indexed(df_local):
     assert column_masked.trim(0, 1).masked
 
 
+def test_column_indexed_all_masked():
+    indices = np.array([-1, -1])
+    col = vaex.column.ColumnNumpyLike(np.arange(2))
+    column = vaex.column.ColumnIndexed(col, indices, masked=True)
+    assert column[0:1].tolist() == [None]
+    assert column[0:2].tolist() == [None, None]
+
+
 @pytest.mark.skipif(pa.__version__.split(".")[0] == '1', reason="segfaults in arrow v1")
 @pytest.mark.parametrize("i1", list(range(0, 8)))
 @pytest.mark.parametrize("i2", list(range(0, 8)))