Repository: vaexio/vaex

Issue:
[BUG-REPORT] ColumnIndexed Slicing Fails With Some Numpy Arrays
**Description**
I managed to trigger this as part of my application's logic. I tracked it down to an issue in ColumnIndexed's __getitem__ function that does not handle instances where the adjusted index negative index is larger than the size of the resulting filtered array.

This snippet from the `__getitem__` function shows where the data is filtered and indices are adjusted. I am going to submit a PR that fixes this issue.
```
    def __getitem__(self, slice):
        ...
        if isinstance(ar_unfiltered, Column):
            # TODO: this is a workaround, since we do not require yet
            # that Column classes know how to deal with indices, we get
            # the minimal slice, and get those (not the most efficient)
            if self.masked:
                unmasked_indices = indices[~mask]
                if len(unmasked_indices) > 0:
                    i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)
                else:
                    i1, i2 = 0, 0
            else:
                i1, i2 = np.min(indices), np.max(indices)
            ar_unfiltered = ar_unfiltered[i1:i2+1]  # HERE: the data is filtered to exclude masked values on the two ends of the array
            # Unrelated to the bug: is this if statement even necessary given that they both do the same thing...?
            if self.masked:  # HERE: the indices are adjusted to the new filtered; ex: [-1, 1, 2] becomes [-2, 0, 1]
                indices = indices - i1
            else:
                indices = indices - i1
        ...
```

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: I'm on master commit 45de210d54f153c1898797ef06078f9a9fcdd6ec
 {'vaex': '4.0.0a8',
 'vaex-core': '4.0.0a13',
 'vaex-viz': '0.5.0.dev1',
 'vaex-hdf5': '0.7.0a6',
 'vaex-server': '0.4.0a1',
 'vaex-astro': '0.8.0.dev1',
 'vaex-ui': '0.3.0',
 'vaex-jupyter': '0.6.0.dev1',
 'vaex-ml': '0.11.0a4',
 'vaex-graphql': '0.2.0.dev1'}
 - Vaex was installed via: pip / conda-forge / from source: source
 - OS: Windows 10

**Additional information**
Reproducible code:

```
import vaex
import numpy as np

# Generate the input
indices = np.array([-1, 1])
vals = np.arange(0, len(indices))
mask = indices == -1
ar = np.ma.array(vals, mask=mask)

# Get columns
col = vaex.column.ColumnMaskedNumpy(ar.data, ar.mask)
col_index = vaex.column.ColumnIndexed(col, indices, masked=True)

col_index[0:len(indices)]
```

Error traceback:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-12-58e5200721be> in <module>
      9 col_index = vaex.column.ColumnIndexed(col, indices, masked=True)
     10 
---> 11 col_index[0:len(indices)]

c:\users\imprint\documents\other\vaex\packages\vaex-core\vaex\column.py in __getitem__(self, slice)
    222             ar = ar_unfiltered.take(vaex.array_types.to_arrow(take_indices))
    223         else:
--> 224             ar = ar_unfiltered[indices]
    225         assert not np.ma.isMaskedArray(indices)
    226         if self.masked:

~\Miniconda3\envs\api\lib\site-packages\numpy\ma\core.py in __getitem__(self, indx)
   3223         # mask of being reshaped if it hasn't been set up properly yet
   3224         # So it's easier to stick to the current version
-> 3225         dout = self.data[indx]
   3226         _mask = self._mask
   3227 

IndexError: index -2 is out of bounds for axis 0 with size 1
```

[BUG-REPORT] ColumnIndexed Slicing Fails With Some Numpy Arrays
**Description**
I managed to trigger this as part of my application's logic. I tracked it down to an issue in ColumnIndexed's __getitem__ function that does not handle instances where the adjusted index negative index is larger than the size of the resulting filtered array.

This snippet from the `__getitem__` function shows where the data is filtered and indices are adjusted. I am going to submit a PR that fixes this issue.
```
    def __getitem__(self, slice):
        ...
        if isinstance(ar_unfiltered, Column):
            # TODO: this is a workaround, since we do not require yet
            # that Column classes know how to deal with indices, we get
            # the minimal slice, and get those (not the most efficient)
            if self.masked:
                unmasked_indices = indices[~mask]
                if len(unmasked_indices) > 0:
                    i1, i2 = np.min(unmasked_indices), np.max(unmasked_indices)
                else:
                    i1, i2 = 0, 0
            else:
                i1, i2 = np.min(indices), np.max(indices)
            ar_unfiltered = ar_unfiltered[i1:i2+1]  # HERE: the data is filtered to exclude masked values on the two ends of the array
            # Unrelated to the bug: is this if statement even necessary given that they both do the same thing...?
            if self.masked:  # HERE: the indices are adjusted to the new filtered; ex: [-1, 1, 2] becomes [-2, 0, 1]
                indices = indices - i1
            else:
                indices = indices - i1
        ...
```

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: I'm on master commit 45de210d54f153c1898797ef06078f9a9fcdd6ec
 {'vaex': '4.0.0a8',
 'vaex-core': '4.0.0a13',
 'vaex-viz': '0.5.0.dev1',
 'vaex-hdf5': '0.7.0a6',
 'vaex-server': '0.4.0a1',
 'vaex-astro': '0.8.0.dev1',
 'vaex-ui': '0.3.0',
 'vaex-jupyter': '0.6.0.dev1',
 'vaex-ml': '0.11.0a4',
 'vaex-graphql': '0.2.0.dev1'}
 - Vaex was installed via: pip / conda-forge / from source: source
 - OS: Windows 10

**Additional information**
Reproducible code:

```
import vaex
import numpy as np

# Generate the input
indices = np.array([-1, 1])
vals = np.arange(0, len(indices))
mask = indices == -1
ar = np.ma.array(vals, mask=mask)

# Get columns
col = vaex.column.ColumnMaskedNumpy(ar.data, ar.mask)
col_index = vaex.column.ColumnIndexed(col, indices, masked=True)

col_index[0:len(indices)]
```

Error traceback:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-12-58e5200721be> in <module>
      9 col_index = vaex.column.ColumnIndexed(col, indices, masked=True)
     10 
---> 11 col_index[0:len(indices)]

c:\users\imprint\documents\other\vaex\packages\vaex-core\vaex\column.py in __getitem__(self, slice)
    222             ar = ar_unfiltered.take(vaex.array_types.to_arrow(take_indices))
    223         else:
--> 224             ar = ar_unfiltered[indices]
    225         assert not np.ma.isMaskedArray(indices)
    226         if self.masked:

~\Miniconda3\envs\api\lib\site-packages\numpy\ma\core.py in __getitem__(self, indx)
   3223         # mask of being reshaped if it hasn't been set up properly yet
   3224         # So it's easier to stick to the current version
-> 3225         dout = self.data[indx]
   3226         _mask = self._mask
   3227 

IndexError: index -2 is out of bounds for axis 0 with size 1
```



Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/column.py b/packages/vaex-core/vaex/column.py
--- a/packages/vaex-core/vaex/column.py
+++ b/packages/vaex-core/vaex/column.py
@@ -213,15 +213,15 @@ def __getitem__(self, slice):
                 indices = indices - i1
             else:
                 indices = indices - i1
+        take_indices = indices
+        if self.masked:
+            # arrow and numpy do not like the negative indices, so we set them to 0
+            take_indices = indices.copy()
+            take_indices[mask] = 0
         if isinstance(ar_unfiltered, supported_arrow_array_types):
-            take_indices = indices
-            if self.masked:
-                # arrow does not like the -1 index, so we set them to 0
-                take_indices = indices.copy()
-                take_indices[mask] = 0
             ar = ar_unfiltered.take(vaex.array_types.to_arrow(take_indices))
         else:
-            ar = ar_unfiltered[indices]
+            ar = ar_unfiltered[take_indices]
         assert not np.ma.isMaskedArray(indices)
         if self.masked:
             # TODO: we probably want to keep this as arrow array if it originally was


Test patch:
diff --git a/tests/column_test.py b/tests/column_test.py
--- a/tests/column_test.py
+++ b/tests/column_test.py
@@ -119,6 +119,14 @@ def test_column_indexed_all_masked():
     assert column[0:2].tolist() == [None, None]
 
 
+def test_column_indexed_some_masked():
+    indices = np.array([-1, 1])
+    col = vaex.column.ColumnNumpyLike(np.arange(2))
+    column = vaex.column.ColumnIndexed(col, indices, masked=True)
+    assert column[0:1].tolist() == [None]
+    assert column[0:2].tolist() == [None, 1]
+
+
 @pytest.mark.skipif(pa.__version__.split(".")[0] == '1', reason="segfaults in arrow v1")
 @pytest.mark.parametrize("i1", list(range(0, 8)))
 @pytest.mark.parametrize("i2", list(range(0, 8)))