Repository: vaexio/vaex

Issue:
Creating a filtered dataset generates error
This code from your official tutorial shows error in Colab:
```
df_negative = df[df.x < 0]
df_negative[['x', 'y', 'z', 'r']]
```
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> /usr/local/lib/python3.6/dist-packages/IPython/core/formatters.py in __call__(self, obj, include, exclude)
>     968                 d['include'] = include
>     969                 d['exclude'] = exclude
> --> 970                 return method(**d)
>     971             return None
>     972         else:
> 
> 6 frames
> /usr/local/lib/python3.6/dist-packages/vaex/dataframe.py in copy(self, column_names, virtual)
>    4591                     df._selection_masks[key] = vaex.superutils.Mask(df._length_original)
>    4592                 # and make sure the mask is consistent with the cache chunks
> -> 4593                 np.asarray(df._selection_masks[key])[:] = np.asarray(self._selection_masks[key])
>    4594         for key, value in self.selection_history_indices.items():
>    4595             if self.get_selection(key):
> 
> ValueError: could not broadcast input array from shape (330000) into shape (5)
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> <ipython-input-24-aeaf77ba01d1> in <module>()
>       1 df_negative = df[df.x < 0]
> ----> 2 df_negative[['x', 'y', 'z', 'r']]
> 
> 2 frames
> /usr/local/lib/python3.6/dist-packages/IPython/core/formatters.py in format(self, obj, include, exclude)
>     150             return {}, {}
>     151 
> --> 152         format_dict, md_dict = self.mimebundle_formatter(obj, include=include, exclude=exclude)
>     153 
>     154         if format_dict or md_dict:
> 
> TypeError: 'NoneType' object is not iterable


Discussion:
Hi @qo4on 

Thanks for trying out vaex. Unfortunately, I can not reproduce your issue.
Few things to try:
- make sure the `df = vaex.example()` downloaded correctly (it may take some time in colab)
- how do you installed vaex? `pip install vaex` usually works for me, but sometimes requires restart of the runtime, i do not understand why
- maybe you can share a bit more of the notebook code, so I can try to reproduce the issue?

This is how it looks like for me:
<img width="1279" alt="image" src="https://user-images.githubusercontent.com/18574951/64926047-32c34f80-d7f9-11e9-8249-edf1e5829bd6.png">


To verify that the selection worked, one can do:
<img width="607" alt="image" src="https://user-images.githubusercontent.com/18574951/64926070-630aee00-d7f9-11e9-9321-77ff069ea93b.png">

Thank you for your answer, JovanVeljanoski.
To reproduse this error please run this code with Python 3 GPU runtime in Colab:
https://drive.google.com/open?id=14v3mTDLq4I_8fXBuwXh9Fzj-3CcsmLw_

Also I had troubles with installing tensorflow 2.0 RC1 after importing vaex recently. But now it works.
Google drive shows 404 error for the link above. I don't know why. The same [file ](https://colab.research.google.com/drive/14v3mTDLq4I_8fXBuwXh9Fzj-3CcsmLw_)in Colab.
I found the issue. When you run
```
df_negative = df[df.x < 0]
df_negative[['x', 'y', 'z', 'r']]
```
it works. But if you run the same code after selection it doesn't:
```
df.select(df.x < 0)
df.evaluate(df.x, selection=True)
df_negative = df[df.x < 0]
df_negative[['x', 'y', 'z', 'r']]
```

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -3701,11 +3701,10 @@ def take(self, indices, filtered=True, dropfilter=True):
             # if the indices refer to the filtered rows, we can discard the
             # filter in the final dataframe
             df.set_selection(None, name=FILTER_SELECTION_NAME)
-        else:
-            # if we will not drop the filter, we will have to invalidate the cache
-            # since it refers to the previous dataframe rows
-            df._invalidate_selection_cache()
-            pass
+        # if we will not drop the filter, we will have to invalidate the cache
+        # since it refers to the previous dataframe rows
+        # TODO perf: we could instead of dropping the cache, take out the rows we need
+        df._invalidate_selection_cache()
         return df
 
     @docsubst
diff --git a/packages/vaex-viz/vaex/viz/mpl.py b/packages/vaex-viz/vaex/viz/mpl.py
--- a/packages/vaex-viz/vaex/viz/mpl.py
+++ b/packages/vaex-viz/vaex/viz/mpl.py
@@ -821,12 +821,12 @@ def label(index, label, expression):
                 ax.set_title(labelsz[j])
 
             max_labels = 10
-            xexpression = xexpressions[i]
+            xexpression = xexpressions[subplot_index]
             if self.iscategory(xexpression):
                 labels = self.category_labels(xexpression)
                 step = max(len(labels) // max_labels, 1)
                 pylab.xticks(np.arange(len(labels))[::step], labels[::step], size='small')
-            yexpression = yexpressions[i]
+            yexpression = yexpressions[subplot_index]
             if self.iscategory(yexpression):
                 labels = self.category_labels(yexpression)
                 step = max(len(labels) // max_labels, 1)


Test patch:
diff --git a/tests/selection_test.py b/tests/selection_test.py
new file mode 100644
--- /dev/null
+++ b/tests/selection_test.py
@@ -0,0 +1,20 @@
+import numpy as np
+
+import vaex
+
+
+def test_selection_and_filter():
+    x = np.arange(-10, 11, 1)
+    y = np.arange(21)
+    df = vaex.from_arrays(x=x, y=y)
+
+    df.select(df.x < 0)
+    selected_list = df.evaluate(df.x, selection=True).tolist()
+
+    df_filtered = df[df.x < 0]
+    filtered_list = df_filtered['x'].tolist()
+    assert filtered_list == selected_list
+
+    # make sure we can slice, and repr
+    df_sliced = df_filtered[:5]
+    repr(df_filtered)