[arrow_dataset.py] add new filter function
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

The `.map()` function is super useful, but can IMO a bit tedious when filtering certain examples.
I think, filtering out examples is also a very common operation people would like to perform on datasets.

This PR is a proposal to add a `.filter()` function in the same spirit than the `.map()` function.

Here is a sample code you can play around with:

```python
ds = nlp.load_dataset("squad", split="validation[:10%]")


def remove_under_idx_5(example, idx):
    return idx < 5


def only_keep_examples_with_is_in_context(example):
    return "is" in example["context"]


result_keep_only_first_5 = ds.filter(remove_under_idx_5, with_indices=True, load_from_cache_file=False)
result_keep_examples_with_is_in_context = ds.filter(only_keep_examples_with_is_in_context, load_from_cache_file=False)

print("Original number of examples: {}".format(len(ds)))
print("First five examples number of examples: {}".format(len(result_keep_only_first_5)))
print("Is in context examples number of examples: {}".format(len(result_keep_examples_with_is_in_context)))
```

Comments: I agree that a `.filter` method would be VERY useful and appreciated. I'm not a big fan of using `flatten_nested` as it completely breaks down the structure of the example and it may create bugs. Right now I think it may not work for nested structures. Maybe there's a simpler way that we've not figured out yet.

Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
```python
for i in range(num_examples):
    example = map_nested(lambda x: x[i], batch)
    # ... then test to keep it or not
```

> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
> 
> ```python
> for i in range(num_examples):
>     example = map_nested(lambda x: x[i], batch)
>     # ... then test to keep it or not
> ```

Awesome I'll check it out :-) 

> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
> 
> ```python
> for i in range(num_examples):
>     example = map_nested(lambda x: x[i], batch)
>     # ... then test to keep it or not
> ```

Awesome this function is definitely much nicer!

Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:
```python
one_example = {
    "title": "blabla",
    "paragraphs": [
        "p1", "p2", ...
    ]
}
```

We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch.

> Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:
> 
> ```python
> one_example = {
>     "title": "blabla",
>     "paragraphs": [
>         "p1", "p2", ...
>     ]
> }
> ```

They both work. I'm using it on trivia_qa which is pretty nested. If you use the option `dict_only=True` I think it's fine.

> We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch.

Why? 

Actually it's fine. I guess this is going to be yet another thing to be unit-tested just to make sure ^^

Yes, I will need to add tests and documentation! 
@thomwolf - would a function like this be ok? It abstracts `.map()` a bit which might be hard to understand. 

I tried on some datasets with nested structure and it works fine ! Great work :D 


Awesome :-), I will add documentation and some simple unittests

Ok merging!