Batched `map` not allowed to return 0 items
State: closed | Author: pcuenca (MEMBER)
Labels: name: bug

## Describe the bug
I'm trying to use `map` to filter a large dataset by selecting rows that match an expensive condition (files referenced by one of the columns need to exist in the filesystem, so we have to `stat` them). According to [the documentation](https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset), `a batch mapped function can take as input a batch of size N and return a batch of size M where M can be greater or less than N and can even be zero`.

However, when the returned batch has a size of zero (neither item in the batch fulfilled the condition), we get an `index out of bounds` error. I think that `arrow_writer.py` is [trying to infer the returned types using the first element returned](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L100), but no elements were returned in this case.

For this error to happen, I'm returning a dictionary that contains empty lists for the keys I want to keep, see below. If I return an empty dictionary instead (no keys), then a different error eventually occurs.

## Steps to reproduce the bug
```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    # The following line simulates no matches found, so we return an empty batch
    result = {'key': []}
    return result

filtered_dataset = dataset.map(
    select_rows,
    remove_columns = dataset.column_names,
    batched = True,
    num_proc = 1,
    desc = "Selecting rows with images that exist"
)
```

The code above immediately triggers the exception. If we use the following instead:

```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    result = {'key': []}   # or defaultdict or whatever
    
    # code to check for condition and append elements to result
    # some_items_found will be set to True if there were any matching elements in the batch
    
    return result if some_items_found else {}
```

Then it _seems_ to work, but it eventually fails with some sort of schema error. I believe it may happen when an empty batch is followed by a non-empty one, but haven't set up a test to verify it.

In my opinion, returning a dictionary with empty lists and valid column names should be accepted as a valid result with zero items.

## Expected results
The dataset would be filtered and only the matching fields would be returned.

## Actual results
An exception is encountered, as described. Using a workaround makes it fail further along the line.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-53-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Comments: Hi ! Thanks for reporting. Indeed it looks like type inference makes it fail. We should probably just ignore this step until a non-empty batch is passed.

Sounds good! Do you want me to propose a PR? I'm quite busy right now, but if it's not too urgent I could take a look next week.

Sure if you're interested feel free to open a PR :)

You can also ping me anytime if you have questions or if I can help !

Sorry to ping you, @lhoestq, did you have a chance to take a look at the proposed PR? Thank you!

Yes and it's all good, thank you :)

Feel free to close this issue if it's good for you

Everything's good, thanks!