Added method to remove excess nesting in a DatasetDict
Opened by CakeCrusher (closed)

Added the ability for a DatasetDict to remove additional nested layers within its features to avoid conflicts when collating. It is meant to accompany [this PR](https://github.com/huggingface/transformers/pull/18119) to resolve the same issue [#15505](https://github.com/huggingface/transformers/issues/15505).

@stas00 @lhoestq 

Comments:
Hi ! I think the issue you linked is closed and suggests to use `remove_columns`.

Moreover if you end up with a dataset with an unnecessarily nested data, please modify your processing functions to not output nested data, or use `map(..., batched=True)` if you function take batches as input

Hi @lhoestq , you are right about the issues this pull has steered beyond that issue. I created this [colab notebook](https://colab.research.google.com/drive/16aLu6QrDSV_aUYRdpufl5E4iS08qkUGj?usp=sharing) to present the error. I tried using batch and that won't resolve it either. I'm looking into that error right now.

I think you just need to pass one example at a time to your tokenizer, this way you don't end up with nested data:
```python

def preprocessFunction(row):
  collatedContext = tokenizer.eos_token.join([row["context"+str(i+1)] for i in range(int(AMT_OF_CONTEXT))])
  response = row["response"]
  tokenizedContext = tokenizer(
    collatedContext, max_length=max_context_length, truncation=True  # don't pass as a list here
  )
  with tokenizer.as_target_tokenizer():
    tokenized_response = tokenizer(
      response, max_length=max_response_length, truncation=True  # don't pass a a list here
    )
  tokenizedContext["labels"] = tokenized_response["input_ids"]
  return tokenizedContext
```

Yes that is correct, the purpose of this pull is to advise of a more general solution like with `def remove_excess_nesting(self)` or maybe automate the solution (stas00 advised not to automate it as it could "not be backwards compatible").

I'm not sure I understand how having `remove_excess_nesting` would make more sense than just fixing the preprocessFunction to simply not return nested samples, can you elaborate ?

Figuring out the issue can be a bit difficult to figure out. Only until I added batch does it make a little more sense with the error

> sequence item 0: expected str instance, list found

but batch was never intended.

When you run the colab you will notice that only until collating do you learn there is this error. So i figured it would be better to address it during at the `DatasetDict` level.
I think it would be ideal if the user could be notified at the preprocess function.

I'm not arguing that `remove_excess_nesting` is the right solution but what I aim to address is dealing with unnecessary nesting as early as possible.

> When you run the colab you will notice that only until collating do you learn there is this error.

I think users can just check the `dataset.features` and they would notice that the data are nested
```python
{
   'input_ids': Sequence(Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None)
   ...
}
```

Sometime nested data are intentional, so you can't know in advance if it's a user's mistake or something planned.

Yes, I understand, it could be intentional and only the collator has problems with it. So, it is not worth handling it any differently in any other non-erroneous data. 

That being said do you think there is any use for the `remove_excess_nesting` method? Or maybe it should be applied in a different way? If not feel free to close this PR. 

I think users can write it and use `map` themselves if needed, it is pretty straightforward to implement.

I'm closing this PR if you don't mind, and thank you for the discussion :)

No problem @lhoestq , thanks for walking me through it.