[Feature request] Pass any Iterable to tokenizer.__call__()
Author: jarednielsen
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Currently `tokenizer.__call__` accepts `List[List[str]]`, for pre-tokenized inputs. It should also accept `List[np.ndarray[str]]`.

## Motivation

The `nlp` library stores a batch of pre-tokenized strings as `List[np.ndarray[str]]` after creating them with `dset.map(batched=True)`. Currently it requires a memory-intensive `tokenizer([list(ex) for ex in batch])` to pass these pre-tokenized strings into `batch_encode_plus`.

## Your contribution

I could contribute this.


Comments:
This should be handle in `nlp` in my opinion.

It's related to https://github.com/huggingface/nlp/issues/387

cc @lhoestq 

Nice find on the `nlp` side! There are also other use cases when users might want to pass in a NumPy array, or other type of Iterable. Any reason we shouldn't extend to all Iterables like https://github.com/huggingface/nlp/pull/370?

The fast tokenizers only support python inputs at the moment. We could change it to allow numpy arrays as well but I would expect this to be potentially a significant work to update.

Ah yes, Rust's static typing makes this more complex. And my own benchmarks show significant memory usage when casting from a numpy array to python list. Fixing the `nlp` output format should work well then.

I just did the change. Let me know if it's good for you :)

Thanks for the fix @lhoestq! That does indeed resolve the need for a list comprehension, but I'm still hitting the same speed bottleneck. It seems that the list comprehension is just moved inside of the `to_pylist()` function. Here's a reproducible benchmark:

```python
import nlp
from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")

dset_size = 10000
max_seq_length = 512
dset = nlp.Dataset.from_dict(
    {"examples": [[str(i) for i in range(max_seq_length)] for _ in range(dset_size)]}
)

dset = dset.map(
    lambda batch: tokenizer(
        batch["examples"], is_pretokenized=True,  # rather than [ex for ex in batch["examples"]]
    ),
    batched=True,
    remove_columns=["examples"],
)
```

This takes 37 seconds to run, processing around 270 examples/second, or 3.7 seconds/batched iteration. At this pace it takes around 1 hour to encode 10GB of text, such as Wikipedia, and even longer for a larger dataset like C4.

I would love to take full use of the tokenizers functionality of "they can encode 1GB of text in ~20sec on a standard server's CPU". That would allow encoding Wikipedia in 2 minutes rather than an hour. Are there any further improvements that would un-bottleneck the batched map function?

The bottleneck is indeed the conversion of arrow types to python lists.
I've been looking for a faster way to do it but I couldn't find a satisfactory solution.

If we manage to be full rust/c++ on this we could achieve full speed:
- We could add support for numpy arrays as input for tokenizers (arrow to numpy is very fast)
- Or we could leverage arrow's rust API to do the processing in rust
Both options are not trivial though.

If we do so, we can expect to have the tokenization as the bottleneck, instead of the conversion from arrow to python lists.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.