Support Tensorflow tensors as input in tokenizers - [Ongoing]
Author: MarkusSagen
Pull request: no

# 🚀 Feature request
Currently, huggingface tokenizers support as input either a list of strings or strings. It would be great if the tokenizers could support `Tensorflow tensor of strings` or `Keras.layers.Input` as input also.

## Motivation
When working with end-to-end TensorFlow model, and especially with TensorFlow Reusable SavedModels, I need to be able to bundle the tokenizer and model in one model/class. The data the model is passed is in a TensorFlow format similar to the following example https://www.tensorflow.org/text/tutorials/classify_text_with_bert#define_your_model and my models are defined in a similar format. My current problem is that I need to be able to both trace the model and pass a tensor as input.

## Your contribution
I would love to know if it is at all feasible to add support for tokenizers to take tf.tensor as input. If so, where those features should be added? I would gladly help to contribute to this feature in that case.   

Or if it is not feasible, if there could be possible to map existing huggingface tokenizers to TensorFlow-text tokenizers, would that work, and if so, would that be something that would be interesting to have in huggingface?

@Rocketknight1 


Comments:
Hi @MarkusSagen, this is an interesting issue! Our fast tokenizers are implemented in Rust, and tracing that will be impossible. However, there is usually a slower pure-Python version as well. It may be possible to trace these as a Tensorflow graph, but I've never done it myself. The tokenizers are designed for cross-framework compatibility, so I don't think we'd be able to support a special TF-compiled version of all of them because it's quite niche (even most TF users don't need it!)

In addition, many of our models have their own unique tokenizers and quirks, and I don't think there'll be a clean pure-TF solution that works for them all. Still, I believe it should be possible to do this for a specific model you're interested in, and in particular it should be possible for BERT and BERT-like models, because of the [tf.text.BertTokenizer](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer) class.

You could try extracting the vocabulary from one of our tokenizers, creating a `tf.text.BertTokenizer`, and then checking to see if you got the same output from it as you get from our tokenizers on some sample strings. If so, then you could create a model that uses the `BertTokenizer` as the first layer and then one of our models as the subsequent layer.

If you attempt it, feel free to ping me here with any issues you run into. We probably can't rewrite our tokenizers to support this out of the box, but if you get it working we think it'd make a great addition to our [tutorial notebooks](https://huggingface.co/docs/transformers/notebooks)!

Thank you for getting back so quickly on this! Good to know, I assumed that that might be the only way to get the tokenizer to except tensors as input. I will test it out and get back to you if there are any issues

Hi again @Rocketknight1! I started working on this issue today and seems very promising so far, at least for BERT tokenizers. I'll keep you updated on the progress and add a notebook when everything works well for the BERT tokenizer and have been tested
https://github.com/Hugging-Face-Supporter/TFTokenizers

The problem isn't so much tracing but that the input can't be a Tensorflow Tensor. Even though the tokenizers can output TF Tensors. I would be happy if I could map the tokenizer over a tf.data.Dataset before feeding it to a model. The tokenizer is deterministic at this point anyway. 

EDIT: Eh, HuggingFace Datasets actually reads parquet files. That's so much of an advantage that I'm not terribly concerned with it. It would be nice though since the batch padding would then happen in the model instead of in the conversion to TF Dataset.

hey hi 
any update on  this