How to use transformers for batch inference 
Author: wangdong1992
Pull request: no

I use transformers to train text classification models，for a single text, it can be inferred normally. The code is as follows

from transformers import BertTokenizer, TFAlbertForSequenceClassification
text = 'This is a sentence'
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
result = model(encoding)

When I predict more than one text at a time, an error will be reported. The code is as follows

texts = ['This is a sentence', 'This is another sentence']
encodings = []
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
for text in texts:
    encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
    encodings.append(encoding)
result = model(np.array(encodings))

The error information is as follows:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr ‘Tindices’ of string is not in the list of allowed values: int32, int64
; NodeDef: {{node ResourceGather}}; Op<name=ResourceGather; signature=resource:resource, indices:Tindices → output:dtype; attr=batch_dims:int,default=0; attr=validate_indices:bool,default=true; attr=dtype:type; attr=Tindices:type,allowed=[DT_INT32, DT_INT64]; is_stateful=true> [Op:ResourceGather]


Comments:
For encoder-only Transformer models (like ALBERT), refer to the docs (in this case of [`TFAlbertForSequenceClassification`](https://huggingface.co/docs/transformers/model_doc/albert#transformers.TFAlbertForSequenceClassification.call.example)):

```
from transformers import AlbertTokenizer, TFAlbertForSequenceClassification
import tensorflow as tf

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2')

texts = ['This is a sentence', 'This is another sentence']
inputs = tokenizer(texts, return_tensors="tf")

outputs = model(inputs)
logits = outputs.logits
```
You can just provide a list of strings to the tokenizer, and it will prepare them for the model.

For decoder-only Transformer models (like GPT-2, any GPT model basically), refer to [this answer](https://github.com/huggingface/transformers/issues/10704#issuecomment-798870853).

@NielsRogge Can we also write a `loop for Pytorch batch DataLoaders ` and do inferencing. As DataLoaders are very fast?

```
for batch in Batches:
    
     inp=tokenizer(batch, return_tensors="tf")
     model(inp)
```

> @NielsRogge Can we also write a `loop for Pytorch batch DataLoaders ` and do inferencing. As DataLoaders are very fast?
> 
> ```
> for batch in Batches:
>     
>      inp=tokenizer(batch, return_tensors="tf")
>      model(inp)
> ```

Hi @pratikchhapolika, I am interested to know is writing loop for pytorch batch Dataloaders doable? 

@Sun-SunQian you should probably ask this kind of question on the [forum](https://discuss.huggingface.co/)! Higher chances of getting an answer 😉 

Hi! 
Did you find a solution for this?  

Please see example here:
https://huggingface.co/tiiuae/falcon-40b/discussions/50