Multi-GPU inference with Tensorflow backend
Author: fingoldo
Pull request: no

Is this already supported maybe? I know that multi-GPU TRAINING is supported with TF* models pretty well. But not inference. What is the recommended way when one wants to do inference for a large batch of text (tens of millions rows)? Currently only one of GPUs gets loaded. Tensorflow have a [guide ](https://www.tensorflow.org/tutorials/distribute/save_and_load) on how to use model saved in the native tf format to do distributed inference under a scope:

```python
another_strategy = tf.distribute.MirroredStrategy()
with another_strategy.scope():
  loaded = tf.saved_model.load(saved_model_path)
  inference_func = loaded.signatures[DEFAULT_FUNCTION_KEY]

  dist_predict_dataset = another_strategy.experimental_distribute_dataset(
      predict_dataset)

  # Calling the function in a distributed manner
  for batch in dist_predict_dataset:
    another_strategy.run(inference_func,args=(batch,))
```
However, it seems that transformers do not support saving in this native format? At least TFDistilBertForSequenceClassification, when loaded back, has damaged input signatures (no attention_mask, wrong sequence length, fake None inputs) and can't process anything. And this very tracker is crowded with similar questions which are left unanswered. Can anyone shed some light on best approach to distributed inference please? Also adding a bullet on this to the documentation would be extremely helpful for many folks.

Comments:
Hello!!

Can you please share the code you are using as for me it works as expected:
```python
from transformers import TFBertModel, BertTokenizer
import tensorflow as tf

first_strategy = tf.distribute.MirroredStrategy()

with first_strategy.scope():
	model = TFBertModel.from_pretrained("bert-base-cased")
	model.save_pretrained("my_model")

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
another_strategy = tf.distribute.OneDeviceStrategy("/cpu:0")

with another_strategy.scope():
  restored_keras_model_ds = TFBertModel.from_pretrained("my_model")

  inputs = tokenizer("Hello world.", return_tensors="tf")
  predict_dataset = tf.data.Dataset.from_tensor_slices(inputs).batch(1)
  dist_predict_dataset = another_strategy.experimental_distribute_dataset(predict_dataset)
  for batch in dist_predict_dataset:
  	another_strategy.run(restored_keras_model_ds, args=(batch,))
```

Thank you guys so much for the response! It was not obvious to use save_pretrained under the scope. Your example runs successfully, however on a 8 GPUs machine I observe (with bigh enough input list, of course) a weird pattern when maximum 2 GPUs are busy, and the rest are simply stale. Then after some seconds new pair of GPUs become active and rest are [waiting.](https://pasteboard.co/JKmszWl.png) It happens no matter what strategy I try, MirroredStrategy or MultiWorkerMirroredStrategy. @jplu what strategy would you recommend to utilize all 8 GPUs?

This simply means that TF needs no more than 2 GPUs to run your inference.

But it's taking more than 40 seconds to run it. It definitely needs to utilize more ...

> 
> 2021-01-19 12:37:55,915 - INFO - <ipython-input-108-8a0316d72a4a> - root - infer_natively - line: 4 - Tokenizing dataset of length 100000...
> 2021-01-19 12:38:07,630 - INFO - <ipython-input-108-8a0316d72a4a> - root - infer_natively - line: 9 - Converting dataset to tf dataset using batch size 244...
> 2021-01-19 12:38:07,634 - INFO - <ipython-input-108-8a0316d72a4a> - root - infer_natively - line: 12 - Distributing tf dataset across replicas...
> 2021-01-19 12:38:07,714 - INFO - <ipython-input-108-8a0316d72a4a> - root - infer_natively - line: 16 - Inferencing using 8 GPUs
> 2021-01-19 12:39:38,318 - INFO - <ipython-input-108-8a0316d72a4a> - root - infer_natively - line: 36 - Done. nbatches processed: 26

Tensorflow doesn't take the time as reference but the size. If your data can fit on 2 GPUs then it uses only 2. I suggest you to read this to better understand how it works. https://www.tensorflow.org/guide/gpu

> as reference but the size. If your data can fit on 2 GPUs then it uses only 2. I suggest you to 

Following this link, I was not able to find any mentioning of when tf can select lower number of GPUs to run inference on, depending on data size. I tried with a million sentences and I'm still observing that pattern when only 2 GPUs are heavily loaded, and the rest has 0% utilization. and that pair of active GPUs changes randomly as the time goes. So something is definitely wrong with implementation. I was asking tf "please use all devices for this huge workload", and you are saying it just like "it can be done using 2 GPUs dude so i'm using 2, I don't care how long you gonna wait for the result" ? :-)

@jplu so if you know how to make it use all 8 GPUs in my particular case for 1 million of input sentences please advise, it would solve the issue completely.

Really sorry I don't know what to tell you more, if you have mostly TF related questions I suggest you to open an issue on the TF github repo.

Hi,

I'm having similar issues with inference when using multi-gpu, the `predict` function returns empty output despite being actually processing the input. 

```python
from transformers import BertTokenizerFast, TFBertForSequenceClassification
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()
#strategy = tf.distribute.OneDeviceStrategy("/gpu:0")
with strategy.scope():
    tf_model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')

    tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
    inputs = tokenizer('This is a test', 'Esto es una prueba',
                   return_tensors='tf', max_length=200,
                   padding='max_length', truncation=True,
                   return_attention_mask=True,
                   return_token_type_ids=False)
    print(tf_model.predict([inputs["input_ids"], inputs["attention_mask"]],
                           verbose=1))
    print(tf_model([inputs["input_ids"], inputs["attention_mask"]]))
```
```
All model checkpoint layers were used when initializing TFBertForSequenceClassification.

Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
WARNING:tensorflow:From /venv/lib/python3.7/site-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py:601: get_next_as_optional (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Iterator.get_next_as_optional()` instead.
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
1/1 [==============================] - 0s 241us/step
TFSequenceClassifierOutput(loss=None, logits=None, hidden_states=None, attentions=None)
TFSequenceClassifierOutput(loss=None, logits=<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[-0.47814545,  0.35146457]], dtype=float32)>, hidden_states=None, attentions=None)
```
Is this expected to happen? It would be great to be able to use predict function for performance reasons.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.