[TFTrainer] Error "iterating over `tf.Tensor` is not allowed"
Author: EibrielInv
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2 (from pip)
- Platform: Linux-4.15.0-91-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.0 (True) (Same error on TF2.2 and TF2.1)
- Using GPU in script?: Yes - GeForce GTX 1080 Ti
- Using distributed or parallel set-up in script?: No

### Who can help
 Trainer: @sgugger  tensorflow: @jplu 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
1.  Install Tensorflow 2.3.0, Transformers 3.0.2

1. Run the following code:

```python3
from transformers import TFGPT2LMHeadModel, TFTrainer, TFTrainingArguments
import tensorflow as tf

tfds_train_dataset = tf.data.Dataset.from_tensor_slices(
    tf.random.uniform([4000, 1024], minval=1, maxval=10, dtype=tf.int32))

model = TFGPT2LMHeadModel.from_pretrained("gpt2")

training_args = TFTrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
)

trainer = TFTrainer(
    model=model,
    args=training_args,
    train_dataset=tfds_train_dataset,
)

trainer.train()
```

2. Results in the following output + error:
```
2020-08-09 01:41:28.331697: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-09 01:41:30.461375: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-09 01:41:30.466239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-09 01:41:30.466271: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-09 01:41:30.468575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-09 01:41:30.470629: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-09 01:41:30.471013: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-09 01:41:30.473522: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-09 01:41:30.474947: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-09 01:41:30.481193: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-09 01:41:30.482710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-09 01:41:30.483080: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-09 01:41:30.512602: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3210790000 Hz
2020-08-09 01:41:30.514335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4c678f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-09 01:41:30.514408: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-09 01:41:30.648534: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4c92000 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-09 01:41:30.648597: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-08-09 01:41:30.650365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-08-09 01:41:30.650446: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-09 01:41:30.650523: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-09 01:41:30.650586: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-09 01:41:30.650646: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-09 01:41:30.650708: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-09 01:41:30.650767: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-09 01:41:30.650829: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-09 01:41:30.653179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-09 01:41:30.653232: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-09 01:41:31.392168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-09 01:41:31.392212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-08-09 01:41:31.392225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-08-09 01:41:31.393566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7389 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-08-09 01:41:34.003855: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2020-08-09 01:41:34.145974: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
All model checkpoint weights were used when initializing TFGPT2LMHeadModel.

All the weights of TFGPT2LMHeadModel were initialized from the model checkpoint at gpt2.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.
Traceback (most recent call last):
  File "gpt2-training_bug.py", line 26, in <module>
    trainer.train()
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/transformers/trainer_tf.py", line 412, in train
    for step, training_loss in enumerate(self._training_steps(train_ds, optimizer)):
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/transformers/trainer_tf.py", line 459, in _training_steps
    for i, loss in enumerate(self._accumulate_next_gradients(ds)):
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/transformers/trainer_tf.py", line 492, in _accumulate_next_gradients
    yield _accumulate_next()
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: in user code:

    /home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/transformers/trainer_tf.py:486 _accumulate_next  *
        per_replica_features, per_replica_labels = next(iterator)
    /home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:503 __iter__
        self._disallow_iteration()
    /home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:496 _disallow_iteration
        self._disallow_when_autograph_enabled("iterating over `tf.Tensor`")
    /home/gabriel/venv/GPT-Hug/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Start Training

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
The following bug on Tensorflow could be related: https://github.com/tensorflow/tensorflow/issues/42119

Was just a Dataset setup issue. The correct setup for the Dataset can be seen here https://github.com/huggingface/transformers/issues/6551