Unable to train xlnet with tensorflow
Author: nicolas-ferland
Pull request: no

## Environment info
- `transformers` version: '2.0.0'
- Platform: jupyter notebook
- Python version: 3.7.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.1.0 GPU
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger, @TevenLeScao, @jplu

## Information

Model I am using (Bert, XLNet ...): XLNet

The problem arises when using:
my own modified scripts: (give details below)

The tasks I am working on is:
my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
# I get my input, output from a dataframe. It's just a series of text and a series of
# integers representing classes.
x = df['description']
y_label =  pd.Categorical(df['target'])
y_cat = y_label.categories
y = y_label.codes
n_label = len(y_cat)

# I use the tokenizer. Then convert it to a numpy array
xlnet_tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
train_tokenized_inputs = [xlnet_tokenizer.encode(text)
         for text in x.values.tolist()]
# It needs to be at least 1 and no more than 2000
train_max_length = max(1,min(np.array([len(inp) for inp in train_tokenized_inputs]).max(), 2000))
train_padded_inputs = (tf.keras.preprocessing.sequence.pad_sequences(train_tokenized_inputs, maxlen=train_max_length, 
                                                           value=0,
                                                      padding='post', truncating='post',dtype='int32'))

# I use the xlnet model
clf = TFAutoModelForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=n_label)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
clf.compile(optimizer='adam',loss=loss)
clf.fit(x=train_padded_inputs, y=y,
    batch_size=32,
    epochs=1,
    verbose=1,
    callbacks=None,
    validation_split=0.2,
    validation_data=None,
    shuffle=True,
    class_weight=None,
    sample_weight=None,
    initial_epoch=0,
    steps_per_epoch=None,
    validation_steps=None,
    validation_freq=1,
    max_queue_size=10,
    workers=1,
    use_multiprocessing=False,)
```

The error message is:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-68-c147be84f56e> in <module>
     15     max_queue_size=10,
     16     workers=1,
---> 17     use_multiprocessing=False,)

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820 
    821   def evaluate(self,

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    233           max_queue_size=max_queue_size,
    234           workers=workers,
--> 235           use_multiprocessing=use_multiprocessing)
    236 
    237       total_samples = _get_total_number_of_samples(training_data_adapter)

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    550         batch_size=batch_size,
    551         check_steps=False,
--> 552         steps=steps_per_epoch)
    553     (x, y, sample_weights,
    554      val_x, val_y,

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name, steps, validation_split, shuffle, extract_tensors_from_dataset)
   2344     # First, we build the model on the fly if necessary.
   2345     if not self.inputs:
-> 2346       all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
   2347       is_build_called = True
   2348     else:

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _build_model_with_inputs(self, inputs, targets)
   2570     else:
   2571       cast_inputs = inputs
-> 2572     self._set_inputs(cast_inputs)
   2573     return processed_inputs, targets, is_dict_inputs
   2574 

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _set_inputs(self, inputs, outputs, training)
   2657           kwargs['training'] = training
   2658       try:
-> 2659         outputs = self(inputs, **kwargs)
   2660       except NotImplementedError:
   2661         # This Model or a submodel is dynamic and hasn't overridden

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    771                     not base_layer_utils.is_in_eager_or_tf_function()):
    772                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 773                     outputs = call_fn(cast_inputs, *args, **kwargs)
    774                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    775                     # circular dependencies.

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

TypeError: in converted code:

    /opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py:916 call  *
        output = self.sequence_summary(output)
    /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:773 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py:459 call  *
        output = self.first_dropout(output)
    /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py:416 converted_call
        return py_builtins.overload_of(f)(*args)

    TypeError: 'NoneType' object is not callable
```

In addition, I tried to use TFTrainer in case I could solve my problem with it.

`from transformers import TFTrainer`

Gets this error message

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-51-aece35bcf827> in <module>
----> 1 from transformers import TFTrainer

ImportError: cannot import name 'TFTrainer' from 'transformers' (/opt/conda/lib/python3.7/site-packages/transformers/__init__.py)
```


## Expected behavior

I expect the code to run and the model to be fine-tuned on my dataset.

I expect that I shouldn't need the TFTrainer as the explanation on huggingface.co says the model is a standard tensorflow 2 layer. But I expect that I should be able to import it.


Comments:
Hello! Can you try with master instead of the old `2.0.0` release? In order to know if the problem is still here or not.

By "with master", do you mean installed from source?

git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .

After installing from source transformers 4.2.0.dev0,

I have this error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-1503c944af5c> in <module>
----> 1 from transformers import AutoTokenizer, TFAutoModel

~/transformers/src/transformers/__init__.py in <module>
     38 
     39 # Data
---> 40 from .data import (
     41     DataProcessor,
     42     InputExample,

~/transformers/src/transformers/data/__init__.py in <module>
     18 
     19 from .metrics import glue_compute_metrics, xnli_compute_metrics
---> 20 from .processors import (
     21     DataProcessor,
     22     InputExample,

~/transformers/src/transformers/data/processors/__init__.py in <module>
     18 
     19 from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
---> 20 from .squad import SquadExample, SquadFeatures, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features
     21 from .utils import DataProcessor, InputExample, InputFeatures, SingleSentenceClassificationProcessor
     22 from .xnli import xnli_output_modes, xnli_processors, xnli_tasks_num_labels

~/transformers/src/transformers/data/processors/squad.py in <module>
     22 
     23 from ...file_utils import is_tf_available, is_torch_available
---> 24 from ...models.bert.tokenization_bert import whitespace_tokenize
     25 from ...tokenization_utils_base import BatchEncoding, PreTrainedTokenizerBase, TruncationStrategy
     26 from ...utils import logging

~/transformers/src/transformers/models/bert/__init__.py in <module>
     43 
     44 if is_tf_available():
---> 45     from .modeling_tf_bert import (
     46         TF_BERT_PRETRAINED_MODEL_ARCHIVE_LIST,
     47         TFBertEmbeddings,

~/transformers/src/transformers/models/bert/modeling_tf_bert.py in <module>
     21 import tensorflow as tf
     22 
---> 23 from ...activations_tf import get_tf_activation
     24 from ...file_utils import (
     25     MULTIPLE_CHOICE_DUMMY_INPUTS,

~/transformers/src/transformers/activations_tf.py in <module>
     66     "gelu": tf.keras.layers.Activation(gelu),
     67     "relu": tf.keras.activations.relu,
---> 68     "swish": tf.keras.activations.swish,
     69     "silu": tf.keras.activations.swish,
     70     "gelu_new": tf.keras.layers.Activation(gelu_new),

AttributeError: module 'tensorflow_core.python.keras.api._v2.keras.activations' has no attribute 'swish'
```

It's due to the line of code:
from transformers import AutoTokenizer, TFAutoModel

The next release of transformers (from source) now requires TF >= 2.3

It seems to work now, but I have a lot of warnings. Should I be worried about any of them?

```
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fb1e4e82280>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fb1e4e82280>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fb1e4e82280>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.
```

They look ok  for me!

I'm fine-tuning the model on my dataset and the accuracy is 0.02 after one epoch and it didn't really change during training. Also, it takes 7 hours per epoch. I'm wondering if the low accuracy might be due to the things mentioned in the warnings.

`WARNING:tensorflow:Gradients do not exist for variables ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0'] when minimizing the loss.`
If there are no gradients, it cannot learn. Do you know if it's really correct that all those layers have no gradient? I would expect the layers to have gradients.

```
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fa069898210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fa069898210>> and will run it as-is.
```
This warning seems to say that it is a bug worth mentioning to the TensorFlow team. Could it be the cause of the bad training time?

`The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).`
Do those parameters must be set? I have no set them or tried to modify them, I'm simply using the .fit method that all tensorflow 2 models have.


Also, do we need to use training=True somewhere? It's mentioned to use it when using the call of the model, but I'm calling .fit() rather than using the model's call, so I don't have this option as far as I know.

I'm asking because the training doesn't seem to be working and I'm wondering if it could be the problem.

Unfortunately, no issues for me with `TFXLNetForSequenceClassification`, just test over the MRPC dataset and got around 0.93 accuracy on training and 0.83 accuracy on validation. The version I used is the master branch from source.

The issue might come from the way you are training the model. Are you using the same one that you shared in your first post?

yes, but some parameters are different. For example, I needed a batchsize of 1 to fit in memory. Even 2 crashes. Also, I'm using callbacks. But I don't think it can make that the model doesn't learn.

        # Save the model after each epoch.
        ModelCheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(
            filepath = self.params['save_model_weight_filepath']+'_{epoch:02d}.hdf5', 
            monitor='val_loss', verbose=0, save_best_only=False,
            save_weights_only=False, mode='auto', save_freq='epoch'
        )
        
        # Stop when val loss stops decreasing.
        EarlyStopping_callback = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss', min_delta=self.params['min_delta'], patience=self.params['patience'], verbose=0,
    mode='auto', baseline=None, restore_best_weights=True

        history = self.clf.fit(x=padded_inputs, y=y,
            batch_size=1,
            epochs=40,
            verbose=1,
            validation_split=0.2,
            max_queue_size=10,
            workers=-1,
            use_multiprocessing=True,
               callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])

After epoch 2, the accuracies and losses are loss: 6.7701 - accuracy: 0.0197 - val_loss: 11.5031 - val_accuracy: 0.0025

Epoch 3 is still in process with loss: 6.7662 - accuracy: 0.0204

It doesn't seem to learn at all.

Also, I have this warning 
`WARNING:tensorflow:Callbacks method `on_test_batch_end` is slow compared to the batch time (batch time: 0.0089s vs `on_test_batch_end` time: 0.6593s). Check your callbacks.`
but none of my callbacks are used on batch_end, they are used on epoch ends, so infrequently and shouldn't affect the time too much.

Ok, from what I see in your script, the reason why your model don't learn anything is because the labels are not seen by the model which is normal with the way you set your dataset. The models in the lib have to be feed in a specific way, the data have to be a `Tuple(x, y)` where `x` can be either a list or a dict with tf.Tensor or np.ndarray, same for `y`. And then feed your model with:

```python
history = model.fit(
    train_dataset,
    epochs=3,
)
```

You can take example on how to do this in our examples or on our datasets website https://huggingface.co/docs/datasets/torch_tensorflow.html to know how to format your dataset.

I tried that (using a tuple with x and y, my x and y were already numpy arrays) and I got an error.

```
~/ticket-analysis-releasev3/ticket-analysis/src/model/xlnet/xlnet.py in fit(self, df)
    181             workers=self.params['workers'],
    182             use_multiprocessing=self.params['use_multiprocessing'],
--> 183                callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])
    184         self.history_df = pd.DataFrame({'epochs':history.epoch, 'loss': history.history['loss'], 
    185                                    'validation_loss': history.history['val_loss'], 'accuracy': history.history['accuracy'],

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698 
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:757 train_step
        self.trainable_variables)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:2737 _minimize
        trainable_variables))
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:562 _aggregate_gradients
        filtered_grads_and_vars = _filter_grads(grads_and_vars)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:1271 _filter_grads
        ([v.name for _, v in grads_and_vars],))

    ValueError: No gradients provided for any variable: ['tfxl_net_for_sequence_classification/transformer/mask_emb:0', 'tfxl_net_for_sequence_classification/transformer/word_embedding/weight:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._0/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._1/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._2/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._3/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._4/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._5/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._6/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._7/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._8/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._9/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._10/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/q:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/k:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/v:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/o:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_r_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_s_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/r_w_bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/seg_embed:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/rel_attn/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_norm/gamma:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_norm/beta:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_1/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_1/bias:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_2/kernel:0', 'tfxl_net_for_sequence_classification/transformer/layer_._11/ff/layer_2/bias:0', 'tfxl_net_for_sequence_classification/sequence_summary/summary/kernel:0', 'tfxl_net_for_sequence_classification/sequence_summary/summary/bias:0', 'tfxl_net_for_sequence_classification/logits_proj/kernel:0', 'tfxl_net_for_sequence_classification/logits_proj/bias:0'].
```

I also tried with dataset, but I got this error

```
Some layers from the model checkpoint at xlnet-base-cased were not used when initializing TFXLNetForSequenceClassification: ['lm_loss']
- This IS expected if you are initializing TFXLNetForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFXLNetForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFXLNetForSequenceClassification were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: ['sequence_summary', 'logits_proj']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch 1/40
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f21748f1210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f21748f1210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-db016531efec> in <module>
      1 # Try to use tensorflow dataset
      2 st = time.time()
----> 3 model.fit(df)
      4 print('time', time.time()-st)

~/ticket-analysis-releasev3/ticket-analysis/src/model/xlnet/xlnet.py in fit(self, df)
    187             workers=self.params['workers'],
    188             use_multiprocessing=self.params['use_multiprocessing'],
--> 189                callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])
    190         self.history_df = pd.DataFrame({'epochs':history.epoch, 'loss': history.history['loss'], 
    191                                    'validation_loss': history.history['val_loss'], 'accuracy': history.history['accuracy'],

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698 
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /home/jovyan/transformers/src/transformers/models/xlnet/modeling_tf_xlnet.py:1452 call  *
        transformer_outputs = self.transformer(
    /home/jovyan/transformers/src/transformers/models/xlnet/modeling_tf_xlnet.py:625 call  *
        inputs["input_ids"] = tf.transpose(inputs["input_ids"], perm=(1, 0))
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2107 transpose_v2
        return transpose(a=a, perm=perm, name=name, conjugate=conjugate)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2188 transpose
        return transpose_fn(a, perm, name=name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:11535 transpose
        "Transpose", x=x, perm=perm, name=name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal
        compute_device)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal
        op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1975 __init__
        control_input_ops, op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1815 _create_c_op
        raise ValueError(str(e))

    ValueError: Dimension must be 3 but is 2 for '{{node tfxl_net_for_sequence_classification/transformer/transpose}} = Transpose[T=DT_INT32, Tperm=DT_INT32](IteratorGetNext, tfxl_net_for_sequence_classification/transformer/transpose/perm)' with input shapes: [1,11929,2000], [2].
```

For dataset, I used this code
```
        train_dataset = (padded_inputs, y)
        train_dataset, val_dataset = train_test_split(train_dataset, test_size=0.02)
        train_dataset = tf.data.Dataset.from_tensors(train_dataset)
        val_dataset = tf.data.Dataset.from_tensors(val_dataset)
        
        
        # Fit model
        self.clf = TFAutoModelForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=self.n_label)
        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
        self.clf.compile(optimizer='adam',loss=loss,metrics=['accuracy'])
        history = self.clf.fit(
            train_dataset, #x=padded_inputs, y=y,
            validation_data = val_dataset,
            batch_size=self.params['batchsize'],
            epochs=self.params['epochs'],
            verbose=1,
            #validation_split=self.params['validation_split'],
            max_queue_size=self.params['max_queue_size'],
            workers=self.params['workers'],
            use_multiprocessing=self.params['use_multiprocessing'],
               callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])
```

This is still not ok. Here an example for MRPC, from which your can take inspiration from:
```
import tensorflow as tf
from datasets import load_dataset
from transformers import AutoTokenizer
dataset = load_dataset('glue', 'mrpc', split='train')
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = dataset.map(lambda e: tokenizer(e['sentence1'], truncation=True, padding='max_length'), batched=True)

dataset.set_format(type='numpy', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
features = {x: dataset[x] for x in ['input_ids', 'token_type_ids', 'attention_mask']}
tfdataset = tf.data.Dataset.from_tensor_slices((features, dataset["label"])).batch(1)
```

And your data must have the shape:
```
<BatchDataset shapes: ({input_ids: (None, 512), token_type_ids: (None, 512), attention_mask: (None, 512)}, (None,)), types: ({input_ids: tf.int64, token_type_ids: tf.int64, attention_mask: tf.int64}, tf.int32)>
```
Here it is a tuple, where the first element is a dict that has tensors (built from numpy arrays), and the label is a label id.

What kind of format or object is the dataset obtained from
dataset = load_dataset('glue', 'mrpc', split='train')
?
I'm not loading a public dataset but using my own so I can't take this part from the code. Do you know how I can generate it from a input numpy array and a label numpy array?

Also, I don't think xlnet uses 'token_type_ids', 'attention_mask', should I use
dataset.set_format(type='numpy', columns=['input_ids', 'label'])
features = {x: dataset[x] for x in ['input_ids']}


The example I gave is just to show you how should looks like your dataset. And yes XLNet can take both attention_mask and token_type_ids arguments. The steps are simple:
1. Tokenize your dataset
2. Create a tf.data.Dataset and format it to make it looking like I showed you: `({"input_ids": [[ex1],[ex2],...], "attention_mask":[[ex1],[ex2],...], "token_type_ids":[ex1],[ex2],...]}, [label_id_ex1, label_id_ex_2,...])`
3. Run your training with `model.fit(training_dataset, epochs=3)`

And that's it.



Okay, I'm using

            x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=0.02)
            tokenized_inputs = [xlnet_tokenizer.encode(text) for text in x_train.values.tolist()]
            max_length = max(1,min(np.array([len(inp) for inp in tokenized_inputs]).max(), self.params['MAX_LENGTH']))
            padded_inputs = (tf.keras.preprocessing.sequence.pad_sequences(tokenized_inputs, maxlen=max_length, 
                                                                       value=0,
                                                                  padding='post', truncating='post',dtype='int32'))
            train_dataset = tf.data.Dataset.from_tensor_slices(({"input_ids":padded_inputs}, y_train)).batch(1)
            val_tokenized_inputs = [xlnet_tokenizer.encode(text) for text in x_val.values.tolist()]
            val_padded_inputs = (tf.keras.preprocessing.sequence.pad_sequences(val_tokenized_inputs, maxlen=max_length, 
                                                                       value=0,
                                                                  padding='post', truncating='post',dtype='int32'))
            val_dataset = tf.data.Dataset.from_tensor_slices(({"input_ids":x_val}, y_val)).batch(1)
            print('dataset',train_dataset,val_dataset)
        history = self.clf.fit(
            train_dataset,
            validation_data = val_dataset,
            batch_size=self.params['batchsize'],
            epochs=self.params['epochs'],
            verbose=1,
            max_queue_size=self.params['max_queue_size'],
            workers=self.params['workers'],
            use_multiprocessing=self.params['use_multiprocessing'],
               callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])

The format is <BatchDataset shapes: ({input_ids: (None, 2000)}, (None,)), types: ({input_ids: tf.int32}, tf.int16)>

Should it work?

Also, for the prediction, is it correct to use
pred_dataset = tf.data.Dataset.from_tensor_slices(({"input_ids":x_train})).batch(1)
since the model should not use the y at prediction time.

I'm getting this error right now
```
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-6-db016531efec> in <module>
      1 # Try to use tensorflow dataset
      2 st = time.time()
----> 3 model.fit(df)
      4 print('time', time.time()-st)

~/ticket-analysis-releasev3/ticket-analysis/src/model/xlnet/xlnet.py in fit(self, df)
    168 
    169         # Fit model
--> 170         self.clf = TFAutoModelForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=self.n_label)
    171         loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    172         self.clf.compile(optimizer='adam',loss=loss,metrics=['accuracy'])

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    384             return TFBertForSequenceClassification.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    385         elif 'xlnet' in pretrained_model_name_or_path:
--> 386             return TFXLNetForSequenceClassification.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    387         elif 'xlm' in pretrained_model_name_or_path:
    388             return TFXLMForSequenceClassification.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    266 
    267         inputs = tf.constant([[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]])
--> 268         ret = model(inputs, training=False)  # build the network with dummy inputs
    269 
    270         assert os.path.isfile(resolved_archive_file), "Error retrieving file {}".format(resolved_archive_file)

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    983 
    984         with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985           outputs = call_fn(inputs, *args, **kwargs)
    986 
    987         if self._activity_regularizer:

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py in call(self, inputs, **kwargs)
    911 
    912     def call(self, inputs, **kwargs):
--> 913         transformer_outputs = self.transformer(inputs, **kwargs)
    914         output = transformer_outputs[0]
    915 

~/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    983 
    984         with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985           outputs = call_fn(inputs, *args, **kwargs)
    986 
    987         if self._activity_regularizer:

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py in call(self, inputs, attention_mask, mems, perm_mask, target_mapping, token_type_ids, input_mask, head_mask, training)
    607 
    608         ##### Positional encoding
--> 609         pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz, dtype=dtype_float)
    610         pos_emb = self.dropout(pos_emb, training=training)
    611 

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py in relative_positional_encoding(self, qlen, klen, bsz, dtype)
    490             if self.clamp_len > 0:
    491                 fwd_pos_seq = tf.clip_by_value(fwd_pos_seq, -clamp_len, clamp_len)
--> 492             pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz)
    493 
    494         return pos_emb

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py in positional_embedding(pos_seq, inv_freq, bsz)
    437     @staticmethod
    438     def positional_embedding(pos_seq, inv_freq, bsz=None):
--> 439         sinusoid_inp = tf.einsum('i,d->id', pos_seq, inv_freq)
    440         pos_emb = tf.concat([tf.sin(sinusoid_inp), tf.cos(sinusoid_inp)], axis=-1)
    441         pos_emb = pos_emb[:, None, :]

~/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~/.local/lib/python3.7/site-packages/tensorflow/python/ops/special_math_ops.py in einsum(equation, *inputs, **kwargs)
    682       - number of inputs or their shapes are inconsistent with `equation`.
    683   """
--> 684   return _einsum_v2(equation, *inputs, **kwargs)
    685 
    686 

~/.local/lib/python3.7/site-packages/tensorflow/python/ops/special_math_ops.py in _einsum_v2(equation, *inputs, **kwargs)
   1111       if ellipsis_label:
   1112         resolved_equation = resolved_equation.replace(ellipsis_label, '...')
-> 1113       return gen_linalg_ops.einsum(inputs, resolved_equation)
   1114 
   1115     # Send fully specified shapes to opt_einsum, since it cannot handle unknown

~/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_linalg_ops.py in einsum(inputs, equation, name)
   1086       return _result
   1087     except _core._NotOkStatusException as e:
-> 1088       _ops.raise_from_not_ok_status(e, name)
   1089     except _core._FallbackException:
   1090       pass

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6841   message = e.message + (" name: " + name if name is not None else "")
   6842   # pylint: disable=protected-access
-> 6843   six.raise_from(core._status_to_exception(e.code, message), None)
   6844   # pylint: enable=protected-access
   6845 

/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

InternalError: Blas xGEMM launch failed : a.shape=[1,1,10], b.shape=[1,1,384], m=10, n=384, k=1 [Op:Einsum]
```

This error means that your GPU doesn't have enough RAM to run an einsum operation. But yes, your dataset looks better. Still, you are not properly using the tokenizer, use a proper way to use it:
```
from transformers import XLNetTokenizer
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
tokenizer("hello")
``` 
To get tokenized data that looks like:
```
{'input_ids': [24717, 4, 3], 'token_type_ids': [0, 0, 2], 'attention_mask': [1, 1, 1]}
```

I'm getting

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-4dd755fb81bd> in <module>
      1 from transformers import XLNetTokenizer
      2 tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
----> 3 tokenizer("hello")

TypeError: 'XLNetTokenizer' object is not callable
```

Which version of transformers are you using?

It went back to '2.0.0'. I don't know why but I'm trying to et 4.2.0 again.

Please stick to the 4.2.0 release :)

Do we need int64 or is int32 enough? I'm memory limited so anything that allow me to use less memory would help.

int32 is enough.

It is running again, but still not training.

I'm using
```
            x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=self.params['validation_split'])
            # train
            tokenized_inputs = xlnet_tokenizer(x_train.values.tolist(), padding=True, max_length=self.params['MAX_LENGTH'], truncation=True)
            numpy_inputs = {x:np.array(tokenized_inputs[x]) for x in tokenized_inputs.keys()}
            train_dataset = tf.data.Dataset.from_tensor_slices((numpy_inputs, y_train)).batch(1)
            # val
            tokenized_inputs = xlnet_tokenizer(x_val.values.tolist(), padding=True, max_length=self.params['MAX_LENGTH'], truncation=True)
            numpy_inputs = {x:np.array(tokenized_inputs[x]) for x in tokenized_inputs.keys()}
            val_dataset = tf.data.Dataset.from_tensor_slices((numpy_inputs, y_val)).batch(1)
            print(train_dataset,val_dataset)
        history = self.clf.fit(
            train_dataset,
            validation_data = val_dataset,
            batch_size=self.params['batchsize'],
            epochs=self.params['epochs'],
            verbose=1,
            max_queue_size=self.params['max_queue_size'],
            workers=self.params['workers'],
            use_multiprocessing=self.params['use_multiprocessing'],
               callbacks=[ModelCheckpoint_callback, EarlyStopping_callback])
```

The shapes are
`<BatchDataset shapes: ({input_ids: (None, 500), token_type_ids: (None, 500), attention_mask: (None, 500)}, (None,)), types: ({input_ids: tf.int64, token_type_ids: tf.int64, attention_mask: tf.int64}, tf.int16)> <BatchDataset shapes: ({input_ids: (None, 500), token_type_ids: (None, 500), attention_mask: (None, 500)}, (None,)), types: ({input_ids: tf.int64, token_type_ids: tf.int64, attention_mask: tf.int64}, tf.int16)>`

The results are:
epochs | loss | validation_loss | accuracy | validation_accuracy
-- | -- | -- | -- | --
0 | 6.980901 | 7.149147 | 0.015823 | 0.047779
1 | 7.054768 | 7.217787 | 0.015194 | 0.047779
2 | 7.099029 | 7.474302 | 0.014880 | 0.047779
3 | 7.145690 | 7.359528 | 0.015509 | 0.047779
4 | 7.183013 | 7.395905 | 0.013937 | 0.005448
5 | 7.210382 | 7.452353 | 0.016137 | 0.047779



Ok, now your data looks correct. Can you try with just:
```
self.clf.fit(train_dataset, epochs=self.params['epochs'])
```

If it still not working, try to use multiple other model such as Bert and see if it is still the cases. 

It's not working with that, and it's not working with 'bert-base-uncased' either.

This is the output for 'bert-base-uncased'.
```
Downloading: 100%
433/433 [00:02<00:00, 199B/s]

Downloading: 100%
232k/232k [00:00<00:00, 1.25MB/s]

Downloading: 100%
466k/466k [00:00<00:00, 1.32MB/s]

dataset <BatchDataset shapes: ({input_ids: (None, 500), token_type_ids: (None, 500), attention_mask: (None, 500)}, (None,)), types: ({input_ids: tf.int32, token_type_ids: tf.int32, attention_mask: tf.int32}, tf.int16)> <BatchDataset shapes: ({input_ids: (None, 500), token_type_ids: (None, 500), attention_mask: (None, 500)}, (None,)), types: ({input_ids: tf.int32, token_type_ids: tf.int32, attention_mask: tf.int32}, tf.int16)>
Downloading: 100%
536M/536M [00:11<00:00, 45.4MB/s]

All model checkpoint layers were used when initializing TFBertForSequenceClassification.

Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch 1/40
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f0c5c70d210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f0c5c70d210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
9543/9543 [==============================] - 1784s 187ms/step - loss: 6.9231 - accuracy: 0.0181
Epoch 2/40
9543/9543 [==============================] - 1451s 152ms/step - loss: 7.0684 - accuracy: 0.0153
Epoch 3/40
9543/9543 [==============================] - 1157s 121ms/step - loss: 7.1112 - accuracy: 0.0171
Epoch 4/40
9543/9543 [==============================] - 1160s 122ms/step - loss: 7.1445 - accuracy: 0.0170
Epoch 5/40
9543/9543 [==============================] - 1160s 122ms/step - loss: 7.1869 - accuracy: 0.0159
```



Is there anything else that could cause the model not to learn?

The problem seems not to come from the models, I would guess that they might come from the way you build your data before the creation of the tf.data.Dataset or from the data themselves (sometime we cannot learn anything from the data, it happend). But I cannot be sure of anything without being able to reproduce the same behavior on my side sorry.

That's strange because I can get 32% accuracy on validation data using a baseline model that finds the closest text and predicts its label. So it should be possible to learn from the data.

Unfortunately, I cannot really help without being able to reproduce the issue, I'm sorry :(

I'm trying with TFTrainer now

```
        # Preprocess
        train_dataset, val_dataset = self._preprocess(df, fit=True, retrain=False)
        
        # fit model
        training_args = TFTrainingArguments(
        output_dir='./results',          # output directory
        num_train_epochs=3,              # total number of training epochs
        per_device_train_batch_size=1,  # batch size per device during training
        per_device_eval_batch_size=1,   # batch size for evaluation
        warmup_steps=500,                # number of warmup steps for learning rate scheduler
        weight_decay=0.01,               # strength of weight decay
        logging_dir='./logs',            # directory for storing logs
        logging_steps=10,
        )
        
        with training_args.strategy.scope():
            self.clf = TFAutoModelForSequenceClassification.from_pretrained(self.params['transformer_model'], num_labels=self.n_label)

        self.trainer = TFTrainer(
            model=self.clf,                         # the instantiated 🤗 Transformers model to be trained
            args=training_args,                  # training arguments, defined above
            train_dataset=train_dataset,         # training dataset
            eval_dataset=val_dataset             # evaluation dataset
        )
        
        self.trainer.train()
```

I got this error:

```
TypeError                                 Traceback (most recent call last)
<ipython-input-7-6e1fecee0ced> in <module>
      1 st = time.time()
----> 2 model.fit(df)
      3 print('time', time.time()-st)

~/ticket-analysis-releasev3/ticket-analysis/src/model/xlnet/transformers_trainer.py in fit(self, df)
    176         )
    177 
--> 178         self.trainer.train()
    179 
    180     def predict(self, df=None):

~/transformers/src/transformers/trainer_tf.py in train(self)
    455         Train method to train the model.
    456         """
--> 457         train_ds = self.get_train_tfdataset()
    458 
    459         if self.args.debug:

~/transformers/src/transformers/trainer_tf.py in get_train_tfdataset(self)
    136 
    137         self.total_train_batch_size = self.args.train_batch_size * self.args.gradient_accumulation_steps
--> 138         self.num_train_examples = self.train_dataset.cardinality(self.train_dataset).numpy()
    139 
    140         if self.num_train_examples < 0:

TypeError: cardinality() takes 1 positional argument but 2 were given
```

It seems that train_dataset is not correct.

train_dataset and val_dataset  are made as before

```
            x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=self.params['validation_split'])
            # train
            tokenized_inputs = xlnet_tokenizer(x_train.values.tolist(), padding=True, max_length=self.params['MAX_LENGTH'], truncation=True)
            numpy_inputs = {x:np.array(tokenized_inputs[x]) for x in tokenized_inputs.keys()}
            train_dataset = tf.data.Dataset.from_tensor_slices((numpy_inputs, y_train)).batch(1)
            # val
            tokenized_inputs = xlnet_tokenizer(x_val.values.tolist(), padding=True, max_length=self.params['MAX_LENGTH'], truncation=True)
            numpy_inputs = {x:np.array(tokenized_inputs[x]) for x in tokenized_inputs.keys()}
            val_dataset = tf.data.Dataset.from_tensor_slices((numpy_inputs, y_val)).batch(1)
```

and have shape

`dataset <BatchDataset shapes: ({input_ids: (None, 329), token_type_ids: (None, 329), attention_mask: (None, 329)}, (None,)), types: ({input_ids: tf.int32, token_type_ids: tf.int32, attention_mask: tf.int32}, tf.int16)>`




Can you use transformers==4.2.0 and test?

I'm using 4.2.0dev0 obtained from 
pip install -e . 
in the github repo.

Should I do
pip install transformers==4.2.0
instead?

I'm getting this error with transformers==4.2.0

```
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1')
All model checkpoint layers were used when initializing TFBertForSequenceClassification.

Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
WARNING:tensorflow:From /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py:601: get_next_as_optional (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Iterator.get_next_as_optional()` instead.
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fd43ea31d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7fd43ea31d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
INFO:tensorflow:Error reported to Coordinator: in user code:

    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:633 apply_gradients  *
        gradients = self.training_step(features, labels, nb_instances_in_global_batch)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:616 training_step  *
        per_example_loss, _ = self.run_model(features, labels, True)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:719 run_model  *
        outputs = self.model(features, labels=labels, training=training)[:2]
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:1421 call  *
        outputs = self.bert(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:654 call  *
        embedding_output = self.embeddings(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:192 call  *
        return self._embedding(input_ids, position_ids, token_type_ids, inputs_embeds, training=training)
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:221 _embedding  *
        embeddings = self.LayerNorm(embeddings)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__  **
        outputs = call_fn(inputs, *args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1205 call
        scale, offset = _broadcast(self.gamma), _broadcast(self.beta)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1192 _broadcast
        return array_ops.reshape(v, broadcast_shape)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:195 reshape
        result = gen_array_ops.reshape(tensor, shape, name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:8234 reshape
        "Reshape", tensor=tensor, shape=shape, name=name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal
        compute_device)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal
        op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1975 __init__
        control_input_ops, op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1815 _create_c_op
        raise ValueError(str(e))

    ValueError: Cannot reshape a tensor with 768 elements to shape [1,1,329,1] (329 elements) for '{{node tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/ReadVariableOp, tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/shape)' with input shapes: [768], [4] and with input tensors computed as partial shapes: input[1] = [1,1,329,1].
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
    yield
  File "/home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/distribute/mirrored_run.py", line 323, in run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
  File "/home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 258, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:633 apply_gradients  *
        gradients = self.training_step(features, labels, nb_instances_in_global_batch)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:616 training_step  *
        per_example_loss, _ = self.run_model(features, labels, True)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:719 run_model  *
        outputs = self.model(features, labels=labels, training=training)[:2]
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:1421 call  *
        outputs = self.bert(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:654 call  *
        embedding_output = self.embeddings(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:192 call  *
        return self._embedding(input_ids, position_ids, token_type_ids, inputs_embeds, training=training)
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:221 _embedding  *
        embeddings = self.LayerNorm(embeddings)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__  **
        outputs = call_fn(inputs, *args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1205 call
        scale, offset = _broadcast(self.gamma), _broadcast(self.beta)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1192 _broadcast
        return array_ops.reshape(v, broadcast_shape)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:195 reshape
        result = gen_array_ops.reshape(tensor, shape, name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:8234 reshape
        "Reshape", tensor=tensor, shape=shape, name=name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal
        compute_device)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal
        op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1975 __init__
        control_input_ops, op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1815 _create_c_op
        raise ValueError(str(e))

    ValueError: Cannot reshape a tensor with 768 elements to shape [1,1,329,1] (329 elements) for '{{node tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/ReadVariableOp, tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/shape)' with input shapes: [768], [4] and with input tensors computed as partial shapes: input[1] = [1,1,329,1].

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-6e1fecee0ced> in <module>
      1 st = time.time()
----> 2 model.fit(df)
      3 print('time', time.time()-st)

~/ticket-analysis-releasev3/ticket-analysis/src/model/xlnet/transformers_trainer.py in fit(self, df)
    176         )
    177 
--> 178         self.trainer.train()
    179 
    180     def predict(self, df=None):

/opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py in train(self)
    547                         continue
    548 
--> 549                     self.distributed_training_steps(batch)
    550 
    551                     self.global_step = iterations.numpy()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698 
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py in bound_method_wrapper(*args, **kwargs)
   3733     # However, the replacer is still responsible for attaching self properly.
   3734     # TODO(mdan): Is it possible to do it here instead?
-> 3735     return wrapped_fn(*args, **kwargs)
   3736   weak_bound_method_wrapper = weakref.ref(bound_method_wrapper)
   3737 

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:672 distributed_training_steps  *
        self.args.strategy.run(self.apply_gradients, inputs)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:633 apply_gradients  *
        gradients = self.training_step(features, labels, nb_instances_in_global_batch)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:616 training_step  *
        per_example_loss, _ = self.run_model(features, labels, True)
    /opt/conda/lib/python3.7/site-packages/transformers/trainer_tf.py:719 run_model  *
        outputs = self.model(features, labels=labels, training=training)[:2]
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:1421 call  *
        outputs = self.bert(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:654 call  *
        embedding_output = self.embeddings(
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:192 call  *
        return self._embedding(input_ids, position_ids, token_type_ids, inputs_embeds, training=training)
    /opt/conda/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:221 _embedding  *
        embeddings = self.LayerNorm(embeddings)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__  **
        outputs = call_fn(inputs, *args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1205 call
        scale, offset = _broadcast(self.gamma), _broadcast(self.beta)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:1192 _broadcast
        return array_ops.reshape(v, broadcast_shape)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:195 reshape
        result = gen_array_ops.reshape(tensor, shape, name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:8234 reshape
        "Reshape", tensor=tensor, shape=shape, name=name)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal
        compute_device)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal
        op_def=op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1975 __init__
        control_input_ops, op_def)
    /home/jovyan/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1815 _create_c_op
        raise ValueError(str(e))

    ValueError: Cannot reshape a tensor with 768 elements to shape [1,1,329,1] (329 elements) for '{{node tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/ReadVariableOp, tf_bert_for_sequence_classification/bert/embeddings/LayerNorm/Reshape/shape)' with input shapes: [768], [4] and with input tensors computed as partial shapes: input[1] = [1,1,329,1].
```

Hello!

This seems a different issue for a different model than XLNet, can you give a bit more context please? Such as how to properly reproduce the error you get? Thanks :)

Sorry, I have been very busy with other stuffs so I put that on pause.

I tried 'bert-base-uncased' rather than XLNet last time and used TFTrainer rather than .fit from tenforflow because the model didn't seem to learn anything using only tensorflow.

Do you know if TFtrainer can help? And if yes, how to use it correctly?

You have multiple examples in the `examples` folder https://github.com/huggingface/transformers/tree/master/examples

Do I need to add positional encoding? Or is that done automatically by the tokenizer or the model?

Are there different positional encoding I could use?

You are not forced to provide positional ids, the model creates them for you by default. If you want to use your own ones you have to provide them to the model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.