[Question]  How to serialize and load a trained RoBERTa model?
Author: hapazv
Pull request: no

First, I apologize for my English, I am learning the language, I am also learning a little about the networks transformers, tensorflow, keras, Bert and Roberta, I am new to this.

For a kaggle challenge, I wrote a code based on RoBERTa, the results were very good, but wanting to replicate the same code in Colab Pro, it was not possible due to the version of TPU's found in the tool, so I decided to save the weights from Kaggle's training to load them into Colab, but I haven't been able to do it, I'm doing something wrong and I don't understand what's going on.


At the end I will leave the link of the code in Kaggle, so soon I will put in broad strokes as it is written:
`
```
import numpy as np 
import pandas as pd
from transformers import AutoTokenizer, TFAutoModel
import tensorflow as tf
import numpy as np
import pandas as pd
from tensorflow.keras.layers import GlobalAveragePooling1D, Dense
from tensorflow.keras import  Model
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
`
`
modelo = 'joeddav/xlm-roberta-large-xnli'
tokenizer = AutoTokenizer.from_pretrained(modelo)

def token(x):
    tokens = list(tokenizer.tokenize(x))
    tokens.append('
')
    t = tokenizer.convert_tokens_to_ids(tokens)
    return t
def roberta_encode(hypotheses, premises, tokenizer):
  Pad = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
  sentence1 = tf.ragged.constant([token(s) for s in np.array(hypotheses)],  dtype=tf.int32)
  sentence2 = tf.ragged.constant([token(s) for s in np.array(premises)], dtype=tf.int32)
  cls = [tokenizer.convert_tokens_to_ids([tokenizer.cls_token])]*sentence1.shape[0]
  tokens = tf.concat([cls, sentence1, sentence2], axis=-1)
  tokens = tokens[:, :max_len] #quitar para la version full
  
  tokens = tokens.to_tensor(default_value=Pad)
  pad = max_len - tf.shape(tokens)[1]
  tokens = tf.pad(tokens, [[0, 0], [0, pad]], constant_values=Pad)
  input_word_ids = tf.reshape(tokens, [-1, max_len])

   
  input_mask = tf.cast(input_word_ids != Pad, tf.int32)
  input_mask = tf.reshape(input_mask, [-1, max_len])

  
  input_type_ids = tf.concat([tf.zeros_like(cls), tf.zeros_like(sentence1), tf.ones_like(sentence2)], 
                             axis=-1).to_tensor()
```

`
`
 ```
 inputs = {
    'input_word_ids': input_word_ids,
    'input_mask': input_mask,
    'input_type_ids': input_type_ids}

  return inputs

def build_dataset(x, y, mode, batch_size):#función vista en varios notebook's
    if mode == "train":
        dataset = (
            tf.data.Dataset
            .from_tensor_slices((x, y))
            .repeat()
            .shuffle(5678)
            .batch(batch_size)
            .prefetch(tf.data.experimental.AUTOTUNE)
        )
    elif mode == "valid":
        dataset = (
            tf.data.Dataset
            .from_tensor_slices((x, y))
            .batch(batch_size)
            .cache()
            .prefetch(tf.data.experimental.AUTOTUNE)
        )
    elif mode == "test":
        dataset = (
            tf.data.Dataset
            .from_tensor_slices(x)
            .batch(batch_size)
            )
    else:
        raise NotImplementedError
    return dataset
`
def build_model(model, max_len):
    
    tf.keras.backend.clear_session()
    tf.random.set_seed(0)
    
    with strategy.scope():
        
        input_word_ids = tf.keras.Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
        model = TFAutoModel.from_pretrained(modelo)
        roberta = model([input_word_ids])[0]
        output = GlobalAveragePooling1D()(roberta)
        output = Dense(3, activation='softmax')(output)
        
        model = Model(inputs=[input_word_ids], outputs = output)
        model.compile(optimizer=Adam(lr=1e-5), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    
    model.summary()
    
    return model

model = build_model(modelo, max_len)
Some layers from the model checkpoint at joeddav/xlm-roberta-large-xnli were not used when initializing TFXLMRobertaModel: ['classifier']
- This IS expected if you are initializing TFXLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFXLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFXLMRobertaModel were initialized from the model checkpoint at joeddav/xlm-roberta-large-xnli.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFXLMRobertaModel for predictions without further training.
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_word_ids (InputLayer)  [(None, 120)]             0         
_________________________________________________________________
tfxlm_roberta_model (TFXLMRo TFBaseModelOutputWithPool 559890432 
_________________________________________________________________
global_average_pooling1d (Gl (None, 1024)              0         
_________________________________________________________________
dense (Dense)                (None, 3)                 3075      
=================================================================
Total params: 559,893,507
Trainable params: 559,893,507
Non-trainable params: 0
________________________________________________________

steps_per_epoch = len(x_train) // batch_size
stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', verbose=1, patience=2, mode='min', restore_best_weights=True)

model.fit(train_dataset, validation_data=valid_dataset, steps_per_epoch=steps_per_epoch, epochs=4, callbacks=[stop])
Epoch 1/4
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['input_mask', 'input_type_ids'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:430: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 256002048 elements. This may consume a large amount of memory.
  num_elements)
2487/2487 [==============================] - 852s 284ms/step - loss: 0.2515 - accuracy: 0.9074 - val_loss: 2.3169 - val_accuracy: 0.4474
Epoch 2/4
2487/2487 [==============================] - 683s 275ms/step - loss: 0.1742 - accuracy: 0.9391 - val_loss: 2.1128 - val_accuracy: 0.4446
Epoch 3/4
2487/2487 [==============================] - 685s 276ms/step - loss: 0.1359 - accuracy: 0.9527 - val_loss: 2.6941 - val_accuracy: 0.4377
Epoch 4/4
2487/2487 [==============================] - 685s 276ms/step - loss: 0.1070 - accuracy: 0.9631 - val_loss: 2.6835 - val_accuracy: 0.4423
Restoring model weights from the end of the best epoch.
Epoch 00004: early stopping
<tensorflow.python.keras.callbacks.History at 0x7f51f47b7150>`

```

try saving the weights like this:

`model.save('RobertaClasi.hdf5')`


but once it gave me an error message and on another occasion when wanting to load the model it was not possible.

I really appreciate any indication and at the end I leave the link of the code in kaggle.

[https://www.kaggle.com/hugoarmandopazvivas/contradictory-my-dear-watson-hapv?scriptVersionId=62195702](url)

@Rocketknight1 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.