how to make a multi-task deep neural network baseline using huggingface transformers?
Author: mobassir94
Pull request: no
Labels: name: wontfix

I was trying to build a [multi-task deep neural network][1] using [xlm roberta large model][2] for a multilingual classification problem. my training dataset contains 4 columns : 

1. ID

2. comment_text (according to id number,each users english comment is stored in this column. example comment : "you are a loser")

3. toxic (this column contains 1/0,0 means not toxic,1 means toxic)

4. personal_attack(this column also contains 0/1,,0 means the comment is not a personal attack type comment and 1 means opposite)

here is my models code : 

    def build_model(transformer, max_len=512):
        input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
        sequence_output = transformer(input_word_ids)[0]
        cls_token = sequence_output[:, 0, :]
        out = Dense(1, activation='sigmoid',name = 'y_train')(cls_token)
        out1 = Dense(1, activation='sigmoid',name = 'y_aux')(cls_token)
        model = Model(inputs=input_word_ids, outputs=[out, out1])
        model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])
        
        return model

here is the code for train and test dataset : 

    train_dataset = (
    tf.data.Dataset
    .from_tensor_slices((x_train,
                         
    { 'y_train':train.toxic.values,
    'y_aux':train.identity_attack.values}
                         
    ))
    .repeat()
    .shuffle(2048)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
    )
    
    test_dataset = (
        tf.data.Dataset
        .from_tensor_slices(x_test)
        .batch(BATCH_SIZE)
    )

then for training model i used this code : 


    EPOCHS = 3
    n_steps = x_train.shape[0] // BATCH_SIZE
    train_history = model.fit(
        train_dataset,
        steps_per_epoch=n_steps,
        epochs=EPOCHS
    )

i don't wish to perform validation so just train_dataset was given for model.fit()

after 3 epoch i get performance like this : 

    Epoch 3/3
    1658/1658 [==============================] - 887s 535ms/step - loss: 0.0591 - y_train_loss: 0.0175 - y_aux_loss: 0.0416 - y_train_accuracy: 0.9940 - y_aux_accuracy: 0.9821

now in my test set i have 1 columns : 

1. comments( this column contains comments of non english language (remember in train set we only had English comments and here in test set all comments are non english)


so i expect my model to predict on these test set whether the given test set comment is toxic or not?
as you can see from 3rd epoch result that i am calculating y_train_accuracy: 0.9940 - y_aux_accuracy: 0.9821

now i want my model to predict y_test or toxic/not toxic only

for that i tried : 

    sub['toxic'] = model.predict(test_dataset, verbose=1)

sub is a dataframe that contains all the id of test set and using **test_dataset** i was trying to predict each and every test set comments but i get this error : 

     
    499/499 [==============================] - 126s 253ms/step
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-23-1dc84858379e> in <module>
    ----> 1 sub['toxic'] = model.predict(test_dataset, verbose=1)
          2 sub.to_csv('submission.csv', index=False)
    
    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in __setitem__(self, key, value)
       2936         else:
       2937             # set column
    -> 2938             self._set_item(key, value)
       2939 
       2940     def _setitem_slice(self, key, value):
    
    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in _set_item(self, key, value)
       2998 
       2999         self._ensure_valid_index(value)
    -> 3000         value = self._sanitize_column(key, value)
       3001         NDFrame._set_item(self, key, value)
       3002 
    
    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in _sanitize_column(self, key, value, broadcast)
       3634 
       3635             # turn me into an ndarray
    -> 3636             value = sanitize_index(value, self.index, copy=False)
       3637             if not isinstance(value, (np.ndarray, Index)):
       3638                 if isinstance(value, list) and len(value) > 0:
    
    /opt/conda/lib/python3.7/site-packages/pandas/core/internals/construction.py in sanitize_index(data, index, copy)
        609 
        610     if len(data) != len(index):
    --> 611         raise ValueError("Length of values does not match length of index")
        612 
        613     if isinstance(data, ABCIndexClass) and not copy:
    
    ValueError: Length of values does not match length of index


now i have 4 questions :

1. is my implementation correct?

2. why i am getting that error? if i treat this problem as simple multilingual classification task like compute 1 loss for y true then i get no error at all,so where i am having trouble?

3. how can i solve the issue?

4. as it is my first time with multi task learning using huggingface transformers,what are your suggestions for updating my model so that it can generalize better?



  [1]: https://arxiv.org/abs/1706.05098
  [2]: https://huggingface.co/jplu/tf-xlm-roberta-large


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.