Multilabel Classification with TFBertForSequenceClassification
Author: johnwu0604
Pull request: no
Labels: name: wontfix

I'm currently trying to train a multi label classifier, but in my trained model I'm get the same output no matter the input that I put in. 

I've modified the TFBertForSequenceClassification class to include a sigmoid activation output layer as shown below:

```
class TFBertForMultilabelClassification(TFBertPreTrainedModel):

    def __init__(self, config, *inputs, **kwargs):
        super(TFBertForMultilabelClassification, self).__init__(config, *inputs, **kwargs)
        self.num_labels = config.num_labels

        self.bert = TFBertMainLayer(config, name='bert')
        self.dropout = tf.keras.layers.Dropout(config.hidden_dropout_prob)
        self.classifier = tf.keras.layers.Dense(config.num_labels,
                                                kernel_initializer=get_initializer(config.initializer_range),
                                                name='classifier',
                                                activation='sigmoid')

    def call(self, inputs, **kwargs):
        outputs = self.bert(inputs, **kwargs)

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output, training=kwargs.get('training', False))
        logits = self.classifier(pooled_output)

        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here

        return outputs  # logits, (hidden_states), (attentions)
```

Here is a method which converts my InputExamples to InputFeatures for BERT:
```
def convert_examples_to_features(examples, tokenizer, label_list, max_seq_length):
    """Converts examples to features using specified tokenizer

    Args:
        examples (list): Examples to convert.
        tokenizer (obj): The tokenzier object.
        label_list (list): A list of all the labels.
        max_sequence_length (int): Maximum length of a sequence

    Returns:
        tf.Dataset: A tensorflow dataset.
    """

    features = []
    for ex_index, example in enumerate(examples):

        # Encode inputs using tokenizer
        inputs = tokenizer.encode_plus(
            example.text_a[:max_seq_length],
            add_special_tokens=True,
            max_length=max_seq_length,
            truncate_first_sequence=True 
        )
        input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]

        # The mask has 1 for real tokens and 0 for padding tokens. Only real tokens are attended to.
        attention_mask = [1] * len(input_ids)

        # Zero-pad up to the sequence length.
        padding_length = max_seq_length - len(input_ids)
        input_ids = input_ids + ([0] * padding_length)
        attention_mask = attention_mask + ([0] * padding_length)
        token_type_ids = token_type_ids + ([0] * padding_length)

        # Create features and add to feature list
        features.append(
                InputFeatures(input_ids=input_ids,
                              attention_mask=attention_mask,
                              token_type_ids=token_type_ids,
                              label=example.label))
    # Generator for creating tensorflow dataset
    def gen():
        for ex in features:
            yield  ({'input_ids': ex.input_ids,
                        'attention_mask': ex.attention_mask,
                        'token_type_ids': ex.token_type_ids},
                    ex.label)

    return tf.data.Dataset.from_generator(gen,
            ({'input_ids': tf.int32,
              'attention_mask': tf.int32,
              'token_type_ids': tf.int32},
             tf.int64),
            ({'input_ids': tf.TensorShape([max_seq_length]),
              'attention_mask': tf.TensorShape([max_seq_length]),
              'token_type_ids': tf.TensorShape([max_seq_length])},
             tf.TensorShape([len(label_list)])))
```

Then I used the following code to train my model:
```
# Get pretrained weights and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForMultilabelClassification.from_pretrained('bert-base-uncased', num_labels=len(label_list))

# Convert examples to features 
train_dataset = convert_examples_to_features(train_examples, tokenizer, label_list, max_seq_length)
valid_dataset = convert_examples_to_features(valid_examples, tokenizer, label_list, max_seq_length)
test_dataset = convert_examples_to_features(test_examples, tokenizer, label_list, max_seq_length)

# Shuffle train data and put into batches
train_dataset = train_dataset.shuffle(100).batch(batch_size)
valid_dataset = valid_dataset.batch(batch_size)
test_dataset = test_dataset.batch(batch_size)

# Prepare training: instantiate optimizer, loss and learning rate schedule 
optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
loss = tf.keras.losses.BinaryCrossentropy()
metric = tf.keras.metrics.CategoricalAccuracy()

# Compile the model
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate model
history = model.fit(train_dataset, epochs=num_epochs, validation_data=valid_dataset)

# Save the trained model 
if not os.path.exists(export_dir):
    os.makedirs(export_dir)
model.save_pretrained(export_dir)
```

I've also tried the model with a linear and relu activation function in addition to other optimizers and the result is still the same output no matter what input I put into the model. Does anyone have any insight to where my problem could be?

Comments:
Have you figured out what the problem is? I'm facing the same thing....

Have you figured out what the problem is? I'm facing the same thing....

Hello, did you figure out the solution here?

Hey guys, sorry for the late update. Here's my solution: I set a lower learning rate and the problem is fixed. It seems that when we do transfer learning, we cannot set a high learning rate because the model is not well connected to the softmax layer you add.(Just some intuition) In addition, it's also possible that you forget to call model.eval() to invalidate the Dropout layer or something. But this is not the case for me.

> Hey guys, sorry for the late update. Here's my solution: I set a lower learning rate and the problem is fixed. It seems that when we do transfer learning, we cannot set a high learning rate because the model is not well connected to the softmax layer you add.(Just some intuition) In addition, it's also possible that you forget to call model.eval() to invalidate the Dropout layer or something. But this is not the case for me.

Do you also use TFBertForSequenceClassification for multi-label classification?Multi-label classification requires sigmoid function.

> Hey guys, sorry for the late update. Here's my solution: I set a lower learning rate and the problem is fixed. It seems that when we do transfer learning, we cannot set a high learning rate because the model is not well connected to the softmax layer you add.(Just some intuition) In addition, it's also possible that you forget to call model.eval() to invalidate the Dropout layer or something. But this is not the case for me.

I used your method but it was unsuccessful

> > Hey guys, sorry for the late update. Here's my solution: I set a lower learning rate and the problem is fixed. It seems that when we do transfer learning, we cannot set a high learning rate because the model is not well connected to the softmax layer you add.(Just some intuition) In addition, it's also possible that you forget to call model.eval() to invalidate the Dropout layer or something. But this is not the case for me.
> 
> Do you also use TFBertForSequenceClassification for multi-label classification?Multi-label classification requires sigmoid function.

Typically, **sigmoid** function is used in binary classification problems, instead **softmax** function is used in multi-class classification problems

> > > Hey guys, sorry for the late update. Here's my solution: I set a lower learning rate and the problem is fixed. It seems that when we do transfer learning, we cannot set a high learning rate because the model is not well connected to the softmax layer you add.(Just some intuition) In addition, it's also possible that you forget to call model.eval() to invalidate the Dropout layer or something. But this is not the case for me.
> > 
> > 
> > Do you also use TFBertForSequenceClassification for multi-label classification?Multi-label classification requires sigmoid function.
> 
> Typically, **sigmoid** function is used in binary classification problems, instead **softmax** function is used in multi-class classification problems
I use the sigmoid function, but the output of different words is the same


@thomwolf Is this problem solved

I have facing the same issue

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Try adjusting learning rate, the dropout probability(`config.hidden_dropout_prob`) and batch size.

@venkatasg Can you share the hyperparameters that worked for your experiments?  Trying to get a sense of order-of-magnitude for each of those.

The problem is probably that the model is overfitting to the data. For BERT, the following hyperparameters worked for me:

- batch size: 16/32
- learning rate: 1e-5, 2e-5
- dropouts: 0.1
- weight decay: 0

However, what works for me might not work for you. Keep experimenting with the hyperparameters(and random seeds)

I just finished a multi label classifier training and got the exact same result:

> same output no matter the input that I put in


I've been facing a similar issue as described here - I got a good accuracy but the predictions just would not make any sense.

Luckily, I've received help from the [HuggingFace.co community](https://discuss.huggingface.co/t/fine-tune-for-multiclass-or-multilabel-multiclass/4035) and it turned out that one has to initialize the model with the correct labels because the model is otherwise learning something but its just not clear what numeric label represents what string...

`bert = TFAutoModel.from_pretrained(tranformersPreTrainedModelName, label2id=label2Index, id2label=index2label)
`

The full code for the solution that works for me with public data can [be found here](https://github.com/Dirkster99/PyNotes/blob/master/Transformers/LocalModelUsage_Finetuning/66_Transformer_4_Language_Classification_MultiClass.ipynb).

Hope this helps...