Why is the pooler output used for sequence classification (if it does not represent the input semantic well)?
Author: mkaze
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details
In the [documentation](https://huggingface.co/transformers/model_doc/bert.html#tfbertmodel) of `TFBertModel`, it is stated that the `pooler_output` is not a good semantic representation of input (emphasis mine):

> `pooler_output (tf.Tensor of shape (batch_size, hidden_size))`:
> Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. **This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.**

However, if we look at the [source code][1] of `TFBertForSequenceClassification`, we see that it's the pooler output which is used for classification of input sequence:

```python
        outputs = self.bert(inputs, **kwargs)

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output, training=kwargs.get("training", False))
        logits = self.classifier(pooled_output)
```
**I was wondering why this is the case, and doesn't this contradict what is stated in the documentation?**

FYI, to try the approach suggested in the documentation (i.e. averaging or pooling hidden state), I also created two custom classifiers where one of them is using only the `CLS` token representation, i.e. `outputs[0][:,0]`, and the other is using the concatenation of `CLS` token representation and the average of last hidden state, i.e. `mean(outputs[0], axis=1)`. However, in the experiments on my dataset, both performed poorly (around 73% validation accuracy, with 99% train accuracy) compared to using the built-in `TFBertForSequenceClassification` (which achieved around 79% validation accuracy, with 99% train accuracy). 

Even I tried the concatenation of `CLS` token representation for all the last four layers (as suggested in BERT paper); however, it also performed poorly (around 70% validation accuracy, with 99% train accuracy).

I am (almost) certain that my implementation is correct; however, I include it here for reference:

```python
config = BertConfig.from_pretrained("dbmdz/bert-base-german-cased",
                                    num_labels=len(le.classes_),
                                    output_hidden_states=True)
tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
bert_model = TFBertModel.from_pretrained("dbmdz/bert-base-german-cased",
                                         config=config,
                                         trainable=True)
seq_max_len = 128

input_ids_layer = tf.keras.layers.Input(shape=(seq_max_len,), dtype='int32')
attention_mask_layer = tf.keras.layers.Input(shape=(seq_max_len,), dtype='int32')
inputs = [input_ids_layer, attention_mask_layer]
outputs = bert_model(inputs)

hidden_states = outputs[2]
last_four_hidden_states = tf.keras.layers.concatenate(list(hidden_states[-4:]))
cls_vector = tf.keras.layers.Lambda(lambda x: x[:,0])(last_four_hidden_states)
sentence_vector = tf.keras.layers.Dropout(0.1)(cls_vector)
out = tf.keras.layers.Dense(len(le.classes_), name="classifier")(sentence_vector)

model = tf.keras.models.Model(inputs, out)

opt = tf.keras.optimizers.Adam(learning_rate=1e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")

model.compile(loss=loss, optimizer=opt, metrics=[metric])
history = model.fit([train_input_ids, train_attention_mask],
                    train_labels,
                    epochs=30,
                    batch_size=16,
                    validation_data=([test_input_ids, test_attention_maks], test_labels))
```

**A link to original question on Stack Overflow**: This question is more related to theoretical aspects of machine learning and therefore is considered as off-topic on SO.

[1]: https://github.com/huggingface/transformers/blob/fa49b9afeab5545f14b3661b35195b829fcf8ef5/src/transformers/modeling_tf_bert.py#L923-L926

Comments:
@mkaze hello, what's your tensorflow version and transformers version?

@etveritas When I was working on this, I was using the latest versions at the time, i.e. TF 2.1 and Transformers 2.8. 

@mkaze oh, thank you~

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> the concatenation of CLS token representation for all the last four layers (as suggested in BERT paper);

Why are you using this 4-layer implementation suggestion from the paper? The paper clearly recommends this logic only for feature-based version of the classifier (which is mentioned only for comparison purposes), and not for the primary fine-tuning version of the model. I don't really get it.