Should weight distribution change more when fine-tuning transformers-based classifier?
Author: marrrcin
Pull request: no
Labels: name: Discussion

name: wontfix

## ❓Should weight distribution change more when fine-tuning transformers-based classifier?
This question was posted on DataScience stack exchange:
[https://datascience.stackexchange.com/questions/68641/should-weight-distribution-change-more-when-fine-tuning-transformers-based-class](https://datascience.stackexchange.com/questions/68641/should-weight-distribution-change-more-when-fine-tuning-transformers-based-class)

## Details
I'm using pre-trained DistilBERT model with custom classification head, which is almost the same as in the [reference implementation ](https://github.com/huggingface/transformers/blob/fb560dcb075497f61880010245192e7e1fdbeca4/src/transformers/modeling_distilbert.py#L579)

```python
class PretrainedTransformer(nn.Module):
    def __init__(
        self, target_classes):
        super().__init__()
        base_model_output_shape=768
        self.base_model = DistilBertModel.from_pretrained("distilbert-base-uncased")
        self.classifier = nn.Sequential(
            nn.Linear(base_model_output_shape, out_features=base_model_output_shape),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(base_model_output_shape, out_features=target_classes),
        )

        for layer in self.classifier:
            if isinstance(layer, nn.Linear):
                layer.weight.data.normal_(mean=0.0, std=0.02)
                if layer.bias is not None:
                    layer.bias.data.zero_()

    def forward(self, input_, y=None):
        X, length, attention_mask = input_
        base_output = self.base_model(X, attention_mask=attention_mask)[0]
        base_model_last_layer = base_output[:, 0]
        cls = self.classifier(base_model_last_layer)
        return cls
```

During training, I use linear LR warmup schedule with max LR=5-e5 and cross entropy loss. In general, the model is able to learn on my dataset and reach high precision/recall metrics.

**My question is:**

Should weights distributions and biases in classification layers change more during training? It seems like the weights almost do not change at all, even when I do not initialize them as in the code (to mean=0.0 and std=0.02). Is this an indication that something is wrong with my model or it's just because the layers I've added are redundant and model does not learn nothing new?

Take look at the image of weight from the tensorboard:
<img width="1021" alt="Screenshot 2020-02-24 at 20 56 36" src="https://user-images.githubusercontent.com/6958772/75526050-4bbf6600-5a11-11ea-8d62-37407f968e06.png">

Comments:
I am curious: what kind of behaviour do you see when you freeze the whole base model, and only train the classifier? Also, you may want avoid the use of the `cls` variable name because it is a reserved keyword for the class. In general, the classifier is trained quite quickly (often in two or epochs or less), so you are right in saying that in relative terms the weights of the layers that you add matter very little compared to the rest of the model.

When I freeze the base model, the overall learning pace is drastically slower - 5 epochs is only enough to reach fraction of the quality when base model not frozen.

Histograms when base model is frozen:
<img width="929" alt="Screenshot 2020-03-02 at 13 56 24" src="https://user-images.githubusercontent.com/6958772/75678405-dc09df00-5c8d-11ea-995e-853fbaa15b71.png">


@BramVanroy do you have any further insights on this?

Something else that might cause this is that your layers are stuck in some local optimum, or that they are nullified by ReLU. What happens if you use, e.g., gelu instead of relu? But that can't explain everything (because the first linear layer also barely changes its weights. So I'm not sure. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.