Pooled results for DistilBert
Author: renjiege
Pull request: no

When I look at the architecture of Bert and DistilBert, I notice that Bert has a pooler layer while DistilBert does not. As a result, The output shape of the Distilbert is [batch_size, seq_length, hidden_size] while the shape of Bert is to [batch_size, hidden_size].  I wonder why it is the case and if it is possible to add the pooler layer for DistilBert?

My code 
```
bert = AutoModel.from_pretrained('bert-base-uncased')
distilbert = AutoModel.from_pretrained('distilbert-base-uncased')
```

Related question: https://github.com/huggingface/transformers/issues/1328

Comments:
I think BERT's pooler layers is used for next sentence prediction (NSP - one of the objective functions in BERT pretraining), see `BertForNextSentencePrediction` (and it is then used for downstream classification tasks). Since `DistilBERT` doesn't use NSP, there is no pooler layer.

However, you can simply use `DistilBertForSequenceClassification` if you intend to use the pooler layer to perform a sequence classification task.

Thanks, @ydshieh. I followed your recommendation and found it very helpful. `DistilBertForSequenceClassification` works fine, but its last layer, the classifier layer's dim=2. I wonder if there is a flexible way to change the output layer to the default dim = 768? 

You can do something like

```
import torch
from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=768)
logits = model(torch.ones(3, 5, dtype=torch.int32)).logits

# verify the shape
print(logits.size())
```

But `SequenceClassification` models' `num_labels` is for the number of labels in a sequence classification task, and the output is not to be considered as `pooler output`. You are using `768`, and I think you really want to have a `pooler` output. I am not sure what kind of task you are performing, **but if it is not sequence classification, and you need a `pooler` output from `DistilBert` for your specific task**, you can just create a custom model like:


```
class MyDistilBert(DistilBertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels
        self.config = config

        self.distilbert = DistilBertModel(config)
       
        # Use the name `pooler` to make the naming more meaningful for your need
        # self.pre_classifier = nn.Linear(config.dim, config.dim)
        self.pooler = nn.Linear(config.dim, config.dim)
        
        # Removed if you task is not sequence classification
        # self.classifier = nn.Linear(config.dim, config.num_labels)
```

See the implementation of `DistilBertForSequenceClassification` for more details
https://github.com/huggingface/transformers/blob/52d2e6f6e904ef9b75c78716ce77b98196ed837a/src/transformers/models/distilbert/modeling_distilbert.py#L667-L775


`DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=768)` works. Thanks a lot. I am trying to fine tune a DistilBert model for text classification. I added some head layers on top of the pre-trained model, and then train these layers while freezing DistilBert model layers.