AttributeError: 'SequenceClassifierOutput' object has no attribute 'pooler_output'
Author: pratikchhapolika
Pull request: no

I am training the model `Distilbert` model for `multi-label` sequence classification.

```
from transformers import DistilBertTokenizer, DistilBertModel
import torch

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased',return_dict=True)
n_classes=100
criterion = nn.BCELoss()
Batch_size=16

class SRTagger(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.save_hyperparameters()
    self.bert = DistilBertModel.from_pretrained('distilbert-base-uncased',return_dict=True)
    self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)
    self.n_training_steps = n_training_steps
    self.n_warmup_steps = n_warmup_steps
    self.criterion = nn.BCELoss()

  def forward(self, input_ids, attention_mask, labels=None):
    output = self.bert(input_ids, attention_mask=attention_mask)
    output = self.classifier(output.pooler_output)
    output = torch.sigmoid(output)    
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output
```

It throws me the error: 

**AttributeError: 'SequenceClassifierOutput' object has no attribute 'pooler_output'**

Can I make changes to above code to overcome this?

Comments:
As stated in the [docs](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertmodel) of DistilBERT, DistilBertModel doesn't include a `pooler_output` in its outputs. It only includes:
* `last_hidden_state`
* `hidden_states`
* `attentions`. 

By default, only `last_hidden_state` is returned. 

> As stated in the [docs](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertmodel) of DistilBERT, DistilBertModel doesn't include a `pooler_output` in its outputs. It only includes:
> 
> * `last_hidden_state`
> * `hidden_states`
> * `attentions`.
> 
> By default, only `last_hidden_state` is returned.

What changes can we make to above code? Like including some layers to tackle this?

Actually, the easiest way to fine-tune DistilBERT for multi-label classification is my initializing a `DistilBertForSequenceClassification` model, setting `problem_type` to be "multi_label_classification":

```
from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification")
```

The problem_type argument is something that was added recently, the supported models are stated in the [docs](https://huggingface.co/transformers/master/main_classes/configuration.html?highlight=problem_type).
In that way, it will automatically use the appropriate loss function for multi-label classification, which is the BCEWithLogitsLoss as can be seen [here](https://github.com/huggingface/transformers/blob/d70919e6d5629f3f4d19aefff3eec1219055f003/src/transformers/models/distilbert/modeling_distilbert.py#L764). 

In that way, you can easily provide your `labels` - which should be of shape (batch_size, num_labels).

> Actually, the easiest way to fine-tune DistilBERT for multi-label classification is my initializing a `DistilbertForSequenceClassification` model, setting `problem_type` to be "multi_label_classification":
> 
> ```
> from transformers import DistilBertForSequenceClassification
> 
> model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification")
> ```
> 
> The problem_type argument is something that was added recently, the supported models are stated in the [docs](https://huggingface.co/transformers/master/main_classes/configuration.html?highlight=problem_type). In that way, it will automatically use the appropriate loss function for multi-label classification, which is the BCEWithLogitsLoss as can be seen [here](https://github.com/huggingface/transformers/blob/d70919e6d5629f3f4d19aefff3eec1219055f003/src/transformers/models/distilbert/modeling_distilbert.py#L764).
> 
> In that way, you can easily provide your `labels` - which should be of shape (batch_size, num_labels).

Then can we apply this on top of it?

**self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes=100)**

How can we pass number of classes/labels here?

> Then can we apply this on top of it?
> self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes=100)

No, that's actually what `DistilBertForSequenceClassification` does for you: it adds a linear layer on top of the base DistilBERT model. If you call the `.from_pretrained()` method on it, it will initialize the base model with the pretrained weights from the checkpoint on the [hub](https://huggingface.co/distilbert-base-uncased), but the linear layer on top (also called the classifier head) will have randomly initialized weights. This is also returned as a warning when you run it:

```
from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification")
```
returns the following warning:
```
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.bias', 'classifier.weight', 'pre_classifier.weight', 'classifier.bias']
```
These are the parameters of the classification head on top, which need to be fine-tuned (together with the base model) on your custom dataset.

> > Then can we apply this on top of it?
> > self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes=100)
> 
> No, that's actually what `DistilBertForSequenceClassification` does for you: it adds a linear layer on top of the base DistilBERT model. If you call the `.from_pretrained()` method on it, it will initialize the base model with the pretrained weights from the checkpoint on the [hub](https://huggingface.co/distilbert-base-uncased), but the linear layer on top (also called the classifier head) will have randomly initialized weights. This is also returned as a warning when you run it:
> 
> ```
> from transformers import DistilBertForSequenceClassification
> 
> model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification")
> ```
> 
> returns the following warning:
> 
> ```
> Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.bias', 'classifier.weight', 'pre_classifier.weight', 'classifier.bias']
> ```
> 
> These are the parameters of the classification head on top, which need to be fine-tuned (together with the base model) on your custom dataset.

**Got it. But where do I pass the number of classes?**

You can pass it when initializing the model (sorry should have mentioned that), like so:

```
from transformers import DistilBertForSequenceClassification

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification",
num_labels=10)
```


Thanks. Closing!!