Modifying the distill bert architecture
Author: IamSparky
Pull request: no

Currently getting the following error while running the distillbert model :
 **TypeError: forward() got an unexpected keyword argument 'token_type_ids'**

I constructed the model as follows:

```
class Distill_BERTBaseUncased(nn.Module):
    def __init__(self):
        super(Distill_BERTBaseUncased, self).__init__()
        self.bert = transformers.DistilBertModel.from_pretrained(DISTILL_BERT_PATH, return_dict=False)
        self.bert_drop = nn.Dropout(0.5)
        self.out = nn.Linear(768 * 2, 1)

    def forward(self, ids, mask, token_type_ids):
        o1, _ = self.bert(ids, attention_mask=mask, token_type_ids = token_type_ids)
        
        mean_pooling = torch.mean(o1, 1)
        max_pooling, _ = torch.max(o1, 1) 
        cat = torch.cat((mean_pooling, max_pooling), 1)
        
        bo = self.bert_drop(cat)
        output = self.out(bo)
        return output
    
model = Distill_BERTBaseUncased()
model = model.to(device)
```

Please help me in resolving this as I took 3 parameters as input for the model which is present in the forward function in the model class !

Comments:
If you look at the [documentation of `DistilBertModel`](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertmodel), you can see it doesn't use token_type_ids.

> If you look at the [documentation of `DistilBertModel`](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertmodel), you can see it doesn't use token_type_ids.

I removed the token_type_ids but still getting this error !!!!

```
class DISTILLBERTBaseUncased(nn.Module):
    def __init__(self):
        super(DISTILLBERTBaseUncased, self).__init__()
        self.bert = transformers.DistilBertModel.from_pretrained(DISTILL_BERT_PATH, return_dict=False)
        self.bert_drop = nn.Dropout(0.5)
        self.out = nn.Linear(768, 1)

    def forward(self, ids, mask):
        _, x = self.bert(ids, attention_mask=mask)
        
        bo = self.bert_drop(x)
        output = self.out(bo)
        return output
    
model = DISTILLBERTBaseUncased()
model = model.to(device)
```

![image](https://user-images.githubusercontent.com/42636586/119998516-6571a380-bfee-11eb-92ca-435e15146aa4.png)


Yes it gives you an error since the model does only return a single thing, namely a tuple containing a single element (as you set `return_dict=False`). This is also explained in the documentation by the way. You should replace

`_, x = self.bert(input_ids=ids, attention_mask=mask) `

by 

`outputs = self.bert(input_ids=ids, attention_mask=mask)`

You can then access the last hidden states using `outputs[0]`. 

Still facing the error here
![image](https://user-images.githubusercontent.com/42636586/120001929-eda57800-bff1-11eb-87a9-224ff0a7e83c.png)
![image](https://user-images.githubusercontent.com/42636586/120002001-fe55ee00-bff1-11eb-8ca1-618582d2e649.png)


Yes, `outputs` is a tuple. You should use `outputs[0]`, which is a PyTorch tensor.

Sorry for interrupting here, but Github issues are not the place to ask such questions. They are related to understanding PyTorch, working with Transformers. For such questions, you can use the [forum](https://discuss.huggingface.co/).

We would like to keep Github issues for bugs/feature requests.

Thanks!

Thanks , I really appreciate you and the time you put time here in the thread for helping me out. Finally everything seems like working.

Coolbeas