BertModel Inputs
Author: pn12
Pull request: no

# ❓ Questions & Help

Hi,

I have been using BertModel for Question and Answering . In examples - I see there are no start_position , end_position being provided to the Model .

How is the Model able to train in this case using Input_ids , Mask and Attention Head ?

Might be a naive question , but I have dig into the source codes and referred to runsquad.py - could not gain clarity.

Can anybody suggest?

Thanks


<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Can you please link the example you mean? :-) 

Hi , 

In the example code below; I have highlighted the inputs** being provided to the model i.e. ids , attention_mask and token_type_ids.

If I add start_positions and end_positions as inputs here - I get an error saying unknown inputs : start_positions and end_positions.

So, my question is - how is the model able to train when starts and ends not being provided as inputs to the model training part?

Thanks 

```
class QAModel (transformers.BertPreTrainedModel):
    def __init__(self, conf):
        super(QAModel, self).__init__(conf)
        self.bert = transformers.BertModel.from_pretrained(config.BERT_PATH, config=conf)
        self.drop_out = nn.Dropout(0.1)
        self.l0 = nn.Linear(768 * 2, 2)
        torch.nn.init.normal_(self.l0.weight, std=0.02)
    
    def forward(self, ids, mask, token_type_ids):
        **_, _, out = self.bert(
           ids,
            attention_mask=mask,
            token_type_ids=token_type_ids**
        )

        out = torch.cat((out[-1], out[-2]), dim=-1)
        out = self.drop_out(out)
        logits = self.l0(out)

        start_logits, end_logits = logits.split(1, dim=-1)

        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        return start_logits, end_logits

```


Hey @pn12,

I'm not sure if this answers you question: If you want to fine-tune a `Bert` model on Question Answering you have to prove both the `start_positions` and the `end_positions`, see this line: https://github.com/huggingface/transformers/blob/f9414f7553d3f1872b372990ef03205c0d1141df/src/transformers/modeling_bert.py#L1405

Only for validation and evaluation, you don't have to provide those so that the model can predict them.

Thanks to @patil-suraj, you can also take a look at this notebook to check out how to fine-tune a Bert-Like model on Squad: 
https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb

Check out his `DummyDataCollator` to see that he passes those two arguments for training.