Question about Architecture of BERT for QA
Author: ghk829
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I have a question about the architecture of Bert for QA.

In Bert forward function

``` python
class BertForQuestionAnswering(BertPreTrainedModel):
    def __init__(self, config):
        super(BertForQuestionAnswering, self).__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

    @add_start_docstrings_to_callable(BERT_INPUTS_DOCSTRING)
    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        start_positions=None,
        end_positions=None,
    ):

        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output) # The line I don't understand
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        outputs = (start_logits, end_logits,) + outputs[2:]
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.size()) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.size()) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.size(1)
            start_positions.clamp_(0, ignored_index)
            end_positions.clamp_(0, ignored_index)

            loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2
            outputs = (total_loss,) + outputs

        return outputs  # (loss), start_logits, end_logits, (hidden_states), (attentions)


```

I think logits are from linear layer (this is from bert output)

And start_loss and end_loss is calculated by the logits ( just splited by 2)

But, I read BERT article, But It describes

![image](https://user-images.githubusercontent.com/18682286/73066556-abde5c00-3ee9-11ea-90d2-0ff6ed976048.png)

It looks like the model have to use only spans of the paragraph in last layer.

But, I can't get it how the model can know where's start/end span is?

So can you explain it?

It will be really helpful to me if you answer it.

Comments:
Please don't post screenshots. Use code tags instead and preferably post reproducible code.

https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have been getting my head around BERT Pre-training and task specific fine-tuning, which includes a small additional layer to the default BERT Architecture:

https://arxiv.org/pdf/1810.04805.pdf, 4.2 SQuAD v1.1
"As shown in Figure 1, in the question answering task, we represent the input question and passage as a single packed sequence, with the question using the A embedding and the passage using the B embedding. We only introduce a start vector S ∈ RH and an end vector E ∈ RH during fine-tuning."

which in Huggingface code is:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L1835

```
BertForQuestionAnswering(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      ...
    )
    (encoder): BertEncoder(
      ...
    )
  )
  (qa_outputs): Linear(in_features=768, out_features=2, bias=True)
)
```