Electra for question answering
Author: Santosh-Gupta
Pull request: no

# 🚀 Feature request

Electra for question answering

## Motivation

Electra is the highest rated single model (non essemble) on the Squad leaderboard

## Your contribution

I am not sure if I have the skills, but I'm willing to take a crack at it! Looking the other QA architectures, it seems that I'll need to put a single linear layer (two outputs) on top of the Electra discriminator ?


Comments:
You can basically copy+paste the code from BertForQuestionAnswering and just change it for ELECTRA. However, the original ELECTRA implementation to fine-tune on squad looks a bit different (it's more like in XLNet). 

If you want to reproduce the official implementation it's probably best you take a look at the published code: https://github.com/google-research/electra/blob/master/finetune/qa/qa_tasks.py#L419

Any updates about this? I managed the creation of ElectraForQuestionAnswering on my own and the code works. If the progress of this thread is in stand-by I can proceed submitting my pull request

@volker42maru  I implemented the same model as described in the official Electra Repository by google. I am still unable to reproduce the original paper results which is 75% EM on the squad v1 and 82% F1. The maximum I could get was 70% EM and 78% F1. @mmuffo94  Please let me know if you have successfully reproduced the results on the squad v1 dataset. It would be great help. I am using the Electra Small Model for now.

@ankur19030 you are refering to ELECTRA-Small I assume? 

I actually finetuned and evaluated only on squad 2.0, but the score on squad 1.1 even with the Small model should be significantly higher. The different QA Head that is used for ELECTRA might not play such a big role in squad 1.1, because it's mostly used to get better predictions on answerability. You can try using a simple QA head first to check the performance on squad 1.1 with ELECTRA, e.g.:
```
class ElectraForQuestionAnswering(ElectraPreTrainedModel):
    def __init__(self, config):
        super(ElectraForQuestionAnswering, self).__init__(config)
        self.num_labels = config.num_labels

        self.electra = ElectraModel(config)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        start_positions=None,
        end_positions=None,
    ):
        outputs = self.electra(
            input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        outputs = (start_logits, end_logits,) + outputs[2:]
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.size()) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.size()) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.size(1)
            start_positions.clamp_(0, ignored_index)
            end_positions.clamp_(0, ignored_index)

            loss_fct = torch.nn.CrossEntropyLoss(ignore_index=ignored_index)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2
            outputs = (total_loss,) + outputs

        return outputs  # (loss), start_logits, end_logits, (hidden_states), (attentions)
```

However, if you try to use ELECTRA-Base or Large you also want to use `layerwise_lr_decay`, as used in the official implementation (https://github.com/google-research/electra/blob/master/model/optimization.py#L48). For me that made quite a big difference in score.

BTW, be sure to use `google/electra-small-discriminator` weights, not the generator.

@volker42maru I initially tried with the simple QA head only, the same as you described but could not reproduce the results, not even near them even though I am using the same hyper-parameters as the ones used in the official implementation except layer wise LR decay. My EM is <70 on squad v1 which is around 5% less than the results from official repo. But I can say for sure that simply Layerwise LR decay in small model should not cause this much difference, as I have already tried removing Layerwise LR decay from official implementation and it still give the same results. So I have no idea where is the gap ?

Are you sure that's for squad v1 and not v2?

I just trained ELECTRA Small for 1 epoch using the simple QA head from above and I get the following score on squad v1 dev: `'exact': 73.46263008514664, 'f1': 82.46777637449017`

I used mostly default parameters and no `layerwise_lr_decay`:
```
--num_train_epochs 1
--per_gpu_train_batch_size 24
--learning_rate 5e-5
```

@volker42maru I will cross check again then, Thanks 

It's been added https://github.com/huggingface/transformers/pull/4913