TfElectraForSequenceClassification
Author: ypapanik
Pull request: yes

This pull request add functionality for sequence classification with Electra. The only missing bit is that I have put the activation function as "tanh" instead of "gelu" (in the orig. implementation)

Comments:
@LysandreJik this should work (assuming tf >=2.2.0) let me know if I can do anything to help.

Hello !

Thanks a lot for this PR!! Can you rebase on master plz? It will be easier to review :)

@jplu (I think/hope) I just did it. Let me know if everything is ok. Great work by the way, congrats to all of Huggingface team!

Well no, you did a merge not a rebase :smile: can you revert your merge with the following command line:

```
git reset --hard e4741ef
git fetch upstream
git pull --rebase upstream master
git push --force
```

Also be careful because the tests are broken.


:blush: yes sorry, I had done all the above but with a commit before the final push. Now it should (hopefully) be ok. I saw the tests are failing,  but I was able to successfully use electra for some text classification tasks of interest.

Awesome!! Thanks for having re-pushed ^^

Now there are several things to change, the output of the `call` method should be like the PyTorch one, it means `(loss), logits, (hidden_states), (attentions)`. Can you add the following updates:

1) just before the return add:
```python
if labels is not None:
    loss = self.compute_loss(labels, logits)
    outputs = (loss,) + outputs
```

2) add the `labels` parameter to the method.

3) The `TFElectraForSequenceClassification` should inherit from the `TFSequenceClassificationLoss` class.

Sorry my bad, the signature of the `call` method should look like this:
```python
def call(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        training=False,
    ):
```

I also changed the call to self.electra

Here are the errors raised by the tests:

```
=================================== FAILURES ===================================
__________________ TFElectraModelTest.test_attention_outputs ___________________
[gw0] linux -- Python 3.7.7 /usr/local/bin/python

self = <tests.test_modeling_tf_electra.TFElectraModelTest testMethod=test_attention_outputs>

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        decoder_seq_length = (
            self.model_tester.decoder_seq_length
            if hasattr(self.model_tester, "decoder_seq_length")
            else self.model_tester.seq_length
        )
        encoder_seq_length = (
            self.model_tester.encoder_seq_length
            if hasattr(self.model_tester, "encoder_seq_length")
            else self.model_tester.seq_length
        )
        decoder_key_length = (
            self.model_tester.key_length if hasattr(self.model_tester, "key_length") else decoder_seq_length
        )
        encoder_key_length = (
            self.model_tester.key_length if hasattr(self.model_tester, "key_length") else encoder_seq_length
        )
    
        for model_class in self.all_model_classes:
            config.output_attentions = True
            config.output_hidden_states = False
            model = model_class(config)
            outputs = model(inputs_dict)
            attentions = [t.numpy() for t in outputs[-1]]
            self.assertEqual(model.config.output_attentions, True)
            self.assertEqual(model.config.output_hidden_states, False)
>           self.assertEqual(len(attentions), self.model_tester.num_hidden_layers)
E           AssertionError: 13 != 5

tests/test_modeling_tf_common.py:324: AssertionError
_________________ TFElectraModelTest.test_hidden_states_output _________________
[gw0] linux -- Python 3.7.7 /usr/local/bin/python

self = <tests.test_modeling_tf_electra.TFElectraModelTest testMethod=test_hidden_states_output>

    def test_hidden_states_output(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            config.output_hidden_states = True
            config.output_attentions = False
            model = model_class(config)
            outputs = model(inputs_dict)
            hidden_states = [t.numpy() for t in outputs[-1]]
            self.assertEqual(model.config.output_attentions, False)
            self.assertEqual(model.config.output_hidden_states, True)
>           self.assertEqual(len(hidden_states), self.model_tester.num_hidden_layers + 1)
E           AssertionError: 13 != 6

tests/test_modeling_tf_common.py:369: AssertionError
______________________ TFElectraModelTest.test_save_load _______________________
[gw0] linux -- Python 3.7.7 /usr/local/bin/python

self = <tests.test_modeling_tf_electra.TFElectraModelTest testMethod=test_save_load>

    def test_save_load(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            model = model_class(config)
            outputs = model(inputs_dict)
    
            with tempfile.TemporaryDirectory() as tmpdirname:
                model.save_pretrained(tmpdirname)
                model = model_class.from_pretrained(tmpdirname)
                after_outputs = model(inputs_dict)
    
>               self.assert_outputs_same(after_outputs, outputs)

tests/test_modeling_tf_common.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_modeling_tf_common.py:154: in assert_outputs_same
    self.assertLessEqual(max_diff, 1e-5)
E   AssertionError: 0.24899402 not less than or equal to 1e-05
```

Hello! Any news on this PR? :)

> Hello! Any news on this PR? :)

Hi, sorry I am a bit pressed at the moment, I'd be glad if you would want to take over, otherwise it might take some time for me to re-grab this.

Ok, no problem I will try to retake what you have done. Thanks a lot for the update.

Any news on this PR @ypapanik @jplu ? Can I help in some way ?

Sorry, no time on my side to work on this for now.

@maxibor The code should be working I have used it successfully to get a better acc than BERT, but some tests had failed. Meanwhile the main codebase has evolved and there should be some conflicts, easy to resolve probably. 
I don't have time to complete those two things now (tests failing, resolve conflicts), perhaps someone from HF should spare a few minutes? @LysandreJik 

Updated version right here https://github.com/huggingface/transformers/pull/6227

Thanks @jplu for opening the new PR, closing this.