[Request] PerceiverForTokenClassification for NER task
Author: Sanster
Pull request: no

# 🚀 Feature request

I'm reading [Perceiver IO](https://huggingface.co/blog/perceiver) blog, it says PeceiverIO could also do NER task. So I try to implement a `PerceiverForTokenClassification` model.

I use `conll2003` dataset to test my model. Here is my implement (the script is modified from [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py)): [run_perceiver_ner.py](https://github.com/Sanster/transformers-ocr/blob/master/run_perceiver_ner.py)

Train command:
```bash
python3 run_perceiver_ner.py \
--dataset_name conll2003 \
--output_dir /tmp/perceiver_io_ner \
--logging_steps 50 \
--num_train_epochs 50 \
--learning_rate 0.00005 \
--per_device_train_batch_size 32 \
--d_latents 768 \
--d_model 768 \
--do_train \
--do_eval \
--fp16
```

But I have no lucky to make it work, the f1 score is very low:

PereceiverIO train 10 epochs:
![image](https://user-images.githubusercontent.com/3998421/147629282-162d52e2-d24b-4320-9a4d-82e5318354d0.png)

PereceiverIO train 50 epochs:
![image](https://user-images.githubusercontent.com/3998421/147630798-2d907135-4ba0-425b-91da-1eb85c15e344.png)


BERT result for reference:
BERT model train from scratch with 10 epochs
![image](https://user-images.githubusercontent.com/3998421/147623709-73be3adc-8616-43a0-bfef-3cb4bc5d73ac.png)

BERT model with pretrain weights train 10 epochs
![image](https://user-images.githubusercontent.com/3998421/147624296-7b3538e0-a36d-434d-81e6-aeb31e331c4a.png)



Comments:
Hi,

Thanks for your interest in Perceiver and for trying it out on a task that is not implemented yet in the library!

However, for a token classification task such as NER, I would actually just take the same decoder settings as `PerceiverForMaskedLM` (as masked language modeling is also solved as a token classification problem). The only thing that you would need to change is replace the embedding decoder (`PerceiverEmbeddingDecoder`) by a simple linear layer (`nn.Linear`) that takes in the decoder output of shape (batch_size, seq_len, hidden_size) and turns it into a tensor of shape (batch_size, seq_len, num_labels). In other words, I would implement it as follows:

```
from transformers import PerceiverPreTrainedModel, PerceiverTextPreprocessor, PerceiverModel

class PerceiverForTokenClassification(PerceiverPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.num_labels = config.num_labels

        text_preprocessor = PerceiverTextPreprocessor(config)

        trainable_position_encoding_kwargs_decoder = dict(
            num_channels=text_preprocessor.num_channels, index_dims=config.max_position_embeddings
        )

        self.perceiver = PerceiverModel(
            config,
            input_preprocessor=text_preprocessor,
            decoder=PerceiverBasicDecoder(
                config,
                output_num_channels=config.d_latents,
                output_index_dims=config.max_position_embeddings,  # we need to define the seq_len of the inputs beforehand
                num_channels=text_preprocessor.num_channels,
                qk_channels=8 * 32,
                v_channels=text_preprocessor.num_channels,
                num_heads=8,
                use_query_residual=False,
                final_project=False,
                trainable_position_encoding_kwargs=trainable_position_encoding_kwargs_decoder,
            ),
        )
        self.classifier = nn.Linear(text_preprocessor.num_channels, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()
```

Let me know how it goes, if it works well, we can add the model to the library.

Hi,

I did run a [modified version of your script](https://github.com/NielsRogge/transformers/blob/add_perceiver_ner/examples/pytorch/token-classification/run_ner_perceiver.py) with the model defined above. This is what I got:

```
***** eval metrics *****
  epoch                   =        3.0
  eval_LOC_f1             =     0.8895
  eval_LOC_number         =      11884
  eval_LOC_precision      =     0.8934
  eval_LOC_recall         =     0.8856
  eval_MISC_f1            =     0.8058
  eval_MISC_number        =       6078
  eval_MISC_precision     =     0.8182
  eval_MISC_recall        =     0.7938
  eval_ORG_f1             =     0.8215
  eval_ORG_number         =       8869
  eval_ORG_precision      =     0.8187
  eval_ORG_recall         =     0.8243
  eval_PER_f1             =     0.8477
  eval_PER_number         =      10479
  eval_PER_precision      =     0.8598
  eval_PER_recall         =     0.8359
  eval_loss               =     0.1479
  eval_overall_accuracy   =     0.9594
  eval_overall_f1         =      0.848
  eval_overall_precision  =     0.8539
  eval_overall_recall     =     0.8421
  eval_runtime            = 0:01:25.54
  eval_samples            =       3250
  eval_samples_per_second =      37.99
  eval_steps_per_second   =      4.757
```
This is using the default settings (i.e. 3 epochs, linear learning rate schedule, etc.). These could perhaps still be improved using a validation loss.

Thanks for your script, I have reproduced your results and I will try it on my own dataset.

Share some result on conll2003 (train from scratch) using default settings (i.e. 3 epochs, linear learning rate schedule, lr 5e-5 etc.) : 
- BERT: f1 67.5%
- PerceiverIO: f1 28.4%

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.