Cannot load studio-ousia/luke-base for AutoModelForTokenClassification
Author: KerenzaDoxolodeo
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0.dev0 (pulled from repo)
- Platform: 3
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.0 (no)
- Tensorflow version (GPU?): 2.4.1 (no)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

I tried loading LUKE's weight for AutoModelForTokenClassification. I intend to train further for NER. It failed due to a configuration error.

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

from transformers import AutoTokenizer, AutoModel
  
tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") #Succesful

model = AutoModel.from_pretrained("studio-ousia/luke-base") #Succesful

model = AutoModelForTokenClassification.from_pretrained("studio-ousia/luke-base", num_labels=39)
```

```
Some weights of the model checkpoint at studio-ousia/luke-base were not used when initializing LukeModel: ['embeddings.position_ids']
- This IS expected if you are initializing LukeModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LukeModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-b4efbc1b7796> in <module>
      7 model = AutoModel.from_pretrained("studio-ousia/luke-base")
      8 
----> 9 model = AutoModelForTokenClassification.from_pretrained("studio-ousia/luke-base", num_labels=39)

/opt/conda/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    381             return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    382         raise ValueError(
--> 383             f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    384             f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    385         )

ValueError: Unrecognized configuration class <class 'transformers.models.luke.configuration_luke.LukeConfig'> for this kind of AutoModel: AutoModelForTokenClassification.
Model type should be one of BigBirdConfig, ConvBertConfig, LayoutLMConfig, DistilBertConfig, CamembertConfig, FlaubertConfig, XLMConfig, XLMRobertaConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, BertConfig, MegatronBertConfig, MobileBertConfig, XLNetConfig, AlbertConfig, ElectraConfig, FunnelConfig, MPNetConfig, DebertaConfig, DebertaV2Config, IBertConfig.
```

## Expected behavior

Succesful loading

Comments:
cc @NielsRogge

LUKE does not have a `*ForTokenClassification` model, so it's not available in the token classification auto model. The following three models are available: `LukeForEntityClassification`, `LukeForEntityPairClassification`, `LukeForEntitySpanClassification`

Yes, I explicitly did not include the head models of LUKE in any `AutoModel`, as LUKE works a bit differently than other models.

For NER, it does not use a token classification head as other models like BERT do. Instead, LUKE considers all possible entity spans in a sentence, which are then classified accordingly. See the [code example](https://huggingface.co/transformers/master/model_doc/luke.html#transformers.LukeForEntitySpanClassification) for reference. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @NielsRogge ,

I am trying to finetune Luke on a custom dataset for the NER task. As mentioned here, I infer we need to do it through LukeForEntitySpanClassification. Can you please guide me on how I could achieve the same? I am unaware of any existing tutorial for the same.

Thank You

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, @Sreyan88 Were you able to train LUKE on a custom dataset? I am also working on the same and do not have any progress on this yet. Any help is appreciated. Thanks!