Finetune Luke
Author: binhna
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `adapter-transformers` version: 2.3.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.8.0+cu111 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:

### Who can help
@jplu @NielsRogge 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): LUKE

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Convert the CONLL or custom NER dataset to the json format `{"data": [{"tokens": [], "ner_tags": []}, ...]}`
2. Add `field` params in the 349 line of `examples/research_project/run_luke_ner_no_trainer.py`: `raw_datasets = load_dataset(extension, data_files=data_files, field="data")`
3. Run the script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The script runs with no issues
## Problem
It seems like in the `compute_entity_spans_for_luke` function, the `labels_positions` is a dictionary with keys are tuple of `(start, end)` character position of every possible span in the sentence, and values are the `LABEL` in `str` type (because it is from `examples[text_column_name]`). But when adding values to `labels_entity_spans`, it checks if the tuple `(start, end)` of a span is an entity or not, it will add the corresponded label from `labels_positions` else add `0` (this also may need to be more generalize than just add the 0, because if the index of `OTHER` tag not at 0, that would be wrong tho). This would lead to `labels_entity_spans` has `int` and `str` type mixed together, and then fails on `cast_to_python_objects` from `arrow_writer`.

I found other problems as well but it would be too long to address here and some technical problems related to splitting the sample to multiples one due to the `max_entity_length` compared to the original logic code of @ikuyamada. For example generating 20 tokens sentence length would lead to an `entity_ids` with 210 elements length, do we need to split it to multiple samples of `max_entity_length` like the original code?

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Can you  just run the given command without to proceed to any update in the script:
```
export TASK_NAME=ner

python run_luke_ner_no_trainer.py \
  --model_name_or_path studio-ousia/luke-base \
  --dataset_name conll2003 \
  --task_name $TASK_NAME \
  --max_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/
```

And tell me if you get the cast error, and if not to copy/paste the eval results. And then run the original code with the exact same parameters and tell me if you get similar results.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @jplu
I ran the script on conll2003 dataset with the default hyperparameters. However the performance is not upto the mark. Here are the performance numbers I am getting:
{‘precision’: 0.8983516483516484, ‘recall’: 0.40541976620616366, ‘f1’: 0.5587014888943129, ‘accuracy’: 0.8970917690009956}

I tried with different batch sizes (8, 16, 24, 32) and learning rates (1e-5, 1e-8, 2e-5) but with no luck in reproducing the results mentioned in the paper.
Any suggestions on how to match the performance reported in the paper?