examples/legacy/token-classification　doesn't work well
Author: wasabizusi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.12.0.dev0
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
× the official example scripts: (give details below)
〇 my own modified scripts: (give details below)

run.sh

     1	## The relevant files are currently on a shared Google
     2	## drive at https://drive.google.com/drive/folders/1kC0I2UGl2ltrluI9NqDjaQJGw5iliw_J
     3	## Monitor for changes and eventually migrate to nlp dataset
     4	
     5	export MAX_LENGTH=128
     6	export BERT_MODEL=bert-base-multilingual-cased
     7	python3 preprocess.py data/train_tab_space.txt $BERT_MODEL $MAX_LENGTH > train.txt
     8	python3 preprocess.py data/dev_tab_space.txt $BERT_MODEL $MAX_LENGTH > dev.txt
     9	python3 preprocess.py data/test_tab_space.txt $BERT_MODEL $MAX_LENGTH > test.txt
    10	cat train.txt dev.txt test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > labels.txt
    11	export OUTPUT_DIR=germeval-model
    12	export BATCH_SIZE=32
    13	export NUM_EPOCHS=3
    14	export SAVE_STEPS=750
    15	export SEED=1
    16	
    17	python3 run_ner.py \
    18	--task_type NER \
    19	--data_dir . \
    20	--labels ./labels.txt \
    21	--model_name_or_path $BERT_MODEL \
    22	--output_dir $OUTPUT_DIR \
    23	--max_seq_length  $MAX_LENGTH \
    24	--num_train_epochs $NUM_EPOCHS \
    25	--per_gpu_train_batch_size $BATCH_SIZE \
    26	--save_steps $SAVE_STEPS \
    27	--seed $SEED \
    28	--do_train \
    29	--do_eval \
    30	--do_predict

The tasks I am working on is:
× an official GLUE/SQUaD task: (give the name)
〇 my own task or dataset: (give details below)

I use my data
[dev_tab_space.txt](https://github.com/huggingface/transformers/files/7328515/dev_tab_space.txt)
[train_tab_space.txt](https://github.com/huggingface/transformers/files/7328517/train_tab_space.txt)
[test_tab_space.txt](https://github.com/huggingface/transformers/files/7328516/test_tab_space.txt)

Looking at the eval_results.txt,test_results.txt and test_predictions.txt, it doesn't seem to be predicted correctly.

eval_results.txt

     1	eval_loss = 0.0005731257260777056
     2	eval_accuracy_score = 1.0
     3	eval_precision = 0.0
     4	eval_recall = 0.0
     5	eval_f1 = 0.0
     6	eval_runtime = 1.7257
     7	eval_samples_per_second = 130.379
     8	eval_steps_per_second = 16.804
     9	epoch = 3.0

test_results.txt

     1	test_loss = 0.0005731506389565766
     2	test_accuracy_score = 1.0
     3	test_precision = 0.0
     4	test_recall = 0.0
     5	test_f1 = 0.0
     6	test_runtime = 1.7989
     7	test_samples_per_second = 132.302
     8	test_steps_per_second = 16.677

test_predictions.txt

     1	L.D._Schamphelaere O
     2	,_ O
     3	" O
     4	Short_run_digital_color_printing O
     5	, O
     6	" O
     7	_ O
     8	Japan_Hard_Copy'96論文集 O
     9	， O
    10	_ O
    11	p. O
    12	5 O
    13	,_ O
    14	電子写真学会 O
    15	． O
    16	
    17	弘瀬紀寿 O
    18	， O
    19	“ O
    20	RIPの技術動向とデジタルワークフロー O
    21	，” O
    22	印刷雑誌 O
    23	， O
    24	vol. O
    25	81 O
    26	,_ O
    27	pp. O
    28	41 O
    29	- O
    30	46 O
    31	,_ O
    32	1998 O
    33	. O
    34	
    35	加藤茂夫 O
    36	， O
    37	長谷川まどか O
    38	， O
↑After that, 「token O」 is repeated.
However, 「O」label is not included in my data in the first place.

Does anyone have similar experience?
Please help me!

## To reproduce

Steps to reproduce the behavior:

1.Go to transformers/examples/legacy/token-classification
2.Make 「data」 file. And, put 3 text data(train_tab_space.txt test_tab_space.txt dev_tab_space.txt) in it.
3.Rewrite run.sh.
4.Run run.sh.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
It is recommended to use the most up-to-date example scripts. Legacy scripts are not guaranteed to work. From the legacy readme:

> Using these examples together with a recent version of the library usually requires to make small (sometimes big) adaptations to get the scripts working.

Try the up-to-date scripts instead: https://github.com/huggingface/transformers/tree/master/examples/pytorch/token-classification

Thank you for your advice. There is a reason to use examples/legacy/token-classification.
I want to use cl-tohoku/bert-base-japanese and bert-base-multilingual-cased, because my dataset is japanese dataset. However, those two models could not be used in examples/pytorch/token-classification with the following error.
「ValueError: This example script only works for models that have a fast tokenizer.」

In examples/legacy/token-classification, The example data （GermEval 2014） presented by huggingface worked fine.

Hi @wasabizusi after looking at your training data, it seems that you're using PoS tagging as downstream task? If you, you need to pass all possible tags for the label argument.

Could you paste the output of the `labels.txt` file here :thinking: I can have a look on it later.


Hi @stefan-it .
labels.txt is output as follows.

D
DAND
DC
DCL
DCO
DE
DED
DHY
DLBR
DN
DP
DPP
DRBR
DS
DSL
DSP
DUN
DV
DZC
DZE
DZLBR
DZRBR
DZS
ETC
RA
RAOT
RB
RC
RD
RE
RL
RM
RN
RP
RPP
RT
RTR
RURL
RV
RW
RY
UNKNOWN
sp

Sorry. I forgot to attach the original labels.txt.
[labels.txt](https://github.com/huggingface/transformers/files/7334822/labels.txt)



Ah, I see, could you please try to use `--task_type POS` instead of `NER`. 


Thank you for your advice!
I ran the program, but precision, recall, and f1 were all 1.0. And, test_predictions.txt was generated, but nothing was written.
When I checked the data downloaded by legacy/token-classification/run_pos.sh, I think that the data format may be different.(For some reason, run_pos.sh didn't work properly...)
I will consider processing the data.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.