[wip/token clasification] Introduce datasets and metrics in token classification examples
Author: vblagoje
Pull request: yes
Labels: name: wontfix

This PR further simplifies token classification examples by introducing the use of HF nlp datasets and metrics.

In summary, we introduce:

- use of GermEval canonical dataset 
- two additional local datasets: [Conll2003](https://github.com/vblagoje/transformers/blob/b58c6186fd589ea6f4c86e10df5a54aa63516d10/examples/token-classification/conll2003.py) and [UDEnglishEWT](https://github.com/vblagoje/transformers/blob/b58c6186fd589ea6f4c86e10df5a54aa63516d10/examples/token-classification/ud_english_ewt.py) (demonstrate custom nlp dataset use in training)
- nlp metrics, splits, removal of seqeval
- removal of all the clutter related to direct download and preprocessing of raw datasets
- further minor simplifications

I have verified the training of all the PyTorch and PL examples works but I was not able to verify the TensorFlow portion due to a faulty GPU setup on my cloud machine. Perhaps @jplu could help me there? Also @stefan-it could you please have a look as well. Therefore, I leave wip tag on, although the PR should be 99% ready. 


Comments:
> Thanks a lot for this PR!
> 
> The files `conll2003.py` and `ud_english_ewt.py` should be in the `nlp` package and not here. Can you open a PR there to propose these new datasets please.

Ok no problem. That makes more sense. 
> 
> In overrall I really like it! Nevertheless, it brings an important breaking change, because it is not possible anymore to use its own files without making an `nlp` script. Instead of modifying the existing script can you add a new one? For example `run_ner_with_nlp.py` or something like that?

Can you please clarify what do you mean by 'it is not possible anymore to use its own files without making an nlp script.' Do you mean that it would be preferred to use raw datasets just like before (as it is now)?


> Can you please clarify what do you mean by 'it is not possible anymore to use its own files without making an nlp script.' Do you mean that it would be preferred to use raw datasets just like before (as it is now)?

This is exaclty what I mean, and what you have done would be better suited as a new script.

Well, the idea of `datasets` is that you can also use your own data files or python/pandas objects @jplu.
See here: https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files
and here: https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data
So I'm not sure we need several scripts actually.

True, I know, but CoNLL2003 is the most used NER dataset, it is the must have evaluation when developing a new NER approach. Same thing with UDEnglishEWT for POS. They are so popular that I think it would be a nice to have in `datasets`.

@jplu I looked at the changes needed to accommodate both: dataset use and raw dataset download + use (without dataset API). It makes things a bit more complicated, clutters the token classification example, and defeats the purpose of simplifying things for new HF users.

Experienced users know how to download raw datasets and load them into the models. The proposed use of the new datasets package lowers the barrier to entry, makes everything simple and easy to understand for the new users. 

In summary, we can add new datasets to the datasets project, use them in examples, that's given. However, having both the old approach (raw datasets download, script preprocessing, and use) and the new approach (just datasets) defeats the purpose of having clean, easy to understand and simple examples for the new users. On top of it, maintaining and testing these examples will be additional headache for us. But you guys let me know wdyt @sshleifer @thomwolf @stefan-it and we'll proceed forward. 

@vblagoje Sorry, I'm not saying that you have to modify the existing script, but to create a new one. Basically you don't touch the current `run_tf_ner.py` and you create a new `run_tf_ner_with_datasets.py` where you do what you want here.

The advantage here is that you are not introducing breaking changes in the scripts people already uses. Think of those who are already using this script with their own data and don't want to move their dataset to datasets (for some reason). How do they do?

Or a better compromise IMO, as @thomwolf proposed, to add the possiblity to use a raw dataset, something that you can do with `datasets` pretty easily. Because here the problem is that you force people to use a specific dataset, which is quite odd for me, we should be able to let people use the dataset they want (from `datasets` or their own) in a smart way.

@jplu Yes, I am all for compromise solution, people can use any token level dataset they want (canonical, community or local dataset)- easily. Have a look at https://github.com/vblagoje/transformers/blob/refactor_token_examples/examples/token-classification/tasks.py All the users have to do is to supply the reference to dataset, source/target columns, labels and off they go with token level classification. That's all. 

That's the point, they should not be forced to build their own `datasets` script or modify the existing one. We should provide a simple way to use the built-in local dataset (https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) + having the possibility to select a dataset from the `datasets` list. These two things can be done by adding some arguments to the script.

@jplu ok understood, good idea! Let's do that!


@jplu I looked into how users could use the local dataset without providing a dataset script. Although local datasets give us the convenience of loading datasets easily they can't parse input out-of-the-box without, at least some, parsing customization. For example, GermEval has a specific [format](https://github.com/huggingface/transformers/blob/master/examples/token-classification/tasks.py#L18) and so do other token level datasets. They are almost never in json or csv format so that we can parse them with some generic [load_dataset](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) approach. 

What we can do is keep the level of abstraction of InputExample so that a new classification task returns a list of InputExamples just like [here](https://github.com/huggingface/transformers/blob/master/examples/token-classification/tasks.py#L18) That way users don't have to create a dataset script but can read examples from a local file, parse and return them as InputExample list. For existing datasets, we convert them to the InputExample list. If you have other ideas on how this could be done lmk but I don't see how we can achieve this just by adding some arguments to the script. 

> I looked into how users could use the local dataset without providing a dataset script. Although local datasets give us the convenience of loading datasets easily they can't parse input out-of-the-box without, at least some, parsing customization. For example, GermEval has a specific format and so do other token level datasets. They are almost never in json or csv format so that we can parse them with some generic load_dataset approach.

Indeed, CoNLL files have a specific format with the `-docstart-` header for separating each document. The `csv` script cannot handle this.

> What we can do is keep the level of abstraction of InputExample so that a new classification task returns a list of InputExamples just like here That way users don't have to create a dataset script but can read examples from a local file, parse and return them as InputExample list. For existing datasets, we convert them to the InputExample list. If you have other ideas on how this could be done lmk but I don't see how we can achieve this just by adding some arguments to the script.

Might be another solution yes. Let's try this one and see how it goes.


@jplu The commit change is [here](https://github.com/huggingface/transformers/pull/7041/commits/f381a0b17ed3046a96e5adc96c6a71d592c58cdf) So instead of relying on `get_dataset` API contract users can simply implement `get_input_examples`. We implement both approaches: using datasets and if needed loading from raw datasets. I am now checking that all PyTorch examples indeed work. 

There are still several changes to do but it looks much better. Among what I could see:

- the `--labels` much be kept otherwise we always get the same list of labels.
- `conll2003.py` and `ud_english_ewt.py` should be moved. And a `--dataset_name` parameter should be added to the `run_*` scripts.
- `CoNLL2003` is not for chunking, then it should not belong to the chunking task, it is counter intuitive. Use CoNLL2000 here https://github.com/teropa/nlp/tree/master/resources/corpora/conll2000
- the method `read_examples_from_file` should not be removed.

You should also update the README by giving an example for at least these cases:
1. I want to train a NER model over CoNLL2003
2. I have a `train.txt`, `test.txt` and `dev.txt` with my own labels and want to train a NER model over it
3. I want to train a NER model over GERMEVAL2004

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.