Allow the creation of "entity groups" for NerPipeline #3548
Author: enzoampil
Pull request: yes

This pull request adds an `index` key to the dictionary returned by `NerPipeline`. The index will be necessary in order to identify **entity groups**, where an entity group is a contiguous series of tokens, having the same **entity type**.

Details of what I want to be able to do can be found in issue #3548.

If this PR gets merged, I would also like to ask if you guys would recommend that I implement the **entity group** transformation in `NerPipeline` itself.

Possibly, I can set the parameter `group` at initialization, where if `True`, the *grouped* version of the output will be returned.


E.g.

Instead of the following *ungrouped* output:

```
[{'entity': 'I-PER', 'score': 0.9983270168304443, 'word': 'En'},
 {'entity': 'I-PER', 'score': 0.9952995777130127, 'word': '##zo'},
 {'entity': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian'},
 {'entity': 'I-ORG', 'score': 0.9967807531356812, 'word': 'National'},
 {'entity': 'I-ORG', 'score': 0.9959043264389038, 'word': 'University'},
 {'entity': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AU'},
 {'entity': 'I-ORG', 'score': 0.9763911366462708, 'word': '##N'}]
```

We get something like the following *grouped* output:

```
[{'entity_group': 'I-PER', 'score': 0.9983270168304443, 'word': 'Enzo'},
 {'entity_group': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian National University'},
 {'entity_group': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AUN'}]
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=h1) Report
> Merging [#3607](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/81484b447b7d8504ff5e1cfff38ec35918383963&el=desc) will **decrease** coverage by `1.04%`.
> The diff coverage is `43.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3607/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3607      +/-   ##
==========================================
- Coverage   78.06%   77.02%   -1.05%     
==========================================
  Files         100      100              
  Lines       17134    17159      +25     
==========================================
- Hits        13375    13216     -159     
- Misses       3759     3943     +184     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `72.76% <43.33%> (-2.19%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0.00%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0.00%> (-10.00%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `95.61% <0.00%> (-2.64%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.48% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.20% <0.00%> (-1.35%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.28% <0.00%> (-0.33%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.10% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=footer). Last update [81484b4...34623f3](https://codecov.io/gh/huggingface/transformers/pull/3607?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thank you @enzoampil!

FYI that I will apply the entity grouping functionality explained above in this same PR

**This pull request now applies the entity group transformation illustrated above by setting the parameter: `group`=True**.

This was done by reflecting the transformation inside `NerPipeline`. I've changed the name of the pull request to better reflect the feature being proposed.

cc @julien-c @mfuntowicz @petulla 

Sample code:
```
# Install branch
# Make sure to restart runtime after installing if using Google Colab
!pip install -e git+git://github.com/enzoampil/transformers.git@add_index_to_ner_pipeline#egg=transformers

# Grouped NER
from transformers import pipeline
nlp = pipeline('ner', group=True)
nlp("Enzo works at the Australian National University (AUN)")

# [{'entity_group': 'I-PER', 'score': 0.9968132972717285, 'word': 'Enzo'},
# {'entity_group': 'I-ORG', 'score': 0.9970400333404541, 'word': 'Australian National University'},
# {'entity_group': 'I-ORG', 'score': 0.9831967651844025, 'word': 'AUN'}]

# Ungrouped NER
nlp = pipeline('ner', group=False)
nlp("Enzo works at the Australian National University (AUN)")

# [{'entity': 'I-PER', 'index': 1, 'score': 0.9983270168304443, 'word': 'En'},
# {'entity': 'I-PER', 'index': 2, 'score': 0.9952995777130127, 'word': '##zo'},
# {'entity': 'I-ORG', 'index': 6, 'score': 0.9984350204467773, 'word': 'Australian'},
# {'entity': 'I-ORG','index': 7, 'score': 0.9967807531356812, 'word': 'National'},
# {'entity': 'I-ORG', 'index': 8  'score': 0.9959043264389038, 'word': 'University'},
# {'entity': 'I-ORG', 'index': 10, 'score': 0.9900023937225342, 'word': 'AU'},
# {'entity': 'I-ORG', 'index': 11, 'score': 0.9763911366462708, 'word': '##N'}]
```

Tutorial on how to do Entity Grouping w/ `NerPipeline` [here](https://colab.research.google.com/drive/1CVLP0n3Q5t5qiWpode7jyhUNZpmLg0mS)

I'm very keen to get feedback for the above, so please let me know if I should change anything, or perform additional steps to bring its quality to an acceptable level.

I accidentally deleted the fork for this, so I've recreated this pull request [here](https://github.com/huggingface/transformers/pull/3957). Apologies for any inconvenience caused by this.

I will close this PR so please refer to the one linked above.