Incorrect assertion in pipeline test test_dbmdz_english()
Author: davidbenton
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: macOS-12.4-x86_64-i386-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.2 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: n
- Using distributed or parallel set-up in script?: n

### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`RUN_SLOW=1 RUN_PIPELINE_TESTS=yes pytest tests/pipelines/test_pipelines_token_classification.py::TokenClassificationPipelineTests::test_dbmdz_english`

Fails with two notable diffs: the "UN" entity offsets in the assertion don't match the offsets in the input string itself (off by two characters), and the `index` doesn't match. Output:

```
======================================= FAILURES ========================================
__________________ TokenClassificationPipelineTests.test_dbmdz_english __________________

self = <tests.pipelines.test_pipelines_token_classification.TokenClassificationPipelineTests testMethod=test_dbmdz_english>

    @require_torch
    @slow
    def test_dbmdz_english(self):
        # Other sentence
        NER_MODEL = "dbmdz/bert-large-cased-finetuned-conll03-english"
        model = AutoModelForTokenClassification.from_pretrained(NER_MODEL)
        tokenizer = AutoTokenizer.from_pretrained(NER_MODEL, use_fast=True)
        sentence = """Enzo works at the UN"""
        token_classifier = pipeline("ner", model=model, tokenizer=tokenizer)
        output = token_classifier(sentence)
>       self.assertEqual(
            nested_simplify(output),
            [
                {"entity": "I-PER", "score": 0.997, "word": "En", "start": 0, "end": 2, "index": 1},
                {"entity": "I-PER", "score": 0.996, "word": "##zo", "start": 2, "end": 4, "index": 2},
                {"entity": "I-ORG", "score": 0.999, "word": "UN", "start": 22, "end": 24, "index": 7},
            ],
        )
E       AssertionError: Lists differ: [{'en[24 chars] 0.998, 'index': 1, 'word': 'En', 'start': 0, [179 chars] 20}] != [{'en[24 chars] 0.997, 'word': 'En', 'start': 0, 'end': 2, 'i[179 chars]: 7}]
E       
E       First differing element 0:
E       {'ent[15 chars]'score': 0.998, 'index': 1, 'word': 'En', 'start': 0, 'end': 2}
E       {'ent[15 chars]'score': 0.997, 'word': 'En', 'start': 0, 'end': 2, 'index': 1}
E       
E         [{'end': 2,
E           'entity': 'I-PER',
E           'index': 1,
E       -   'score': 0.998,
E       ?                ^
E       
E       +   'score': 0.997,
E       ?                ^
E       
E           'start': 0,
E           'word': 'En'},
E          {'end': 4,
E           'entity': 'I-PER',
E           'index': 2,
E       -   'score': 0.997,
E       ?                ^
E       
E       +   'score': 0.996,
E       ?                ^
E       
E           'start': 2,
E           'word': '##zo'},
E       -  {'end': 20,
E       ?           ^
E       
E       +  {'end': 24,
E       ?           ^
E       
E           'entity': 'I-ORG',
E       -   'index': 6,
E       ?            ^
E       
E       +   'index': 7,
E       ?            ^
E       
E           'score': 0.999,
E       -   'start': 18,
E       ?            ^^
E       
E       +   'start': 22,
E       ?            ^^
E       
E           'word': 'UN'}]

tests/pipelines/test_pipelines_token_classification.py:284: AssertionError
```

### Expected behavior

[a green dot]

Comments:
@davidbenton 
~~Thanks for the info, I was able to reproduce on torch `1.11.0` but this is fixed on `1.12.0`.~~

~~We're aware of some modifications within the `nn.Linear` between those two versions. The errors are relatively minor actually, but since this is a random model (meaning not trained on real data) it's much more sensitive to those tiny fluctuations. 
That's why the test fails on `1.11.0` while it works on `1.12.0`.~~


EDIT: 
I though I had tested the test and it worked on `1.12.0` but if my explanation was correct, then a diff should have had occurred on the test itself.
I found out that this commit touch the file: 95113d136508dfef192a29d23344e941735d1a1d

This commit actually changed the string, and makes this slow test fail.
I am guessing this is an automated change. Looked at the diff, it seems only this slow test was affected.
We can either rollback the string, or update the values.

So the `1.11.0` vs `1.12.0` seems like it doesn't explain the difference here.

@ydshieh Maybe can you confirm ? 

~~Hmm, it fails for me on 1.12.0 also (CPU, env otherwise same as above). How could `end: 24` be correct for a string with length 20? Those offsets should be in `sentence` indices, right?~~

 I see you're on the track now.

Pinging @sgugger too here.

Thanks for reporting @davidbenton !

Since that commit, we do have this failure in Slack CI report. From the changes, I think it try to fix all `the the`. So reverting the change on the expected value is good to me.

Thank you, @davidbenton 

Yes, it slipped through the crack as the contributor was trying to fix typos and I didn't pay attention this one was intentional.