Repository: huggingface/transformers
Pull request: Fixes the failing test `test_is_split_into_words` in `test_pipelines_token_classification.py`

# What does this PR do?

This test was originally added in https://github.com/huggingface/transformers/pull/38818 .  
However, running the test currently results in a failure:

```bash
RUN_SLOW=1 python -m pytest -k "test_is_split_into_words" tests/pipelines/test_pipelines_token_classification.py
````

First assertion failure:

```bash
E AssertionError: Lists differ: [[{'entity_group': 'PER', 'score': 0.921, [121 chars]29}]] != [{'entity_group': 'PER', 'score': ANY(floa[129 chars] 29}]
E
E First differing element 0:
E [{'entity_group': 'PER', 'score': 0.921, [120 chars] 29}]
E {'entity_group': 'PER', 'score': ANY(floa[38 chars]: 11}
E
E Second list contains 1 additional elements.
E First extra element 1:
E {'entity_group': 'LOC', 'score': ANY(float), 'word': 'New York', 'start': 21, 'end': 29}
E
E - [[{'end': 11,
E ? -
E
E + [{'end': 11,
E - 'entity_group': 'PER',
E ? -
E
E + 'entity_group': 'PER',
E - 'score': 0.921,
E + 'score': ANY(float),
E - 'start': 6,
E ? -
E
E + 'start': 6,
E - 'word': 'Sarah'},
E ? -
E
E + 'word': 'Sarah'},
E - {'end': 29,
E ? -
E
E + {'end': 29,
E - 'entity_group': 'LOC',
E ? -
E
E + 'entity_group': 'LOC',
E - 'score': 0.999,
E + 'score': ANY(float),
E - 'start': 21,
E ? -
E
E + 'start': 21,
E - 'word': 'New York'}]]
E ? - -
E
E + 'word': 'New York'}]

tests/pipelines/test_pipelines_token_classification.py:331: AssertionError
```

Second assertion failure:

```bash
E       AssertionError: Lists differ: [[{'e[25 chars]re': 0.921, 'word': 'Sarah', 'start': 6, 'end'[255 chars]40}]] != [[{'e[25 chars]re': ANY(float), 'word': 'Sarah', 'start': 6, [277 chars]42}]]
E       
E       First differing element 1:
E       [{'en[24 chars]re': 0.999, 'word': 'Wolfgang', 'start': 11, '[86 chars] 40}]
E       [{'en[24 chars]re': ANY(float), 'word': 'Wolfgang', 'start': [98 chars] 42}]
E       
E       Diff is 738 characters long. Set self.maxDiff to None to see it.

tests/pipelines/test_pipelines_token_classification.py:344: AssertionError
```

After applying the fix, the same command passes successfully:

```bash
======================================================================================= test session starts ========================================================================================
platform linux -- Python 3.10.11, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/shutotakahashi/projects/transformers
configfile: pyproject.toml
plugins: asyncio-1.0.0, anyio-4.9.0, xdist-3.7.0, timeout-2.4.0, rerunfailures-15.1, order-1.3.0, rich-0.2.0, hypothesis-6.135.14
asyncio: mode=strict, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 21 items / 20 deselected / 1 selected                                                                                                                                                    

tests/pipelines/test_pipelines_token_classification.py::TokenClassificationPipelineTests::test_is_split_into_words PASSED                      
```

No changes were made to the core implementation — only the test code was modified to better reflect the intended behavior.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 


Changed files:
filename: tests/pipelines/test_pipelines_token_classification.py
patch: @@ -328,8 +328,10 @@ def test_is_split_into_words(self):
         self.assertEqual(
             nested_simplify(output),
             [
-                {"entity_group": "PER", "score": ANY(float), "word": "Sarah", "start": 6, "end": 11},
-                {"entity_group": "LOC", "score": ANY(float), "word": "New York", "start": 21, "end": 29},
+                [
+                    {"entity_group": "PER", "score": ANY(float), "word": "Sarah", "start": 6, "end": 11},
+                    {"entity_group": "LOC", "score": ANY(float), "word": "New York", "start": 21, "end": 29},
+                ]
             ],
         )
 
@@ -349,8 +351,8 @@ def test_is_split_into_words(self):
                     {"entity_group": "LOC", "score": ANY(float), "word": "New York", "start": 21, "end": 29},
                 ],
                 [
-                    {"entity_group": "PER", "score": ANY(float), "word": "Wolfgang", "start": 12, "end": 20},
-                    {"entity_group": "LOC", "score": ANY(float), "word": "Berlin", "start": 36, "end": 42},
+                    {"entity_group": "PER", "score": ANY(float), "word": "Wolfgang", "start": 11, "end": 19},
+                    {"entity_group": "LOC", "score": ANY(float), "word": "Berlin", "start": 34, "end": 40},
                 ],
             ],
         )