OpenAI GPT NoPaddingTokenFastTokenizerMatchingTest test fails
Author: stas00
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

`tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers`
fails

## To reproduce

```
pytest   -n 1 tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers
================================================================ test session starts =================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
gw0 [1]
F                                                                                                                                              [100%]
====================================================================== FAILURES ======================================================================
____________________________________________ NoPaddingTokenFastTokenizerMatchingTest.test_all_tokenizers _____________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <tests.test_tokenization_fast.NoPaddingTokenFastTokenizerMatchingTest testMethod=test_all_tokenizers>

    def test_all_tokenizers(self):
        for tok_case in self.TOKENIZERS_CLASSES:
            for pretrained_name in tok_case.python_cls.pretrained_vocab_files_map[tok_case.vocab_key].keys():
    
                # Tokenizer.filter makes it possible to filter which Tokenizer to case based on all the
                # information available in Tokenizer (name, rust class, python class, vocab key name)
                if tok_case.filter is None or (
                    tok_case.filter is not None and tok_case.filter(tok_case, pretrained_name)
                ):
                    kwargs = dict(t for t in tok_case.kwargs) if tok_case.kwargs else {}
                    with self.subTest("{} ({})".format(tok_case.name, pretrained_name)):
                        tokenizer_r = tok_case.rust_cls.from_pretrained(pretrained_name, **kwargs)
                        tokenizer_p = tok_case.python_cls.from_pretrained(pretrained_name, **kwargs)
    
>                       self.fast_align_python(tokenizer_r, tokenizer_p, tok_case, pretrained_name)

tests/test_tokenization_fast.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_tokenization_fast.py:836: in fast_align_python
    self.assert_tokenization_python_rust_equals(tokenizer_r, tokenizer_p)
tests/test_tokenization_fast.py:192: in assert_tokenization_python_rust_equals
    self.assertSequenceEqual(input_p[key], input_r[key])
E   AssertionError: Sequences differ: [616,[111 chars] 0, 40477, 4830, 994, 580, 566, 260, 5958, 260[5290 chars] 239] != [616,[111 chars] 0, 4830, 994, 580, 566, 260, 5958, 260, 1490,[5160 chars] 239]
E
E   First differing element 29:
E   40477
E   4830
E
E   First sequence contains 12 additional elements.
E   First extra element 973:
E   1832
E
E   Diff is 8701 characters long. Set self.maxDiff to None to see it.
----------------------------------------------------------------- Captured log call ------------------------------------------------------------------
WARNING  transformers.tokenization_utils_base:tokenization_utils_base.py:2086 Token indices sequence length is longer than the specified maximum sequence length for this model (985 > 512). Running this sequence through the model will result in indexing errors
================================================================== warnings summary ==================================================================
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(args, collections.Mapping), '{} args must be a dict with argument names as keys.'.format(name)

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import OrderedDict, Sequence, defaultdict

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================== short test summary info ===============================================================
FAILED tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers - AssertionError: Sequences differ: [616,[111 ...
=========================================================== 1 failed, 4 warnings in 5.44s ============================================================
```

## Expected behavior

not fail ;)

I started digging and it fails only for `OpenAI GPT` tokenizer and passes `GPT2` tests.

If someone could give me a pointer, I'd be happy to investigate more. 

I'm not sure whether it's the python or rust implementation that is at fault.

I tested that if I force max_length=512, to remove the warning, the test still fails in the same way.

## Environment info

```
- `transformers` version: 3.0.2
- tokenizers: 0.8.1rc1
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


looks like it has been fixed.