Extending vocabulary by a large size crashes RobertaTokenizerFast
Author: RudrakshTuwani
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using: RoBERTa

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import RobertaTokenizerFast

tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base')
tokenizer.add_tokens([str(i) for i in range(60000)])
```

Here's the stack trace:
```
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: CompiledTooBig(10485760)', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/added_vocabulary.rs:299:13
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
  11: rust_begin_unwind
             at src/libstd/panicking.rs:385
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:89
  13: core::option::expect_none_failed
             at src/libcore/option.rs:1272
  14: tokenizers::tokenizer::added_vocabulary::AddedVocabulary::add_tokens
  15: tokenizers::tokenizer::Tokenizer::add_tokens
  16: tokenizers::tokenizer::__init11742626496714830824::__init11742626496714830824::__wrap
  17: method_vectorcall_VARARGS_KEYWORDS
             at /tmp/build/80754af9/python_1593706424329/work/Objects/descrobject.c:332
  18: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  19: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  20: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  21: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  22: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  23: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  24: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  25: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  26: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
  27: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
  28: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  29: method_vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/classobject.c:60
  30: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  31: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  32: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3515
  33: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
  34: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
  35: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  36: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  37: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  38: gen_send_ex
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:222
  39: _PyGen_Send
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:292
  40: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:2053
  41: gen_send_ex
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:222
  42: _PyGen_Send
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:292
  43: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:2053
  44: gen_send_ex
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:222
  45: _PyGen_Send
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:292
  46: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:2053
  47: gen_send_ex
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:222
  48: _PyGen_Send
             at /tmp/build/80754af9/python_1593706424329/work/Objects/genobject.c:292
  49: task_step_impl
             at /usr/local/src/conda/python-3.8.3/Modules/_asynciomodule.c:2638
  50: task_step
             at /usr/local/src/conda/python-3.8.3/Modules/_asynciomodule.c:2931
  51: _PyObject_MakeTpCall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:159
  52: context_run
             at /tmp/build/80754af9/python_1593706424329/work/Python/context.c:634
  53: cfunction_vectorcall_FASTCALL_KEYWORDS
             at /tmp/build/80754af9/python_1593706424329/work/Objects/methodobject.c:437
  54: PyVectorcall_Call
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:199
  55: do_call_core
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4983
  56: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3559
  57: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  58: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  59: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  60: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  61: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  62: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  63: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  64: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  65: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  66: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  67: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  68: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  69: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  70: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  71: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  72: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  73: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  74: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  75: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  76: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  77: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  78: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  79: _PyObject_FastCallDict
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:96
  80: _PyObject_Call_Prepend
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:887
  81: slot_tp_call
             at /tmp/build/80754af9/python_1593706424329/work/Objects/typeobject.c:6521
  82: _PyObject_MakeTpCall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:159
  83: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:125
  84: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  85: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3500
  86: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
  87: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
  88: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  89: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  90: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
  91: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
  92: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
  93: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  94: method_vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/classobject.c:60
  95: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
  96: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
  97: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3515
  98: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
  99: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
 100: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
 101: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
 102: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
 103: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
 104: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
 105: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
 106: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
 107: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
 108: function_code_fastcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:283
 109: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:410
 110: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
 111: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
 112: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3486
 113: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
 114: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
 115: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
 116: method_vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/classobject.c:89
 117: PyVectorcall_Call
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:199
 118: do_call_core
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:5010
 119: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3559
 120: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
 121: _PyFunction_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Objects/call.c:435
 122: _PyObject_Vectorcall
             at /tmp/build/80754af9/python_1593706424329/work/Include/cpython/abstract.h:127
 123: call_function
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4963
 124: _PyEval_EvalFrameDefault
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:3500
 125: _PyEval_EvalCodeWithName
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4298
 126: PyEval_EvalCodeEx
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:4327
 127: PyEval_EvalCode
             at /tmp/build/80754af9/python_1593706424329/work/Python/ceval.c:718
 128: run_eval_code_obj
             at /tmp/build/80754af9/python_1593706424329/work/Python/pythonrun.c:1125
 129: run_mod
             at /tmp/build/80754af9/python_1593706424329/work/Python/pythonrun.c:1147
 130: PyRun_FileExFlags
             at /tmp/build/80754af9/python_1593706424329/work/Python/pythonrun.c:1063
 131: PyRun_SimpleFileExFlags
             at /tmp/build/80754af9/python_1593706424329/work/Python/pythonrun.c:428
 132: pymain_run_file
             at /tmp/build/80754af9/python_1593706424329/work/Modules/main.c:387
 133: pymain_run_python
             at /tmp/build/80754af9/python_1593706424329/work/Modules/main.c:571
 134: Py_RunMain
             at /tmp/build/80754af9/python_1593706424329/work/Modules/main.c:650
 135: Py_BytesMain
             at /tmp/build/80754af9/python_1593706424329/work/Modules/main.c:1096
 136: __libc_start_main
 137: <unknown>
             at ../sysdeps/x86_64/elf/start.S:103
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
Aborted (core dumped
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Tokens should get added normally

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-1032-azure-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
I tried using BertTokenizerFast but the problem persists.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Is there any update on the issue? 
I have the same problem...

Also having the same problem while using `tokenizer.add_tokens` with a `unique_list` that holds the words I am trying to add to my tokenizer.

I also have this issue. 

same issue