Adding GPT-NeoX-20B
Author: zphang
Pull request: yes

# What does this PR do?

Adds GPT-NeoX-20B model and tokenizers.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/15642

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@LysandreJik 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Incredible work!

I have tested the model and seems to work as intended. I did discover one problem with the tokenizer though:

Here is the full script:

```
!git clone https://github.com/zphang/transformers
!cd transformers
!git checkout neox20b
!pip install -e .
!cd ..

from transformers import AutoModelForCausalLM, GPTNeoXTokenizer


model_name = r"EleutherAI/gpt-neox-20b"

model = AutoModelForCausalLM.from_pretrained(model_name)

tokenizer = GPTNeoXTokenizer.from_pretrained(model_name)

input_ids=tokenizer.encode("This is the input text", return_tensors="pt",add_special_tokens=False)
beam_output = model.generate(
      input_ids=input_ids,
      max_length=input_ids.shape[1]+30,
      min_length=input_ids.shape[1]+5,
      early_stopping=True,
      num_return_sequences=4,
      do_sample=True
      )

for j in range(4):
        output = tokenizer.decode(beam_output[j][input_ids.shape[1]:], skip_special_tokens=False)
```


I got the following error:

  File "testing/testDecoderOnly.py", line 104, in testModelSample
    ran = tokenizer.decode(beam_output[j][input_ids.shape[1]:], skip_special_tokens=False)
  File "/home/ec2-user/t5-regression3/transformers/src/transformers/tokenization_utils_base.py", line 3308, in decode
    **kwargs,
  File "/home/ec2-user/t5-regression3/transformers/src/transformers/tokenization_utils.py", line 946, in _decode
    sub_texts.append(self.convert_tokens_to_string(current_sub_text))
  File "/home/ec2-user/t5-regression3/transformers/src/transformers/models/gpt2/tokenization_gpt2.py", line 266, in convert_tokens_to_string
    text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
  File "/home/ec2-user/t5-regression3/transformers/src/transformers/models/gpt2/tokenization_gpt2.py", line 266, in <listcomp>
    text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
KeyError: ' '


Hm yea, I can replicate that issue too. I'm not too familiar with the tokenization code. The fast tokenizer seems to work just fine, but the Python one (which I'm basing of GPT-2's tokenizer) seems to have some issues.

Here's the a minimal reproducible version:

```python
import transformers
model_name = "EleutherAI/gpt-neox-20b"
tokenizer_slow = transformers.GPTNeoXTokenizer.from_pretrained(model_name)
tokenizer_fast = transformers.GPTNeoXTokenizerFast.from_pretrained(model_name)
print("Fast", repr(tokenizer_fast.decode([50274])))
print("Slow", repr(tokenizer_slow.decode([50274])))
```
```
Fast '    '
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [78], in <cell line: 2>()
      1 print("Fast", repr(tokenizer_fast.decode([50274])))
----> 2 print("Slow", repr(tokenizer_slow.decode([50274])))

File ~/code/transformers/src/transformers/tokenization_utils_base.py:3304, in PreTrainedTokenizerBase.decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3301 # Convert inputs to python lists
   3302 token_ids = to_py_obj(token_ids)
-> 3304 return self._decode(
   3305     token_ids=token_ids,
   3306     skip_special_tokens=skip_special_tokens,
   3307     clean_up_tokenization_spaces=clean_up_tokenization_spaces,
   3308     **kwargs,
   3309 )

File ~/code/transformers/src/transformers/tokenization_utils.py:946, in PreTrainedTokenizer._decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, spaces_between_special_tokens, **kwargs)
    944         current_sub_text.append(token)
    945 if current_sub_text:
--> 946     sub_texts.append(self.convert_tokens_to_string(current_sub_text))
    948 if spaces_between_special_tokens:
    949     text = " ".join(sub_texts)

File ~/code/transformers/src/transformers/models/gpt2/tokenization_gpt2.py:266, in GPT2Tokenizer.convert_tokens_to_string(self, tokens)
    264 """Converts a sequence of tokens (string) in a single string."""
    265 text = "".join(tokens)
--> 266 text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
    267 return text

File ~/code/transformers/src/transformers/models/gpt2/tokenization_gpt2.py:266, in <listcomp>(.0)
    264 """Converts a sequence of tokens (string) in a single string."""
    265 text = "".join(tokens)
--> 266 text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
    267 return text

KeyError: ' '
```

I believe the NeoX tokenizer handles spaces a little differently (it has special tokens for single, double, triple spaces, etc). Do you know if someone who's more familiar with tokenization code might be able to chime in?

Great that the fast version works. Gently pinging @SaulLu and @Narsil if they have any answers.

Hey @LysandreJik, thanks for taking a look! I'll look into getting the tests to pass today.

Re: the model script, I did use the new model templating script, but many parts of it seemed to make the assumption that the model with be an encoder-decoder model (e.g. mentioning cross attention). I removed most of other model implementations aside from CasualLM as that's the primary format that NeoX-20B would be used for, but it looks like I missed out some other references to the other model implementations. Other than that, the script was very useful in setting up the boilerplate.

added a PR to the PR to support `AutoTokenizer` initialization from `pretrained_model_name_or_path`:
https://github.com/zphang/transformers/pull/1

Tests are passing now, though the non-fast tokenizer still needs looking at I think

Hi @zphang and @ViktorThink 

Thank you very much for working on the addition of this model! :hugs: 

As you mentioned several times in this PR, I took a closer look at the error currently returned by the slow tokenizer with the snippet:
```python
import transformers
model_name = "EleutherAI/gpt-neox-20b"
tokenizer_slow = transformers.GPTNeoXTokenizer.from_pretrained(model_name)
tokenizer_fast = transformers.GPTNeoXTokenizerFast.from_pretrained(model_name)
print("Fast", repr(tokenizer_fast.decode([50274])))
print("Slow", repr(tokenizer_slow.decode([50274])))
```
Looking at the vocabulary file ([here](https://huggingface.co/EleutherAI/gpt-neox-20b/resolve/main/vocab.json)), it seems that the id `50274` corresponds to `"   "` (3 spaces). 

For me this token does not correspond to a token that could have been produced by learning a byte-level BPE - and thus it's not surprising that the GPT2 tokenizer raises an error . To my knowledge, the GPT2 byte-level BPE  uses a little trick: instead of working on the pure byte sequence, some bytes are shifted to match visible characters - e.g. the space `" "` (`\u0020`) is replaced by `"Ġ"` (`\u0120`). This offset is removed at the very end during decoding. Therefore, this trick implies that we can't have non-visible character in the vocabulary like the space `"\u0020"` used in the previous example. On the contrary, a token of which byte-level BPE could have been aware is `"ĠĠĠ"`. Then, maybe  `"   "` is an added token that should be preserved from the beginning of the tokenization? (If it's the case we need to add them as added tokens in GPTNeoX's tokenizer).

**TL:DR:** I think that to solve this problem it would be necessary to understand how this vocabulary was obtained. Do you have more information about this? :blush: 

Let me check in with the folks who prepared the tokenizer

@SaulLu we added some custom tokens to the tokenizer before training it corresponding to various numbers of spaces. There is a single token representation of one space, two spaces, three spaces, ..., twenty-four spaces. We created a custom tokenizer file that you can find [here](https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/20B_tokenizer.json) which should be consistent with the Tokenizers library.

Thank you very much for the information @StellaAthena! It makes sense! 

So if the original tokenizer is based on the `tokenizers` rust backend, maybe it is not necessary to provide a slow version? WDYT?

In short, I think it would be difficult to align the behavior of slow and fast version which these different space sets added as added tokens. And my opinion is that if the original tokenizer is based on the rust backend, the specific development required to create the slow tokenizer may not be "cost"-effective.

---------------------------
:point_down:  More details on why I think adding the different space sets as added tokens won't give the same tokenization between the slow and the fast versions
 
1. In slow tokenizers added tokens cannot be added by specifying `lstrip`, `rstrip` etc arguments as it is possible to do in fast because of this line:
https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/src/transformers/tokenization_utils.py#L410

2. For the slow tokens, the added tokens therefore have the behaviour of `rstrip=True` and `lstrip=True` around the added token because of this part of the code (we don't go into `if isinstance(tok_extended, AddedToken):` because of the point 1. above but into the second block `else: `):
https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/src/transformers/tokenization_utils.py#L515-L536

Concretely, let's look on an example at the solution which would consist in adding the space sets as added tokens. It would give these 2 different tokenizations:

```python
from transformers import GPT2Tokenizer, GPT2TokenizerFast

model_name = "gpt-neox-20b-with-added-space-tokens" # Assuming that the tokens discussed above are added as tokens and not in the vocabulary

tokenizer_s = GPT2Tokenizer.from_pretrained(model_name)
tokenizer_f = GPT2TokenizerFast.from_pretrained(model_name)

text = '''    def add(a,b):
        return a + b'''

print(tokenizer_s.convert_ids_to_tokens(tokenizer_s.encode(text)))
print(tokenizer_f.convert_ids_to_tokens(tokenizer_f.encode(text)))

print(tokenizer_s.convert_ids_to_tokens(tokenizer_s.encode(text)))
print(tokenizer_f.convert_ids_to_tokens(tokenizer_f.encode(text)))
# ['    ', 'def', 'Ġadd', '(', 'a', ',', 'b', '):', '        ', 'return', 'Ġa', 'Ġ+', 'Ġb']
# ['    ', 'def', 'Ġadd', '(', 'a', ',', 'b', '):', 'Ċ', '        ', 'return', 'Ġa', 'Ġ+', 'Ġb']
```
You can see that the line break is eaten with the slow version.

@SaulLu That makes sense to me. If a slow tokenizer isn’t necessary I don’t see a reason to try to force it

Ok! I wasn't sure if it was always standard/required to provide the slow tokenizer. If we are okay with only providing a fast tokenizer, I can go ahead and remove the slow one.

I'll also try to address the rest of the comments this weekend (have been a little busy).

Thanks everyone for all your feedback!

@zphang if you give me push access to the repo I am happy to help

I have  resolved the merge conflicts in the config files, but I am not confidant in my understanding of how these various configs are supposed to work. I would appreciate it if someone double checked that I didn't do anything stupid.

Are there any further blockers to merging? It would be nice to have this merged in time for ACL next week :)

Apologies, I must have missed the previous comments. I've pushed an update with the desired changes.

There are still four open comment on the modeling file, if you could have a look.

I think I got to all of them now (is there an easy way to check on the GitHub web interface?), let me know if I'm missing any.

I see there are closed but not addressed, maybe you forgot to push your commit?

Terribly sorry! Pushed now.

Thanks again for all your wok on this model!