LLaMA Implementation
Author: zphang
Pull request: yes

# What does this PR do?

Implementation of LLaMA models (https://arxiv.org/abs/2302.13971). Model weights can be requested [here](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform). Weight conversion script is included.

Weights conversion can be run via:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights \
    --model_size 7B \
    --output_dir /output/path
```

Models can then be loaded via:

```python
tokenizer = transformers.LLaMATokenizer.from_pretrained("/output/path/tokenizer/")
model = transformers.LLaMAForCausalLM.from_pretrained("/output/path/llama-7b/")
```

Example:

```bash
batch = tokenizer(
    "The primary use of LLaMA is research on large language models, including",
    return_tensors="pt", 
    add_special_tokens=False
)
batch = {k: v.cuda() for k, v in batch.items()}
generated = model.generate(batch["input_ids"], max_length=100)
print(tokenizer.decode(generated[0]))
```

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/21796

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @younesbelkada

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
does this work with int8?

> does this work with int8?

No idea! I haven't messed with int8 too much myself. It ought to be compatible with whatever is already supported in the HF models.

nice work! thanks for the upload and I hope it gets pulled

_The documentation is not available anymore as the PR was closed or merged._

It looks like the tests which are currently failing are unrelated to the LLaMA code, so this should be good to review/use.

If folks can try it out (particularly with the larger, sharded models) and see if there are any issues, that will be helpful!

> It looks like the tests which are currently failing are unrelated to the LLaMA code, so this should be good to review/use.
> 
> If folks can try it out (particularly with the larger, sharded models) and see if there are any issues, that will be helpful!

At lest the convert script seems to work fine. I was able to convert 7B to 30B. I do not have enough ram to convert 65B.

Great work. thanks for putting this together

After replacing transformers from Kobold with this PR I am able to load the shards as expected. Just I cant generate anything because Kobold still needs some changes.
![image](https://user-images.githubusercontent.com/7269941/222938895-2e8b9d71-6a88-417d-b7ed-14d8216d2ef4.png)


> > does this work with int8?
> 
> No idea! I haven't messed with int8 too much myself. It ought to be compatible with whatever is already supported in the HF models.

Int8 seems not working but float16 is fine, in my hasty put-together test at https://github.com/zsc/llama_infer . Please throw a comment in case you find something!

@zphang  I'm not able to get something like  `tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/")` to work. Is this intentional or just leaving AutoTokenizer for future work?

> @zphang I'm not able to get something like `tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/")` to work. Is this intentional or just leaving AutoTokenizer for future work?

What issue are you having / what is the error?

I have tested the code and these are my findings:

1. The conversion script works.
2. Loading the model works.
3. Loading the tokenizer with `transformers.LLaMATokenizer.from_pretrained` works.
4. Loading the tokenizer with `AutoTokenizer.from_pretrained` does not work and generates this error:

```
OSError: /tmp/converted/tokenizer/ does not appear to have a file named config.json. Checkout
'https://huggingface.co//tmp/converted/tokenizer//None' for available files.

```

5. The generated text seems to be incoherent. If I try these default values for the generation parameters:

```
model.generate(input_ids, eos_token_id=2, do_sample=True, temperature=1, top_p=1, typical_p=1, repetition_penalty=1, top_k=50, min_length=0, no_repeat_ngram_size=0, num_beams=1, penalty_alpha=0, length_penalty=1, early_stopping=False, max_new_tokens=200).cuda()
```

with this prompt:

```
Common sense questions and answers

Question: What color is the sky?
Factual answer:
```

I get

```
Common sense questions and answers

Question: What color is the sky?
Factual answer: Tags: python, django, django-models

Question: Using Django with multiple databases

I am attempting to use django with multiple databases, and I have the following code:

\begin{code}
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': ':memory:',
    },
    'db_one': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'db_one',
    },
    'db_two': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'db_two',
    },
}
```

It seems to me that prompts are being completely ignored.

6. Loading in 8-bit mode with `load_in_8bit=True` works.

This is OK: `tokenizer = transformers.LLaMATokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/")`

If using `tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/"` then it will complain no "config.json". 
```
OSError: /data/llama/hf/7b/tokenizer/ does not appear to have a file named config.json. Checkout 
'https://huggingface.co//data/llama/hf/7b/tokenizer//None' for available files.
```

I then hacked by softlinking `/data/llama/hf/7b/tokenizer/special_tokens_map.json` to `/data/llama/hf/7b/tokenizer/config.json` and it works. So maybe just rename?

Anyway, can now happily play with LLaMA in Hugging Face world and thanks for the great work!

Thanks for the comments. Looks like the saved tokenizer doesn't work for `AutoTokenizer` but works if you directly instantiate from `LLaMATokenizer`. Maybe one of the HF folks can chime in on the best way to address that.

> The generated text seems to be incoherent. If I try these default values for the generation parameters:

Can you check the input_ids you're using to generate? The tokenizer currently adds both BOS and EOS tokens by default, and an EOS might cause the model to ignore your prompt.

Perhaps I can set EOS to not be added by default so it operates closer to expected behavior.

For this prompt:

```
'Common sense questions and answers\n\nQuestion: What color is the sky?\nFactual answer:'
```

these are the input_ids:

```
tensor([[    1, 13103,  4060,  5155,   322,  6089,    13,    13, 16492, 29901,
          1724,  2927,   338,   278, 14744, 29973,    13, 29943, 19304,  1234,
         29901,     2]], device='cuda:0')
```

I do not know how to interpret these numbers, but if there is an EOS token in that tensor and that token is causing the text generation to derail, changing that default would be valuable.

1 is BOS and 2 is EOS. Can you try without the last input id?

I also added an example in my PR message.

I confirm that doing this 
```
    input_ids = input_ids[:, :-1]

```

to remove the last input id before calling `model.generate(...)` causes the text generation to become coherent:

```
Common sense questions and answers

Question: What color is the sky?
Factual answer: The sky is blue. The sky is blue, and it is a fact that it is blue. The sky is indisputably blue.

```

Added a commit that should fix the tokenizer issues, and not add BOS and EOS by default.

Awesome, I confirm that the text generation is coherent by default now.

I still cannot load the tokenizer with `AutoTokenizer.from_pretrained`. The error has now changed to this:

```
  File "/tmp/transformers/src/transformers/models/auto/tokenization_auto.py", line 694, in from_pretrained
    tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
  File "/tmp/transformers/src/transformers/models/auto/auto_factory.py", line 610, in __getitem__
    raise KeyError(key)
KeyError: <class 'transformers.models.llama.configuration_llama.LLaMAConfig'>

```

> > does this work with int8?
> 
> No idea! I haven't messed with int8 too much myself. It ought to be compatible with whatever is already supported in the HF models.

After the fix with EOS, int8 (bitsandbytes) looks decent. Example in https://github.com/zsc/llama_infer/blob/main/README.md

After https://github.com/huggingface/transformers/pull/21955/commits/459e2ac9f551650ced58deb1c65f06c3d483d606, `AutoTokenizer.from_pretrained` now works as expected.


KoboldAI now works

I'd like to see a more memory-efficient conversion script, the current version loads everything into system memory which makes converting the 30B and 65B variants challenging on some systems

Yes, this is a quick and dirty version that loads everything into memory.
One issue is that the way the weights are sharded (for tensor parallelism) is orthogonal to the way that HF shards the weights (by layer). So either we have to load everything in at once, or we have to load/write multiple times. The latter would be slower but useful for folks with less memory.

Has anyone tested loading 65B with `accelerate` to load on multiple GPUs?

I can't load the 7B model to cuda with one A4000
should I just change the gpu?


I'm observing some strange behavior with the tokenizer when encoding sequences beginning with a newline:
```
>>> t = AutoTokenizer.from_pretrained("llama_hf/tokenizer")
>>> res = t.encode("\nYou:")
>>> res
[29871, 13, 3492, 29901]
>>> t.decode(res)
'You:'
```

The newline seems to get lost somewhere along the way.

EDIT: Looking into this, it seems it might be the expected behavior of `sentencepiece`.

> Has anyone tested loading 65B with `accelerate` to load on multiple GPUs?

||fp16|int8(bitsandbytes)|
|--|--|--|
|V100|OK, 5xV100|Bad results, short generated sequences|
|A100|OK, 6xA100 when using "auto"|OK, 3xA100|

Yes, I currently have a 65B fp16 model running on 6xV100 now (5X should be enough). My working code is at https://github.com/zsc/llama_infer/ . If there are CUDA OOM due to bad distribution of weights among cards, one thing worth trying is tweaking the device_map (`accelerate` seems to only counts weights when enforcing the memory cap in device_map, so there is an art for setting custom cap a little lower for every card, especially card 0).

Strangely, int8 (LLM.int8 to be specific) for 65B model works like a charm on A100, but leads to bad results on V100 with abnormally short generated sequences.

> Strangely, int8 (LLM.int8 to be specific) for 65B model works like a charm on A100, but leads to bad results on V100 with abnormally short generated sequences.

I will have a look at this later next week. The V100 takes a different code path than the A100 because the V100 does not support Int8 tensor cores. I think that is the issue here. We will soon publish FP4 inference which should be more universal and easier to use.

Jumping on @thomasw21 comment, we sadly cannot accept any code licensed GPLv3 as it would taint the whole Transformers library under that license. This means that the modeling code should be copied from GPT-NeoX whenever possible (with Copied from statements) since I believe that this model is very close to it and that you should be super familiar with it @zphang ;-) , and that no parts of the modeling code should be copy-pasted from the original Llama code.

We also cannot attribute Copyright to Meta-AI /Meta in all those files, as attributing that copyright would admit the code in the PR is based on theirs and thus get us back to the license problem.

I tried quantizing LLaMa using [GPTQ](https://arxiv.org/abs/2210.17323).
I've confirmed that 4 or 3 bit quantization works well, at least on LLaMa 7b.
| Model([LLaMa-7B](https://arxiv.org/abs/2302.13971))      | Bits | group-size | Wikitext2 |   PTB     |    C4   |
| ---------                                                | ---- | ---------- | --------- | --------- | ------- |
| FP16                                                     |  16  |     -      |    5.67   |    8.79   |   7.05  | 
| RTN                                                      |  4   |     -      |    6.28   |    9.68   |   7.70  | 
| [GPTQ](https://arxiv.org/abs/2210.17323)                 |  4   |    64      |    **6.16**   |    **9.66**   |   **7.52**  | 
| RTN                                                      |  3   |     -      |    25.66   |    61.25   |   28.19  | 
| [GPTQ](https://arxiv.org/abs/2210.17323)                 |  3   |    64      |    **12.24**   |    **16.77**   |   **9.55**  | 

I have released the [code](https://github.com/qwopqwop200/GPTQ-for-LLaMa) for this.


Running into an issue with the following code
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("models/facebook_LLaMA-7b")
output = [[v] for k, v in tokenizer.get_vocab().items() if any(c in str(k) for c in "<>[]")]
debug_output = tokenizer.batch_decode(output)
print(debug_output)
```

This is supposed to only print tokens that are including one of the <>[] characters, but I am getting a wide range back at the beginning causing our code to effect way to many tokens. This includes tokens that are blatantly false such as every singular letter of the alphabet.
Is this to be expected with the kind of tokenizer that is used and do we have to find an alternative implementation or is something going wrong here on a deeper level?

If I try the same thing with a tokenizer from the usual models we run such as the GPT-J tokenizer this code runs properly.

> I tried quantizing LLaMa using [GPTQ](https://arxiv.org/abs/2210.17323). I've confirmed that 4 or 3 bit quantization works well, at least on LLaMa 7b.
> 
> Model([LLaMa-7B](https://arxiv.org/abs/2302.13971))	Bits	group-size	Wikitext2	PTB	C4
> FP16	16	-	5.67	8.79	7.05
> RTN	4	-	6.28	9.68	7.70
> [GPTQ](https://arxiv.org/abs/2210.17323)	4	64	**6.16**	**9.66**	**7.52**
> RTN	3	-	25.66	61.25	28.19
> [GPTQ](https://arxiv.org/abs/2210.17323)	3	64	**12.24**	**16.77**	**9.55**
> I have released the [code](https://github.com/qwopqwop200/GPTQ-for-LLaMa) for this.

Do you plan to publish 4-bit quantized models on the hub? If not, I'd be more than happy to generate them and push them up.

> > [GPTQ를](https://arxiv.org/abs/2210.17323) 사용하여 LLaMa를 양자화해 보았습니다 . 적어도 LLaMa 7b에서는 4비트 또는 3비트 양자화가 잘 작동함을 확인했습니다.
> > Model( [LLaMa-7B](https://arxiv.org/abs/2302.13971) ) Bits group-size Wikitext2 PTB C4 [FP16](https://github.com/qwopqwop200/GPTQ-for-LLaMa) 
> > 16 - 5.67 8.79 7.05 
> > RTN 4 - 
> > 6.28 9.68 7.70 
> > [GPTQ](https://arxiv.org/abs/2210.17323) 	4 64	 **6.16 **	**9.66 **	**7.52** RTN 
> > 3 - 25.66 61.25 28.19 **GPTQ **
> > [3](https://arxiv.org/abs/2210.17323) 	64	 **12.75 ****1** .****	****	****[](https://github.com/qwopqwop200/GPTQ-for-LLaMa)
> 
> 허브에 4비트 양자화 모델을 게시할 계획입니까? 그렇지 않다면 기꺼이 생성하고 밀어 올릴 것입니다.

There are no plans to publish a 4-bit quantization model yet.
And to get advantages in speed and memory, you need to change and use dedicated 3-bit CUDA Kernels.
I haven't even tested 3-bit CUDA Kernels yet.

Maybe this issure need to be fixed?

the  [llama-int8](https://github.com/tloen/llama-int8)  can works.

but this

```
from  transformers import LLaMaForCausalLM,LLaMaPreTrainedModel
model = LLaMaForCausalLM.from_pretrained("Z:/LLaMA/llama-7b-hf")
```

errors:

File [G:\pythons\huggingface\transformers\src\transformers\modeling_utils.py:2632](file:///G:/pythons/huggingface/transformers/src/transformers/modeling_utils.py:2632), in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2622     if dtype_orig is not None:
   2623         torch.set_default_dtype(dtype_orig)
   2625     (
   2626         model,
   2627         missing_keys,
   2628         unexpected_keys,
   2629         mismatched_keys,
   2630         offload_index,
   2631         error_msgs,
-> 2632     ) = cls._load_pretrained_model(
   2633         model,
   2634         state_dict,
   2635         loaded_state_dict_keys,  # XXX: rename?
   2636         resolved_archive_file,
   2637         pretrained_model_name_or_path,
   2638         ignore_mismatched_sizes=ignore_mismatched_sizes,
   2639         sharded_metadata=sharded_metadata,
...
   1268         return modules[name]
-> 1269 raise AttributeError("'{}' object has no attribute '{}'".format(
   1270     type(self).__name__, name))

AttributeError: 'LLaMaLayerNorm' object has no attribute 'bias'


#

> Maybe this issure need to be fixed?
> 
> the [llama-int8](https://github.com/tloen/llama-int8) can works.
> 
> but this
> 
> ```
> from  transformers import LLaMaForCausalLM,LLaMaPreTrainedModel
> model = LLaMaForCausalLM.from_pretrained("Z:/LLaMA/llama-7b-hf")
> ```
> 
> errors:
> 
> File G:\pythons\huggingface\transformers\src\transformers\modeling_utils.py:2632, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 2622 if dtype_orig is not None: 2623 torch.set_default_dtype(dtype_orig) 2625 ( 2626 model, 2627 missing_keys, 2628 unexpected_keys, 2629 mismatched_keys, 2630 offload_index, 2631 error_msgs, -> 2632 ) = cls._load_pretrained_model( 2633 model, 2634 state_dict, 2635 loaded_state_dict_keys, # XXX: rename? 2636 resolved_archive_file, 2637 pretrained_model_name_or_path, 2638 ignore_mismatched_sizes=ignore_mismatched_sizes, 2639 sharded_metadata=sharded_metadata, ... 1268 return modules[name] -> 1269 raise AttributeError("'{}' object has no attribute '{}'".format( 1270 type(self).**name**, name))
> 
> AttributeError: 'LLaMaLayerNorm' object has no attribute 'bias'

I'm not sure how that code even runs, the case doesn't match. You're importing `LLaMaForCausalLM` and the code provides `LLaMAForCausalLM`

Also, where did you pull the model from? Is that the copy from decapoda-research on the hub?

> Jumping on @thomasw21 comment, we sadly cannot accept any code licensed GPLv3 as it would taint the whole Transformers library under that license. This means that the modeling code should be copied from GPT-NeoX whenever possible (with Copied from statements) since I believe that this model is very close to it and that you should be super familiar with it @zphang ;-) , and that no parts of the modeling code should be copy-pasted from the original Llama code.
> 
> We also cannot attribute Copyright to Meta-AI /Meta in all those files, as attributing that copyright would admit the code in the PR is based on theirs and thus get us back to the license problem.

Note that under Apache 2.0 the correct thing to do is credit _EleutherAI_ with the copyright. I have gone ahead and added the proper copyright headers, including adding a notice informing the reading that the code has been modified from it's original version and where it was adapted from.

**Note that I have not changed the code itself in any way.** If it is currently the case that some of the code is based on the FairSeq implementation that still needs to be removed and rewritten using GPT-NeoX (whether the implementation in this library or the GPT-NeoX library itself) as a reference. However once all GPLv3 licensed code has been removed, the headers I added should be correct.

cc: @zphang @sgugger @thomasw21 for visibility.

It might be worth mentioning this in the documentation of the model, as I anticipate this will cause some people confusion.

> Also, where did you pull the model from? Is that the copy from decapoda-research on the hub?

@zoidbb this PR includes a script for converting the models from their original format to a HF-compliant one.

@henk717 your code doesn't work for me with any model. I consistently error on the line `debug_output = tokenizer.batch_decode(output)`

Thanks for your contribution! However, I find that the model usually ends up repeating the same sentence during inference. For example, by running the code below:

```
import transformers
import os

os.environ['CUDA_VISIBLE_DEVICES']='3'

config = transformers.AutoConfig.from_pretrained("./llama-7b-hf/llama-7b")
tokenizer = transformers.AutoTokenizer.from_pretrained("./llama-7b-hf/tokenizer")
model = transformers.AutoModelForCausalLM.from_pretrained("./llama-7b-hf/llama-7b", config=config).cuda()

batch = tokenizer(
    ["The primary use of LLaMA is research on large language models, including "],
    return_tensors="pt"
)

batch = {k: v.cuda() for k, v in batch.items()}
input_ids = batch["input_ids"] # tensor([[1,...]])

generated = model.generate(input_ids, temperature=0.8, top_p=0.95, max_length=256).cuda()
print(tokenizer.decode(generated[0])
```

**the LLaMA-7b-hf output is:**
![1](https://user-images.githubusercontent.com/45878717/223327316-99672393-3d31-4531-bd3c-bc218f1dddd6.png)

**while the original LLaMA(7b) output is (temperature=0.8, top_p=0.95):**
![2](https://user-images.githubusercontent.com/45878717/223327389-5c635bd2-121a-4990-9998-3fbc6f448bd2.png)

Now it seems that the two models are quite different with each other. I wonder if they can output the same sentence, given the same input and parameters?

after install your branch of llama, i can't import LLaMaTokenizer as below
 ImportError: cannot import name 'LLaMaTokenizer' from 'transformers'

but import LLaMaForCausalLM is successful...
how could this happen?

Hey all! Great to see that the community is so fired up for this PR 🤗 🔥 ! @zphang thanks a lot for this contribution, don't hesitate to  ping me once this is ready for review,. In the mean time let's all give him some time, as it was never stated that this should already work or compile!
Let's wait until then to report potential issues ! 😉 

@StellaAthena Fully in line with your change of copyright and also to explicitly mention in the doc page of the model that the code is based on GPT-NeoX and not the original Llama code. This can be put in the usual place we say: "This model was contributed by xxx. The original code can be found [here](yyy)."

> > Has anyone tested loading 65B with `accelerate` to load on multiple GPUs?
> 
> fp16	int8(bitsandbytes)
> V100	OK, 5xV100	Bad results, short generated sequences
> A100	OK, 6xA100 when using "auto"	OK, 3xA100
> Yes, I currently have a 65B fp16 model running on 6xV100 now (5X should be enough). My working code is at https://github.com/zsc/llama_infer/ . If there are CUDA OOM due to bad distribution of weights among cards, one thing worth trying is tweaking the device_map (`accelerate` seems to only counts weights when enforcing the memory cap in device_map, so there is an art for setting custom cap a little lower for every card, especially card 0).
> 
> Strangely, int8 (LLM.int8 to be specific) for 65B model works like a charm on A100, but leads to bad results on V100 with abnormally short generated sequences.

CC: @zsc 
Just a heads up on the issue with V100 and bitsandbytes, I've encountered a bit weird and may be relevant behavior with long inputs, but relaxing the quantization threshold by `BitsAndBytesConfig(llm_int8_threshold=5.0)` (which I believe is set to 6 by default  according to the [doc](https://huggingface.co/docs/transformers/main/main_classes/quantization)) resolved the issue. Ref: https://github.com/huggingface/transformers/issues/21987


@zoidbb  Thank you!!! I'm sorry, the error is the branch of "thomas/llama", not this!

Heads up to folks already using this PR: I will be pushing some changes today to address the maintainers' feedback that will break compatibility. In particular, I will be making a major change to how the RoPE encodings are computed. Please pull and re-convert the weights when the branch is updated. Sorry for the inconvenience.

> 

Thanks for the heads up 

As mentioned above, I've pushed an update (previous git hash was: 6a17e7f) that introduces some breaking changes to incorporate the Transformers maintainers' feedback. (There are still some tweaks to go, but nothing that should be breaking from here.)

**Please rerun the weight conversion scripts.**

Major changes:
- Removal of Decoder class
- `add_bos` on by default. `add_eos` is still off.
- Reimplementation of RoPE based on NeoX version. This required a weight permutation (NeoX's RoPE slices the head_dims in half, the Meta implementation interleaves), which is performed in the conversion script. **Note**: the weights are still the same shape, so if you don't rerun the conversion script and only rename the keys, the weights will load but be completely incorrect.

As before, please let me know if you run into any issues, it is great to have this many people trying and testing the implementation.

Running into issues with the model conversion on the latest update, loading the model fails (On the latest version of the pull request git) and pytorch_model-00033-of-00033 is missing.

Using the custom KoboldAI loader it reports a key error, and using the official HF loader it mentions not all weights could be initialized correctly but does finish the loading process.

> Running into issues with the model conversion on the latest update, loading the model fails (On the latest version of the pull request git) and pytorch_model-00033-of-00033 is missing.
> 
> Using the custom KoboldAI loader it reports a key error, and using the official HF loader it mentions not all weights could be initialized correctly but does finish the loading process.

Confirmed this is an issue on 7B and 13B both, the last file is missing both from disk and from the index file.

@zphang you have a typo in the line defining the filename for the last .bin file, you use n_layers instead of n_layers+1. this leads to part 32 being overwritten, instead of part 33 being written (in the case of 7B).

Hmm now I am a little confused. There is an off-by-one error, but that's it going from 0 to n_layers instead of 1 to n_layers+1. It ought to load just fine though, since it's consistent with the index JSON.

(The +1 is for the embedding layers and final norm)

Pushed fix for off-by-one naming.

@zphang now im getting this error:

```
RuntimeError: Failed to import transformers.models.llama.tokenization_t5 because of the following error (look up to see its traceback):
No module named 'transformers.models.llama.tokenization_t5'
```

Try again now!

Seems to load, running the new model conversion through its paces. Thanks for your work.

I am not having any luck yet, my errors remain identical. For context I use AutoModelForCausalLM in the code.

> I am not having any luck yet, my errors remain identical. For context I use AutoModelForCausalLM in the code.

I'm also using that, and it's working fine. Did you make sure to grab the latest commits and install them? Also you have run the conversion again.

Late night coding moment, my update script tricked me into thinking it was applying the update but it was only cloning the git instead of applying it (Normally I run a different one that does more that does do that) so my last few attempts used the same code.

The update fixed it, our software also has some regex's to clean up output that were not being applied correctly previously because of an issue I did not manage to track down. On this new iteration of the fork that bug got fixed, so while I do not know what the cause behind that was on the old version I am happy to report the latest version is an improvement for us in functionality.

The model behaves much better with these latest changes, I was always confused why I either got long elaborate nonsense or really short answers when using it as a chatbot :P Thanks for your work @zphang this is epic.

Hi thanks for your great work. Can we fine-tune it for now？

@zphang there seems to be an issue when using generate with `num_return_sequences`.
Running this code:
```python
import torch, os
from transformers import AutoTokenizer, AutoModelForCausalLM
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

modeldir = "./llama-7b"
tokenizer = AutoTokenizer.from_pretrained(modeldir)
model = AutoModelForCausalLM.from_pretrained(modeldir, torch_dtype=torch.float16).to("cuda")

input_ids = tokenizer('aaaaa', return_tensors="pt").input_ids.to("cuda")
for i in range(100):
    print(i)
    model.generate(input_ids,
                   max_new_tokens=20,
                   do_sample=True,
                   num_return_sequences=4)
```
I get this output:
```
0
1
2
3
4
5
6
7
8
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/Indexing.cu:1088: indexSelectSmallIndex: block: [18,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/Indexing.cu:1088: indexSelectSmallIndex: block: [18,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/Indexing.cu:1088: indexSelectSmallIndex: block: [18,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
... a bunch more of those ...
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/Indexing.cu:1088: indexSelectSmallIndex: block: [18,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/Indexing.cu:1088: indexSelectSmallIndex: block: [18,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 12
     10 for i in range(100):
     11     print(i)
---> 12     model.generate(input_ids,
     13                    max_new_tokens=20,
     14                    do_sample=True,
     15                    num_return_sequences=4)

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/autograd/grad_mode.py:27](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/autograd/grad_mode.py:27), in _DecoratorContextManager.__call__..decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/generation/utils.py:1452](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/generation/utils.py:1452), in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, **kwargs)
   1444     input_ids, model_kwargs = self._expand_inputs_for_generation(
   1445         input_ids=input_ids,
   1446         expand_size=generation_config.num_return_sequences,
   1447         is_encoder_decoder=self.config.is_encoder_decoder,
   1448         **model_kwargs,
   1449     )
   1451     # 13. run sample
-> 1452     return self.sample(
   1453         input_ids,
   1454         logits_processor=logits_processor,
   1455         logits_warper=logits_warper,
   1456         stopping_criteria=stopping_criteria,
   1457         pad_token_id=generation_config.pad_token_id,
   1458         eos_token_id=generation_config.eos_token_id,
   1459         output_scores=generation_config.output_scores,
   1460         return_dict_in_generate=generation_config.return_dict_in_generate,
   1461         synced_gpus=synced_gpus,
   1462         **model_kwargs,
   1463     )
   1465 elif is_beam_gen_mode:
   1466     if generation_config.num_return_sequences > generation_config.num_beams:

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/generation/utils.py:2468](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/generation/utils.py:2468), in GenerationMixin.sample(self, input_ids, logits_processor, stopping_criteria, logits_warper, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, **model_kwargs)
   2465 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   2467 # forward pass to get next token
-> 2468 outputs = self(
   2469     **model_inputs,
   2470     return_dict=True,
   2471     output_attentions=output_attentions,
   2472     output_hidden_states=output_hidden_states,
   2473 )
   2475 if synced_gpus and this_peer_finished:
   2476     continue  # don't waste resources running the code we don't need

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/nn/modules/module.py:1194](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/nn/modules/module.py:1194), in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:772](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:772), in LLaMAForCausalLM.forward(self, input_ids, attention_mask, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
    769 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    771 # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
--> 772 outputs = self.model(
    773     input_ids=input_ids,
    774     attention_mask=attention_mask,
    775     past_key_values=past_key_values,
    776     inputs_embeds=inputs_embeds,
    777     use_cache=use_cache,
    778     output_attentions=output_attentions,
    779     output_hidden_states=output_hidden_states,
    780     return_dict=return_dict,
    781 )
    783 hidden_states = outputs[0]
    784 logits = self.lm_head(hidden_states)

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/nn/modules/module.py:1194](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/torch/nn/modules/module.py:1194), in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:581](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:581), in LLaMAModel.forward(self, input_ids, attention_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    578 if attention_mask is None:
    579     attention_mask = torch.ones(inputs_embeds.shape[:2], dtype=torch.bool, device=inputs_embeds.device)
--> 581 attention_mask = self._prepare_decoder_attention_mask(
    582     attention_mask, input_shape, inputs_embeds, past_key_values_length
    583 )
    585 hidden_states = inputs_embeds
    587 if self.gradient_checkpointing and self.training:

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:490](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:490), in LLaMAModel._prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length)
    484     combined_attention_mask = _make_causal_mask(
    485         input_shape, inputs_embeds.dtype, past_key_values_length=past_key_values_length
    486     ).to(inputs_embeds.device)
    488 if attention_mask is not None:
    489     # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
--> 490     expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
    491         inputs_embeds.device
    492     )
    493     combined_attention_mask = (
    494         expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
    495     )
    497 return combined_attention_mask

File [~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:77](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/abstract/projs/pytorch_tests/~/mambaforge/envs/reproenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:77), in _expand_mask(mask, dtype, tgt_len)
     73 expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
     75 inverted_mask = 1.0 - expanded_mask
---> 77 return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)

RuntimeError: CUDA error: device-side assert triggered
```
My env is
```
mamba create -n reproenv python=3.10 jupyter pip pytorch pytorch-cuda=11.7 -c pytorch -c nvidia
conda activate reproenv
pip install git+https://github.com/zphang/transformers@llama_push sentencepiece
```

When using the 30B variant I got this error (do_sample=true):
```
File "/secondary/thies/llama/app.py", line 184, in completions
    ids_outputs = model.generate(tok_inputs, max_new_tokens=max_new_tokens, top_p=top_p, top_k=top_k, temperature=temperature, do_sample=True)
  File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/transformers/generation/utils.py", line 1452, in generate
    return self.sample(
  File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/transformers/generation/utils.py", line 2504, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```
7B variant works fine.

> When using the 30B variant I got this error (do_sample=true):
> 
> ```
> File "/secondary/thies/llama/app.py", line 184, in completions
>     ids_outputs = model.generate(tok_inputs, max_new_tokens=max_new_tokens, top_p=top_p, top_k=top_k, temperature=temperature, do_sample=True)
>   File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
>     return func(*args, **kwargs)
>   File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/transformers/generation/utils.py", line 1452, in generate
>     return self.sample(
>   File "/secondary/thies/.virtualenvs/llama/lib/python3.10/site-packages/transformers/generation/utils.py", line 2504, in sample
>     next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
> RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
> ```
> 
> 7B variant works fine.

I can also reproduce this when using int8 on my 6900xt

Also getting 

```
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0`
```


I noticed that 6a17e7f3b45454429f2b0405e673dc4aaa695546 version seems to work and fixes the `RuntimeError: probability tensor contains either `inf`, `nan` or element < 0` error.

I will try to follow up on some of the other reported issues today/tomorrow. I did indeed see an issue with the combined/expanded masks as @BlackSamorez mentioned, although I will need to take a closer look at the fix since there are so many entry points for HF code. 

I also pushed a fix for what I think was causing some of the above issues. Part of it affects the config file, so you should go into the model config.json and set `pad_token_id=0`. With that first, the example code given by @AbstractQbit should run fine now.

I also wrote some very preliminary fine-tuning code https://github.com/zphang/minimal-llama/ if folks are interested in testing it out. It also uses the code from this PR.

Loading model directly results in warning about not using some weights. Is this expected? Other models (e.g. Bloom) does not have this issue.

```from transformers.models.llama.modeling_llama import LLaMAModel
LLaMAModel.from_pretrained('llama-7b')
```

throws this:

```
Some weights of the model checkpoint at llama-7b were not used when initializing LLaMAModel: ['lm_head.weight']
- This IS expected if you are initializing LLaMAModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LLaMAModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).```


`LLaMAModel` excludes the LM Head. You want to use `LLaMAForCausalLM` instead.

<img width="977" alt="image" src="https://user-images.githubusercontent.com/106687776/223898571-7866feb7-ba02-4641-be86-c424aef03883.png">

@zphang  getting the following err. any clue?

> `LLaMAModel` excludes the LM Head. You want to use `LLaMAForCausalLM` instead.

So the error is expected when calling this class? I see other models are silencing it with `_keys_to_ignore_on_load_unexpected = [r"decoder\.version", r"lm_head.weight"]` in XXXPreTrainedModel.

I know why I'm calling this class directly, I was just uneasy that I got this error with LLaMA and not with other models (e.g. Bloom), but later I found this difference in `_keys_to_ignore_on_load_unexpected `

I don't think I see that in OPT or GPT-NeoX-20B.

Can the code be used for finetuning? I tried simply to finetune llama-33b using deepspeed on 8x40gA100, but it seems batch-size can only be 1, while in the same ds config and GPU environment, I can finetune opt-iml-33b with batch_size 8. I wonder if it is the code that can not be well used for finetuning using deepspeed?

> <img alt="image" width="977" src="https://user-images.githubusercontent.com/106687776/223898571-7866feb7-ba02-4641-be86-c424aef03883.png">
> 
> @zphang getting the following err. any clue?

Try installing llama requirements packages (Before that, make sure you installed zphang's llama_push repo)
`git clone https://github.com/facebookresearch/llama`
`pip install -r requirements.txt`

> Can the code be used for finetuning? I tried simply to finetune llama-33b using deepspeed on 8x40gA100, but it seems batch-size can only be 1, while in the same ds config and GPU environment, I can finetune opt-iml-33b with batch_size 8. I wonder if it is the code that can not be well used for finetuning using deepspeed?

I have the same problem, on 4*A100 40G, for BLOOMZ 7B can be 8 batch, llama 7B can only be 1batch.

I get different results in the original PR implementation (which had licensing issues) and the current, modified implementation. 

Using the prompt `This is a story about` and

```
output = model.generate(input_ids, do_sample=False)

```
 I get these outputs:

## Original:

> This is a story about how we got here, and the challenges ahead.
The first time I saw a drone was in 2013 when my wife and I were visiting our daughter at college. We went to her dorm room one afternoon to drop off some things she needed for class that day, and as soon as we walked into the lobby, there it was: a quadcopter hovering just above us, its four propellers spinning lazily in the air. It wasn’t doing anything special—it was just flying around on its own, like an insect buzzing around your head. But it made me feel uneasy; I didn’t know what to make of this new technology. The next year, after I had moved back home from Washington, D.C., where I worked for a nonprofit, I started noticing them more often while walking through neighborhoods or hiking in the woods. Sometimes they would be parked

## Current:

> This is a story about a man who was born in the 1920s and lived through the Great Depression, World War II, and the Korean War. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He

Maybe I am doing something wrong, so can someone double check this?

We are noticing that unlike other models this model / tokenizer isn't adding spaces at the beginning of the generation.
Our interface is designed for continuous generation so if a user submits "Programmers are busy debugging the code" we would expect the generation " of the model generation." resulting in "Programmers are busy debugging the code of the model generation."

Instead we are seeing "of the model generation." resulting in "Programmers are busy debugging the codeof the model generation".

Is this to be expected on the latest version of the PR and conversion?

> We are noticing that unlike other models this model / tokenizer isn't adding spaces at the beginning of the generation. Our interface is designed for continuous generation so if a user submits "Programmers are busy debugging the code" we would expect the generation " of the model generation." resulting in "Programmers are busy debugging the code of the model generation."
> 
> Instead we are seeing "of the model generation." resulting in "Programmers are busy debugging the codeof the model generation".
> 
> Is this to be expected on the latest version of the PR and conversion?

This is expected based on the tokenizer provided by Facebook. The eos and bos tokens are literally empty strings. Perhaps for HuggingFace purposes this should be... reconsidered?

> I get different results in the original PR implementation (which had licensing issues) and the current, modified implementation.
> 
> Using the prompt `This is a story about` and
> 
> ```
> output = model.generate(input_ids, do_sample=False)
> ```
> 
> I get these outputs:
> ## Original:
> 
> > This is a story about how we got here, and the challenges ahead.
> > The first time I saw a drone was in 2013 when my wife and I were visiting our daughter at college. We went to her dorm room one afternoon to drop off some things she needed for class that day, and as soon as we walked into the lobby, there it was: a quadcopter hovering just above us, its four propellers spinning lazily in the air. It wasn’t doing anything special—it was just flying around on its own, like an insect buzzing around your head. But it made me feel uneasy; I didn’t know what to make of this new technology. The next year, after I had moved back home from Washington, D.C., where I worked for a nonprofit, I started noticing them more often while walking through neighborhoods or hiking in the woods. Sometimes they would be parked
> 
> ## Current:
> 
> > This is a story about a man who was born in the 1920s and lived through the Great Depression, World War II, and the Korean War. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He was a man who was a part of the greatest generation. He
> 
> Maybe I am doing something wrong, so can someone double check this?

@oobabooga 

What settings are you using to generate the text? I noticed I had to increase my repetition penalty, otherwise I'd see similar results as you. I'm currently getting pretty good results with these settings:

```
temp 0.8
rep_pen 1.18
top_k 67
top_p 0
do_sample false
```

Ok, I have looked more carefully and the biggest difference is that the original implementation didn't add an `bos_token` to the prompt and the current one does.

Trying again but removing the `bos_token` from the current version I get these outputs for the prompt `My name is`

## Original

> My name is Kyle and I am a 20 year old college student. I am a very outgoing person and I love to meet new people. I am very easy to get along with and I am very open minded. I am a very active person and I love to be outdoors. I love to go hiking, camping, and fishing. I am a very easy going person and I love to have fun. I am very respectful and I am very easy to

## Current

> My name is Kyle and I am a 20 year old college student. I am a very outgoing person and I love to meet new people. I am very easy to get along with and I am very open minded. I am a very active person and I love to be outdoors. I love to go hiking, camping, and fishing. I am a very athletic person and I love to play sports. I am a very competitive person and I love to

There is still a difference, but this time it's subtle. This is how I am testing the original implementation:

``` python
import gc
import importlib
from pathlib import Path
import torch
import os
import subprocess

def install(name):
    subprocess.call(['pip', 'uninstall', '-y', 'transformers'])
    subprocess.call(['pip', 'install', name])

install('git+https://github.com/oobabooga/transformers@llama_push')
import transformers

model = transformers.AutoModelForCausalLM.from_pretrained(Path(f"models/llama-7b-original"), torch_dtype=torch.float16).cuda()
tokenizer = transformers.AutoTokenizer.from_pretrained(Path(f"models/llama-7b-original"))

input_ids = tokenizer.encode("My name is", return_tensors='pt').cuda()
print(input_ids)
output = model.generate(input_ids, do_sample=False, max_new_tokens=100)[0]
print(tokenizer.decode(output, skip_special_tokens=True))

```

And the current:

``` python
import gc
import importlib
from pathlib import Path
import torch
import os
import subprocess

def install(name):
    subprocess.call(['pip', 'uninstall', '-y', 'transformers'])
    subprocess.call(['pip', 'install', name])

install('git+https://github.com/zphang/transformers@llama_push')
import transformers

model = transformers.AutoModelForCausalLM.from_pretrained(Path(f"models/llama-7b-current"), torch_dtype=torch.float16).cuda()
tokenizer = transformers.AutoTokenizer.from_pretrained(Path(f"models/llama-7b-current"))

input_ids = tokenizer.encode("My name is", return_tensors='pt').cuda()
input_ids = input_ids[:,1:]
print(input_ids)
output = model.generate(input_ids, do_sample=False, max_new_tokens=100)[0]
print(tokenizer.decode(output, skip_special_tokens=True))

```

@rohvani the point is to use deterministic sampling to make sure that the two implementations yield the exact same results.

> Ok, I have looked more carefully and the biggest difference is that the original implementation didn't add an `bos_token` to the prompt and the current one does.
> 
> Trying again but removing the `bos_token` from the current version I get these outputs for the prompt `My name is`
> 
> ## Original
> > My name is Kyle and I am a 20 year old college student. I am a very outgoing person and I love to meet new people. I am very easy to get along with and I am very open minded. I am a very active person and I love to be outdoors. I love to go hiking, camping, and fishing. I am a very easy going person and I love to have fun. I am very respectful and I am very easy to
> 
> ## Current
> > My name is Kyle and I am a 20 year old college student. I am a very outgoing person and I love to meet new people. I am very easy to get along with and I am very open minded. I am a very active person and I love to be outdoors. I love to go hiking, camping, and fishing. I am a very athletic person and I love to play sports. I am a very competitive person and I love to
> 
> There is still a difference, but this time it's subtle. This is how I am testing the original implementation:
> 
> ```python
> import gc
> import importlib
> from pathlib import Path
> import torch
> import os
> import subprocess
> 
> def install(name):
>     subprocess.call(['pip', 'uninstall', '-y', 'transformers'])
>     subprocess.call(['pip', 'install', name])
> 
> install('git+https://github.com/oobabooga/transformers@llama_push')
> import transformers
> 
> model = transformers.AutoModelForCausalLM.from_pretrained(Path(f"models/llama-7b-original"), torch_dtype=torch.float16).cuda()
> tokenizer = transformers.AutoTokenizer.from_pretrained(Path(f"models/llama-7b-original"))
> 
> input_ids = tokenizer.encode("My name is", return_tensors='pt').cuda()
> print(input_ids)
> output = model.generate(input_ids, do_sample=False, max_new_tokens=100)[0]
> print(tokenizer.decode(output, skip_special_tokens=True))
> ```
> 
> And the current:
> 
> ```python
> import gc
> import importlib
> from pathlib import Path
> import torch
> import os
> import subprocess
> 
> def install(name):
>     subprocess.call(['pip', 'uninstall', '-y', 'transformers'])
>     subprocess.call(['pip', 'install', name])
> 
> install('git+https://github.com/zphang/transformers@llama_push')
> import transformers
> 
> model = transformers.AutoModelForCausalLM.from_pretrained(Path(f"models/llama-7b-current"), torch_dtype=torch.float16).cuda()
> tokenizer = transformers.AutoTokenizer.from_pretrained(Path(f"models/llama-7b-current"))
> 
> input_ids = tokenizer.encode("My name is", return_tensors='pt').cuda()
> input_ids = input_ids[:,1:]
> print(input_ids)
> output = model.generate(input_ids, do_sample=False, max_new_tokens=100)[0]
> print(tokenizer.decode(output, skip_special_tokens=True))
> ```
> 
> @rohvani the point is to use deterministic sampling to make sure that the two implementations yield the exact same results.

We dont want the same results, as the previous implementation was flawed.

LGTM then.

Regarding the above comments: yes, it is important to separate the tokenizer behavior and model behavior.

On the tokenizer: I think you should not rely on the default appending/prepending of EOS/BOS, especially when testing across different models. Different model families have different special token behavior in different cases (e.g. I believe for most cases in generation, you do not want EOS appended, even though this is the default in some models). It would be better to provide add_eos_token / add_bos_token explicitly in your usage.

On the model: Yes, I believe there may still be some subtle differences between the actual model implementation between the old and refactored version. Off the top of my head, the RoPE embeddings and RMSNorms have totally different implementations: they should be mathematically identical, but may be numerically different. I will continue to look into this, but providing more reproducible examples of differences would be extremely helpful.

Thanks all for the feedback! It has been very helpful in helping me track down issues with my implementation.

To come back on the BOS token point, I actually am not certain that is the issue. In models that behave correctly it is the choice of the model to add the space. But it is not guaranteed behavior. For example on a correct model something like "I like to play foot" might generate "ball with my friends" resulting in "I like to play football with my friends" but these models still have the ability to detect when a space should be used. That means that on the existing models this behavior is not hardcoded.

In the earlier mentioned examples the behavior persists on this LLaMA implementation when it does not make sense and there clearly should have been a space. It also means workarounds for the issue are unreliable because if you wish to continue a partial generation it would be undesirable for us to forcefully inject spaces while the AI may have been trying to use one of those combined words like football when it got cut off. 

> We are noticing that unlike other models this model / tokenizer isn't adding spaces at the beginning of the generation. Our interface is designed for continuous generation so if a user submits "Programmers are busy debugging the code" we would expect the generation " of the model generation." resulting in "Programmers are busy debugging the code of the model generation."
> 
> Instead we are seeing "of the model generation." resulting in "Programmers are busy debugging the codeof the model generation".
> 
> Is this to be expected on the latest version of the PR and conversion?

This is a SentencePiece behavior rather than something model specific - it always trims space/linefeed(s) at the beginning of output. Unless I am mistaken, SPM itself must be modified to fix this behavior.

@zphang very interested in this new feature, do you guys have any timeline of merging it into master?

> It also means workarounds for the issue are unreliable because if you wish to continue a partial generation it would be undesirable for us to forcefully inject spaces while the AI may have been trying to use one of those combined words like football when it got cut off.

Hi, just tested this, and I understand what you mean. This is an issue with the tokenizer decoding rather than encoding. As @syskn mentioned, this is an issue with the sentencepiece, not the implementation (e.g. the T5 tokenizer, which also uses SPM, has the same issue). Given that this behavior also exists in T5, a widely used model, I don't think the HF folks are likely to want to have this depart from the expected behavior then. I think this may be something you need to address downstream on your end.

One simple (if manual) fix may be the following:

```python
vocab = tokenizer.convert_ids_to_tokens(range(tokenizer.vocab_size))
NO_SPACE_SET = {
    i
    for i, tok in enumerate(vocab)
    if not tok.startswith("▁")
}

def llama_decode(model_output):
    decoded = tokenizer.decode(model_output)
    if model_output and model_output[0] not in NO_SPACE_SET:
        decoded = " " + decoded
    return decoded

# Prepends space
print(f"'{llama_decode([7199, 1612, 825])}'")
# doesn't prepend space
print(f"'{llama_decode([53, 19, 1633])}'")
```

(Basically tokenizers are weird and difficult.)

@zpang that is unfortunate because that means that the way sentencepiece is handled isn't suitable for the real world use cases of models that complete text. I will then indeed have to do the always add a space workaround which can hinder the experience.

T5 is an entirely different model not suitable for our use case. A model like LLaMA has a real world use case of completing text in a text editor in a seamless experience rather than an input and an output field. While this behavior isn't bad or even desirable in a seperate output field it can really hinder continuous generation if everything is unified in a single editor.

For now ill resort to implementing a workaround like you suggested, but it is good for the wider AI community to knows this kind of behavior on a model intended to complete text hurts real world performance. It would be much better if the model were able to pick a space as a token rather than me having to pick between no spaces or always spaces.

>T5 is an entirely different model not suitable for our use case. A model like LLaMA has a real world use case of completing text in a text editor in a seamless experience rather than an input and an output field.

I understand where you're coming from, but I'm just trying to provide the context of broader language models. Many language models (in the modern usage of the term) aren't centered on generation. This includes broad classes of models like BERT and T5 that aren't used in GPT-like ways.

And on your second note: I'm not sure if this is a fundamental limitation of sentencepiece, or just the chosen implementation (on the SPM side) for decoding tokens. As described above, the information to decode correctly with the space is there, it's just that the decode method doesn't support it. Thinking about it more, I think we could add an option to the tokenizer implementation to enable this behavior -- this would not be that out of place given the existence of other previous hacks like "add prefix space". (Thinking about it even more, maybe this *should* be the default behavior.)

Also, to answer more questions about merging, the HF folks have asked me to ping them when I think this is good to merge. I plan to spend some time looking through the issues raised above this weekend, and pinging them on Monday, and I think I can raise this as an issue as well.

@zphang  I'm not sure what has changed in the your branch. But I'm getting this error 

```ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.```

While trying this code

```
import gc
import importlib
from pathlib import Path
import torch
import os
import subprocess

def install(name):
    subprocess.call(['pip', 'uninstall', '-y', 'transformers'])
    subprocess.call(['pip', 'install', name])

install('git+https://github.com/zphang/transformers@llama_push')
import transformers

model = transformers.AutoModelForCausalLM.from_pretrained(Path(f"models/llama-7b-current"), torch_dtype=torch.float16).cuda()
tokenizer = transformers.AutoTokenizer.from_pretrained(Path(f"models/llama-7b-current"))

input_ids = tokenizer.encode("My name is", return_tensors='pt').cuda()
input_ids = input_ids[:,1:]
print(input_ids)
output = model.generate(input_ids, do_sample=False, max_new_tokens=100)[0]
print(tokenizer.decode(output, skip_special_tokens=True))
```


On the other hand, If I try with the original implementation (with which this PR was started), i'm getting completely gibberish, This is the output for the input prompt `My name is`

```
My name isanto Dresclockulaireɒ повеisterug Clarkjuierslexvir polcibleteiyenibtzewINF CommissionFEzat Metropolasi summ Package Stadiumiarogleorf EnciqueowieщенędzISO mand Fichieravantskystag _ILvueutilsверropSERTгро PitaudiatzATCH midortenscheid Tableisoreicheanguage riencloudflare!("meisterINTlauychcykhausenrameramePlotнат Kontrola LauritoundialذativvorñaccalachJB shootfaretr Epetuenig százonCreate chamber dob▲ascibilcinaglia
``` 

I was using this model hosted on HF 

"decapoda-research/llama-7b-hf"

@zphang  Hi, I got some error with follow code:
```python
import transformers
import torch

device = "cuda:0" if torch.cuda.is_available() else "cpu"

tokenizer = transformers.LLaMATokenizer.from_pretrained("LLaMA-hf/llama-7b")
model = transformers.LLaMAForCausalLM.from_pretrained("LLaMA-hf/llama-7b").to(device)

batch = tokenizer(
    "The primary use of LLaMA is research on large language models, including",
    return_tensors="pt", 
    add_special_tokens=False
)
batch = {k: v.cuda() for k, v in batch.items()}
generated = model.generate(batch["input_ids"], max_length=100)
print(tokenizer.decode(generated[0]))
```
Error:
```bash
File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:228, in LLaMAAttention.forward(self, hidden_states, past_key_value, attention_mask, output_attentions)
    226     kv_seq_len += offset
    227 cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
--> 228 query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, offset=offset)
    229 # [bsz, nh, t, hd]
    231 if past_key_value is not None:
    232     # reuse k, v, self_attention

File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:142, in apply_rotary_pos_emb(q, k, cos, sin, offset)
    140 cos = cos[..., offset : q.shape[-2] + offset, :]
    141 sin = sin[..., offset : q.shape[-2] + offset, :]
--> 142 q_embed = (q * cos) + (rotate_half(q) * sin)
    143 k_embed = (k * cos) + (rotate_half(k) * sin)
    144 return q_embed, k_embed

File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:136, in rotate_half(x)
    134 x1 = x[..., : x.shape[-1] // 2]
    135 x2 = x[..., x.shape[-1] // 2]
--> 136 return torch.cat((-x2, x1), dim=-1)

RuntimeError: Tensors must have same number of dimensions: got 3 and 4
```
transformers version: [a3dfcc02d249cbd14ce9089f57d4040146f3f090](https://github.com/zphang/transformers/commit/a3dfcc02d249cbd14ce9089f57d4040146f3f090)


@mymusise where did you get these models from?

> @mymusise where did you get these models from?

@amrrs  Did you mean the transformers implementation of llama?
Here, the latest code of this PR: https://github.com/zphang/transformers/tree/llama_push


> @zphang Hi, I got some error with follow code:
> 
> ```python
> import transformers
> import torch
> 
> device = "cuda:0" if torch.cuda.is_available() else "cpu"
> 
> tokenizer = transformers.LLaMATokenizer.from_pretrained("LLaMA-hf/llama-7b")
> model = transformers.LLaMAForCausalLM.from_pretrained("LLaMA-hf/llama-7b").to(device)
> 
> batch = tokenizer(
>     "The primary use of LLaMA is research on large language models, including",
>     return_tensors="pt", 
>     add_special_tokens=False
> )
> batch = {k: v.cuda() for k, v in batch.items()}
> generated = model.generate(batch["input_ids"], max_length=100)
> print(tokenizer.decode(generated[0]))
> ```
> 
> Error:
> 
> ```shell
> File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:228, in LLaMAAttention.forward(self, hidden_states, past_key_value, attention_mask, output_attentions)
>     226     kv_seq_len += offset
>     227 cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
> --> 228 query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, offset=offset)
>     229 # [bsz, nh, t, hd]
>     231 if past_key_value is not None:
>     232     # reuse k, v, self_attention
> 
> File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:142, in apply_rotary_pos_emb(q, k, cos, sin, offset)
>     140 cos = cos[..., offset : q.shape[-2] + offset, :]
>     141 sin = sin[..., offset : q.shape[-2] + offset, :]
> --> 142 q_embed = (q * cos) + (rotate_half(q) * sin)
>     143 k_embed = (k * cos) + (rotate_half(k) * sin)
>     144 return q_embed, k_embed
> 
> File ./venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py:136, in rotate_half(x)
>     134 x1 = x[..., : x.shape[-1] // 2]
>     135 x2 = x[..., x.shape[-1] // 2]
> --> 136 return torch.cat((-x2, x1), dim=-1)
> 
> RuntimeError: Tensors must have same number of dimensions: got 3 and 4
> ```
> 
> transformers version: [a3dfcc02d249cbd14ce9089f57d4040146f3f090](https://github.com/zphang/transformers/commit/a3dfcc02d249cbd14ce9089f57d4040146f3f090)

@zphang  Hi, here's a PR for fix this: https://github.com/zphang/transformers/pull/3/files

Looks like this was little accidentally changed in this commit: https://github.com/zphang/transformers/commit/e2faccb96ae76460408f559bdde51dc6dda28847#diff-06392bad3b9e97be9ade60d4ac46f73b6809388f4d507c2ba1384ab872711c51L135


There is some difference between the implementation of rotary position embedding in [PR
](https://github.com/zphang/transformers/blob/a3dfcc02d249cbd14ce9089f57d4040146f3f090/src/transformers/models/llama/modeling_llama.py#L100)  and [llama inference](https://github.com/facebookresearch/llama/blob/57b0eb62de0636e75af471e49e2f1862d908d9d8/llama/model.py#L47)

maybe the problem is in  [this line](https://github.com/zphang/transformers/blob/a3dfcc02d249cbd14ce9089f57d4040146f3f090/src/transformers/models/llama/modeling_llama.py#L135)
change the following into
`x2 = x[..., :x.shape[-1] // 2]`

> 

You can try to install sentencepiece

```
pip install sentencepiece
```

Thanks all, corrected two mistakes from my commit last night. Hopefully this version is close to final.

Inference with this code requires significantly more memory than with Facebook's impl. Not sure if I'm doing something wrong or if it's a real problem.

- Facebook impl: 30B model (re-sharded for world size 2) fits in ~68GB VRAM and can do 1024 sequence length no problem. ~16 tokens per second.
- This impl: 30B model exceeds 96GB VRAM and puts additional 20ish GB on CPU. Fails for 1024 sequence length; can do 128 but very slow. Quantizing brings it down to 34GB but still slower than Facebook.

@haydenshively What code are you using on the Transformers side? Are you sure you are sending `torch_dtype=torch.float16` or `torch_dtype=torch.bfloat16` as it looks like you have a model in FP32 with all the memory used.

Giving a heads up: the HF folks have requested I rename some of the class names to follow full camel-case (e.g. `LlamaModel` rather than `LLaMAModel`, so expect that to kick in in an upcoming commit.