LLaMA

## [](#overview)Overview

The LLaMA model was proposed in [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. It is a collection of foundation language models ranging from 7B to 65B parameters.

The abstract from the paper is the following:

_We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community._

Tips:

-   Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form)
-   After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:

```
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path```

-   After conversion, the model and tokenizer can be loaded via:

```
from transformers import LlamaForCausalLM, LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/output/path")
model = LlamaForCausalLM.from_pretrained("/output/path")```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it’s thus 130GB of RAM needed.

-   The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. “Banana”), the tokenizer does not prepend the prefix space to the string.

This model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## [](#transformers.LlamaConfig)LlamaConfig

### class transformers.LlamaConfig

[](#transformers.LlamaConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/configuration_llama.py#L31)

( vocab\_size = 32000hidden\_size = 4096intermediate\_size = 11008num\_hidden\_layers = 32num\_attention\_heads = 32hidden\_act = 'silu'max\_position\_embeddings = 2048initializer\_range = 0.02rms\_norm\_eps = 1e-06use\_cache = Truepad\_token\_id = 0bos\_token\_id = 1eos\_token\_id = 2tie\_word\_embeddings = False\*\*kwargs )

This is the configuration class to store the configuration of a [LlamaModel](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaModel). It is used to instantiate an LLaMA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the LLaMA-7B.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

```
>>> from transformers import LlamaModel, LlamaConfig

>>> 
>>> configuration = LlamaConfig()

>>> 
>>> model = LlamaModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.LlamaTokenizer)LlamaTokenizer

### class transformers.LlamaTokenizer

[](#transformers.LlamaTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama.py#L49)

( vocab\_fileunk\_token = '<unk>'bos\_token = '
'eos\_token = '
'pad\_token = Nonesp\_model\_kwargs: typing.Union\[typing.Dict\[str, typing.Any\], NoneType\] = Noneadd\_bos\_token = Trueadd\_eos\_token = Falseclean\_up\_tokenization\_spaces = False\*\*kwargs )

Parameters

-   [](#transformers.LlamaTokenizer.vocab_file)**vocab\_file** (`str`) — Path to the vocabulary file.

Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding.

#### build\_inputs\_with\_special\_tokens

[](#transformers.LlamaTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama.py#L178)

( token\_ids\_0token\_ids\_1 = None )

#### get\_special\_tokens\_mask

[](#transformers.LlamaTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama.py#L189)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.LlamaTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.LlamaTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.LlamaTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.LlamaTokenizer.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama.py#L226)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.LlamaTokenizer.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — List of ids.
-   [](#transformers.LlamaTokenizer.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT

sequence pair mask has the following format:

```
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |```

if token\_ids\_1 is None, only returns the first portion of the mask (0s).

#### save\_vocabulary

[](#transformers.LlamaTokenizer.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama.py#L151)

( save\_directoryfilename\_prefix: typing.Optional\[str\] = None ) → `Tuple(str)`

Parameters

-   [](#transformers.LlamaTokenizer.save_vocabulary.save_directory)**save\_directory** (`str`) — The directory in which to save the vocabulary.

Paths to the files saved.

Save the vocabulary and special tokens file to a directory.

## [](#transformers.LlamaTokenizerFast)LlamaTokenizerFast

### class transformers.LlamaTokenizerFast

[](#transformers.LlamaTokenizerFast)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama_fast.py#L37)

( vocab\_file = Nonetokenizer\_file = Noneclean\_up\_tokenization\_spaces = Falseunk\_token = '<unk>'bos\_token = '
'eos\_token = '
'add\_bos\_token = Trueadd\_eos\_token = False\*\*kwargs )

Parameters

-   [](#transformers.LlamaTokenizerFast.vocab_file)**vocab\_file** (`str`) — [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .model extension) that contains the vocabulary necessary to instantiate a tokenizer.
-   [](#transformers.LlamaTokenizerFast.tokenizer_file)**tokenizer\_file** (`str`) — [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer.
-   [](#transformers.LlamaTokenizerFast.clean_up_tokenization_spaces)**clean\_up\_tokenization\_spaces** (`str`, _optional_, defaults to `False`) — Wether to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces.
-   [](#transformers.LlamaTokenizerFast.bos_token)**bos\_token** (`str`, _optional_, defaults to `"
"`) — The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
-   [](#transformers.LlamaTokenizerFast.eos_token)**eos\_token** (`str`, _optional_, defaults to `"
"`) — The end of sequence token.
-   [](#transformers.LlamaTokenizerFast.unk_token)**unk\_token** (`str`, _optional_, defaults to `"<unk>"`) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.

Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding.

This uses notably ByteFallback and no normalization.

```
from transformers import LlamaTokenizerFast

tokenizer = LlaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer")
tokenizer.encode("Hello this is a test")
>>> [1, 15043, 445, 338, 263, 1243]```

This tokenizer inherits from [PreTrainedTokenizerFast](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### build\_inputs\_with\_special\_tokens

[](#transformers.LlamaTokenizerFast.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3080)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.LlamaTokenizerFast.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — The first tokenized sequence.
-   [](#transformers.LlamaTokenizerFast.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — The second tokenized sequence.

The model input with special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

This implementation does not add special tokens and this method should be overridden in a subclass.

#### get\_special\_tokens\_mask

[](#transformers.LlamaTokenizerFast.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3525)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → A list of integers in the range \[0, 1\]

Parameters

-   [](#transformers.LlamaTokenizerFast.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of ids of the first sequence.
-   [](#transformers.LlamaTokenizerFast.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — List of ids of the second sequence.
-   [](#transformers.LlamaTokenizerFast.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

Returns

A list of integers in the range \[0, 1\]

1 for a special token, 0 for a sequence token.

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.LlamaTokenizerFast.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/tokenization_utils_base.py#L3060)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.LlamaTokenizerFast.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — The first tokenized sequence.
-   [](#transformers.LlamaTokenizerFast.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — The second tokenized sequence.

The token type ids.

Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids)

Should be overridden in a subclass if the model has a special way of building those.

#### save\_vocabulary

[](#transformers.LlamaTokenizerFast.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/tokenization_llama_fast.py#L146)

( save\_directory: strfilename\_prefix: typing.Optional\[str\] = None )

## [](#transformers.LlamaModel)LlamaModel

### class transformers.LlamaModel

[](#transformers.LlamaModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L432)

( config: LlamaConfig )

Parameters

-   [](#transformers.LlamaModel.config)**config** ([LlamaConfig](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. config — LlamaConfig

The bare LLaMA Model outputting raw hidden-states without any specific head on top. This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Transformer decoder consisting of _config.num\_hidden\_layers_ layers. Each layer is a `LlamaDecoderLayer`

#### forward

[](#transformers.LlamaModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L483)

( input\_ids: LongTensor = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None )

The [LlamaModel](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

## [](#transformers.LlamaForCausalLM)LlamaForCausalLM

### class transformers.LlamaForCausalLM

[](#transformers.LlamaForCausalLM)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L612)

( config )

#### forward

[](#transformers.LlamaForCausalLM.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L640)

( input\_ids: LongTensor = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.CausalLMOutputWithPast](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or `tuple(torch.FloatTensor)`

The [LlamaForCausalLM](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaForCausalLM) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, LlamaForCausalLM

>>> model = LlamaForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
>>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)

>>> prompt = "Hey, are you consciours? Can you talk to me?"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> 
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"Hey, are you consciours? Can you talk to me?\nI'm not consciours, but I can talk to you."```

## [](#transformers.LlamaForSequenceClassification)LlamaForSequenceClassification

### class transformers.LlamaForSequenceClassification

[](#transformers.LlamaForSequenceClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L781)

( config )

Parameters

-   [](#transformers.LlamaForSequenceClassification.config)**config** ([LlamaConfig](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The LLaMa Model transformer with a sequence classification head on top (linear layer).

[LlamaForSequenceClassification](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaForSequenceClassification) uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do.

Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch).

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.LlamaForSequenceClassification.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L799)

( input\_ids: LongTensor = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None )

The [LlamaForSequenceClassification](/docs/transformers/v4.30.0/en/model_doc/llama#transformers.LlamaForSequenceClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.