I am trying to do llama model inference:
Local Inference
Hugging face setup
Important Note: Before running the inference, you'll need your Hugging Face access token, which you can get at your Settings page here. Then run huggingface-cli login and copy and paste your Hugging Face access token to complete the login to make sure the scripts can download Hugging Face models if needed.

Multimodal Inference
For Multi-Modal inference we have added multi_modal_infer.py which uses the transformers library.

The way to run this would be:

python multi_modal_infer.py --image_path PATH_TO_IMAGE --prompt_text "Describe this image" --temperature 0.5 --top_p 0.8 --model_name "meta-llama/Llama-3.2-11B-Vision-Instruct"
Multi-modal Inferencing Using gradio UI for inferencing
For multi-modal inferencing using gradio UI we have added multi_modal_infer_gradio_UI.py which used gradio and transformers library.

Steps to Run
The way to run this would be:

Ensure having proper access to llama 3.2 vision models, then run the command given below
python multi_modal_infer_gradio_UI.py --hf_token
Text-only Inference
For local inference we have provided an inference script. Depending on the type of finetuning performed during training the inference script takes different arguments.

To finetune all model parameters the output dir of the training has to be given as --model_name argument. In the case of a parameter efficient method like lora the base model has to be given as --model_name and the output dir of the training has to be given as --peft_model argument. Additionally, a prompt for the model in the form of a text file has to be provided. The prompt file can either be piped through standard input or given as --prompt_file parameter.

Content Safety The inference script also supports safety checks for both user prompt and model outputs. In particular, we use two packages, AuditNLG and Azure content safety.

Note If using Azure content Safety, please make sure to get the endpoint and API key as described here and add them as the following environment variables,CONTENT_SAFETY_ENDPOINT and CONTENT_SAFETY_KEY.

Examples:

Full finetuning of all parameters
cat | python inference.py --model_name <training_config.output_dir> --use_auditnlg

PEFT method
cat | python inference.py --model_name <training_config.model_name> --peft_model <training_config.output_dir> --use_auditnlg

prompt as parameter
python inference.py --model_name <training_config.output_dir> --prompt_file --use_auditnlg
The folder contains test prompts for summarization use-case:

samsum_prompt.txt
...
Note on Llama version < 3.1 The default padding token in HuggingFace Tokenizer is None. To use padding the padding token needs to be added as a special token to the tokenizer, which in this case requires to resize the token_embeddings as shown below:

tokenizer.add_special_tokens(
{


        "pad_token": "<PAD>",
    }
)
model.resize_token_embeddings(model.config.vocab_size + 1)
Padding would be required for batched inference. In this example, batch size = 1 so essentially padding is not required. However, we added the code pointer as an example in case of batch inference. For Llama version 3.1 use the special token 
 (128004) for padding.

Chat completion
The inference folder also includes a chat completion example, that adds built-in safety features in fine-tuned models to the prompt tokens. To run the example:

python chat_completion/chat_completion.py --model_name "PATH/TO/MODEL/7B/" --prompt_file chat_completion/chats.json --quantization 8bit --use_auditnlg
Flash Attention and Xformer Memory Efficient Kernels
Setting use_fast_kernels will enable using of Flash Attention or Xformer memory-efficient kernels based on the hardware being used. This would speed up inference when used for batched inputs. This has been enabled in optimum library from HuggingFace as a one-liner API, please read more here.

python chat_completion/chat_completion.py --model_name "PATH/TO/MODEL/7B/" --prompt_file chat_completion/chats.json --quantization 8bit --use_auditnlg --use_fast_kernels

python inference.py --model_name <training_config.output_dir> --peft_model <training_config.output_dir> --prompt_file --use_auditnlg --use_fast_kernels
Inference with FSDP checkpoints
In case you have fine-tuned your model with pure FSDP and saved the checkpoints with "SHARDED_STATE_DICT" as shown here, you can use this converter script to convert the FSDP Sharded checkpoints into HuggingFace checkpoints. This enables you to use the inference script normally as mentioned above. To convert the checkpoint use the following command:

This is helpful if you have fine-tuned you model using FSDP only as follows:

torchrun --nnodes 1 --nproc_per_node 8 recipes/quickstart/finetuning/finetuning.py --enable_fsdp --model_name /path_of_model_folder/7B --dist_checkpoint_root_folder model_checkpoints --dist_checkpoint_folder fine-tuned --fsdp_config.pure_bf16
Then convert your FSDP checkpoint to HuggingFace checkpoints using:

python -m llama_recipes.inference.checkpoint_converter_fsdp_hf --fsdp_checkpoint_path PATH/to/FSDP/Checkpoints --consolidated_model_path PATH/to/save/checkpoints --HF_model_path_or_name PATH/or/HF/model_name

--HF_model_path_or_name specifies the HF Llama model name or path where it has config.json and tokenizer.json
By default, training parameter are saved in train_params.yaml in the path where FSDP checkpoints are saved, in the converter script we frist try to find the HugingFace model name used in the fine-tuning to load the model with configs from there, if not found user need to provide it.

Then run inference using:

python inference.py --model_name <training_config.output_dir> --prompt_file

When I run the code:
python chat_completion/chat_completion.py --model_name "PATH/TO/MODEL/7B/" --prompt_file chat_completion/chats.json --quantization 8bit --use_auditnlg
Output: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating

The chat_completion.py :

Copyright (c) Meta Platforms, Inc. and affiliates.
This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
import fire
import json
import os
import sys

import torch
from transformers import AutoTokenizer

from llama_recipes.inference.chat_utils import read_dialogs_from_file
from llama_recipes.inference.model_utils import load_model, load_peft_model
from llama_recipes.inference.safety_utils import get_safety_checker
from accelerate.utils import is_xpu_available

def main(
model_name,
peft_model: str=None,
quantization: str = None, # Options: 4bit, 8bit
max_new_tokens =256, #The maximum numbers of tokens to generate
min_new_tokens:int=0, #The minimum numbers of tokens to generate
prompt_file: str=None,
seed: int=42, #seed value for reproducibility
safety_score_threshold: float=0.5,
do_sample: bool=True, #Whether or not to use sampling ; use greedy decoding otherwise.
use_cache: bool=True, #[optional] Whether or not the model should use the past last key/values attentions Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding.
top_p: float=1.0, # [optional] If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation.
temperature: float=1.0, # [optional] The value used to modulate the next token probabilities.
top_k: int=50, # [optional] The number of highest probability vocabulary tokens to keep for top-k-filtering.
repetition_penalty: float=1.0, #The parameter for repetition penalty. 1.0 means no penalty.
length_penalty: int=1, #[optional] Exponential penalty to the length that is used with beam-based generation.
enable_azure_content_safety: bool=False, # Enable safety check with Azure content safety api
enable_sensitive_topics: bool=False, # Enable check for sensitive topics using AuditNLG APIs
enable_saleforce_content_safety: bool=True, # Enable safety check woth Saleforce safety flan t5
use_fast_kernels: bool = False, # Enable using SDPA from PyTorch Accelerated Transformers, make use Flash Attention and Xformer memory-efficient kernels
enable_llamaguard_content_safety: bool = False,
**kwargs
):
if prompt_file is not None:
assert os.path.exists(
prompt_file
), f"Provided Prompt file does not exist {prompt_file}"


    dialogs= read_dialogs_from_file(prompt_file)

elif not sys.stdin.isatty():
    dialogs = "\n".join(sys.stdin.readlines())
    try:
        dialogs = json.loads(dialogs)
    except:
        print("Could not parse json from stdin. Please provide a json file with the user prompts. Exiting.")
        sys.exit(1)
else:
    print("No user prompt provided. Exiting.")
    sys.exit(1)

print(f"User dialogs:\n{dialogs}")
print("\n==================================\n")

# Set the seeds for reproducibility
if is_xpu_available():
    torch.xpu.manual_seed(seed)
else:
    torch.cuda.manual_seed(seed)
torch.manual_seed(seed)

model = load_model(model_name, quantization, use_fast_kernels, **kwargs)
if peft_model:
    model = load_peft_model(model, peft_model)

tokenizer = AutoTokenizer.from_pretrained(model_name)

chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs]

with torch.no_grad():
    for idx, chat in enumerate(chats):
        safety_checker = get_safety_checker(enable_azure_content_safety,
                                    enable_sensitive_topics,
                                    enable_saleforce_content_safety,
                                    enable_llamaguard_content_safety,
                                    )
        # Safety check of the user prompt
        safety_results = [check(dialogs[idx][0]["content"]) for check in safety_checker]
        are_safe = all([r[1] for r in safety_results])
        if are_safe:
            print(f"User prompt deemed safe.")
            print("User prompt:\n", dialogs[idx][0]["content"])
            print("\n==================================\n")
        else:
            print("User prompt deemed unsafe.")
            for method, is_safe, report in safety_results:
                if not is_safe:
                    print(method)
                    print(report)
            print("Skipping the inferece as the prompt is not safe.")
            sys.exit(1)  # Exit the program with an error status
        tokens= torch.tensor(chat).long()
        tokens= tokens.unsqueeze(0)
        attention_mask = torch.ones_like(tokens)
        if is_xpu_available():
            tokens= tokens.to("xpu:0")
        else:
            tokens= tokens.to("cuda:0")
        outputs = model.generate(
            input_ids=tokens,
            attention_mask=attention_mask,
            max_new_tokens=max_new_tokens,
            do_sample=do_sample,
            top_p=top_p,
            temperature=temperature,
            use_cache=use_cache,
            top_k=top_k,
            repetition_penalty=repetition_penalty,
            length_penalty=length_penalty,
            **kwargs
        )

        output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

        # Safety check of the model output
        safety_results = [check(output_text) for check in safety_checker]
        are_safe = all([r[1] for r in safety_results])
        if are_safe:
            print("User input and model output deemed safe.")
            print(f"Model output:\n{output_text}")
            print("\n==================================\n")

        else:
            print("Model output deemed unsafe.")
            for method, is_safe, report in safety_results:
                if not is_safe:
                    print(method)
                    print(report)
if name == "main":
fire.Fire(main)
What happen?