philipp-zettl/vrom-hf-docs huggingface dataset:
Start by running [accelerate config](https://hf.co/docs/accelerate/main/en/package_reference/cli#accelerate-config) in the command line to answer a series of prompts about your training system. This creates and saves a configuration file to help Accelerate correctly set up training based on your setup.

```bash
accelerate config
```

Depending on your setup and the answers you provide, an example configuration file for distributing training with FSDP on one machine with two GPUs may look like the following.
```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```
## Trainer

Pass the path to the saved configuration file to [TrainingArguments](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.TrainingArguments), and from there, pass your [TrainingArguments](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.TrainingArguments) to [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer).
```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    fsdp_config="path/to/fsdp_config",
    fsdp="full_shard",
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    processing_class=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()
```
## Native PyTorch

Accelerate can also be added to any PyTorch training loop to enable distributed training. The [Accelerator](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/accelerator#accelerate.Accelerator) is the main entry point for adapting your PyTorch code to work with Accelerate. It automatically detects your distributed training setup and initializes all the necessary components for training. You don't need to explicitly place your model on a device because [Accelerator](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/accelerator#accelerate.Accelerator) knows which device to move your model to.

```py
from accelerate import Accelerator

accelerator = Accelerator()
device = accelerator.device
```
All PyTorch objects (model, optimizer, scheduler, dataloaders) should be passed to the [prepare](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/accelerator#accelerate.Accelerator.prepare) method now. This method moves your model to the appropriate device or devices, adapts the optimizer and scheduler to use [AcceleratedOptimizer](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/torch_wrappers#accelerate.optimizer.AcceleratedOptimizer) and [AcceleratedScheduler](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/torch_wrappers#accelerate.scheduler.AcceleratedScheduler), and creates a new shardable dataloader.

```py
train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
    train_dataloader, eval_dataloader, model, optimizer
)
```
Replace `loss.backward` in your training loop with Accelerates [backward](https://huggingface.co/docs/accelerate/v1.13.0/en/package_reference/accelerator#accelerate.Accelerator.backward) method to scale the gradients and determine the appropriate `backward` method to use depending on your framework (for example, DeepSpeed or Megatron).

```py
for epoch in range(num_epochs):
    for batch in train_dataloader:
        outputs = model(**batch)
        loss = outputs.loss
        accelerator.backward(loss)
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)
```

Combine everything into a function and make it callable as a script.
```py
from accelerate import Accelerator
  
def main():
  accelerator = Accelerator()

  model, optimizer, training_dataloader, scheduler = accelerator.prepare(
      model, optimizer, training_dataloader, scheduler
  )

  for batch in training_dataloader:
      optimizer.zero_grad()
      inputs, targets = batch
      outputs = model(inputs)
      loss = loss_function(outputs, targets)
      accelerator.backward(loss)
      optimizer.step()
      scheduler.step()

if __name__ == "__main__":
    main()
```

From the command line, call [accelerate launch](https://hf.co/docs/accelerate/main/en/package_reference/cli#accelerate-launch) to run your training script. Any additional arguments or parameters can be passed here as well.

To launch your training script on two GPUs, add the `--num_processes` argument.

```bash
accelerate launch --num_processes=2 your_script.py
```

Refer to the [Launching Accelerate scripts](https://hf.co/docs/accelerate/main/en/basic_tutorials/launch) for more details.
# Parameter-efficient fine-tuning

[Parameter-efficient fine-tuning (PEFT)](https://huggingface.co/docs/peft/index) methods only fine-tune a small number of extra model parameters (adapters) on top of a pretrained model. Because only adapter parameters are updated, the optimizer tracks far fewer gradients and states, reducing memory usage significantly. Adapters are lightweight, making them convenient to share, store, and load.
Transformers integrates directly with the PEFT library through [PeftAdapterMixin](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin), added to all [PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel) classes. You can load, add, train, switch, and delete adapters without wrapping your model in a separate `PeftModel`. All non-prompt-learning PEFT methods are supported (LoRA, IA3, AdaLoRA). Prompt-based methods like prompt tuning and prefix tuning require using the [PEFT library](https://huggingface.co/docs/peft/index) directly.

Install PEFT to get started. The integration requires `peft >= 0.18.0`.

```shell
pip install -U peft
```
## Add an adapter

Create a PEFT config, like `LoraConfig` for example, and attach it to a model with [add_adapter()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.add_adapter).

```py
from peft import LoraConfig, TaskType
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b")

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
)

model.add_adapter(lora_config, adapter_name="my_adapter")
```
### Fully fine-tuning specific layers

To train additional modules alongside an adapter (for example, the language model head), specify them in `modules_to_save`. `modules_to_save` specifies layers that are fully fine-tuned alongside the adapter, so *all* of their parameters are updated. This is useful when certain layers need updates, for example the language model head (`lm_head`), when adapting a causal LM for sequence classification.

```py
lora_config = LoraConfig(
    modules_to_save=["lm_head"],
    ...
)
model.add_adapter(lora_config)
```
### Choosing which layers to adapt

For common architectures (Llama, Gemma, Qwen, etc.), PEFT has predefined default targets (like `q_proj` and `v_proj`), so you don't need to specify `target_modules`. If you want to target different layers, or the model doesn't have predefined targets, pass `target_modules` explicitly as a list of module names or a regex pattern.

```py
lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    ...
)
model.add_adapter(lora_config)
```
## Training

Pass the model with an attached adapter to [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) and call [train()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.train). [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) only updates the adapter parameters (those with `requires_grad=True`) because the base model is frozen.

```py
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
)

trainer.train()
```

During training, [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) checkpoints contain only the adapter weights (`adapter_model.safetensors`) and configuration (`adapter_config.json`), keeping checkpoints small. The base model isn't included.
### Resuming from a checkpoint

[Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) automatically detects adapter checkpoints when resuming. [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) scans the checkpoint directory for subdirectories containing adapter weights and reloads each adapter with the correct trainable state.

```py
trainer.train(resume_from_checkpoint="./output/checkpoint-1000")
```
### Distributed training

PEFT adapters work with distributed training out of the box.

For ZeRO-3, [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) passes `exclude_frozen_parameters=True` when saving checkpoints with a PEFT model. Frozen base model weights are skipped. Only the trainable adapter parameters are saved, reducing checkpoint size and save time.

For FSDP, [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) updates the FSDP auto-wrap policy to correctly handle LoRA layers. For QLoRA (quantized base model + LoRA), [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) also adjusts the mixed precision policy to match the quantization storage dtype.
## Loading an adapter

To load an adapter, the Hub repository or local directory must contain an `adapter_config.json` file and the adapter weights.

[from_pretrained()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) automatically detects adapters. When it finds an `adapter_config.json`, it reads the `base_model_name_or_path` field to load the correct base model, then loads the adapter on top.

```py
from transformers import AutoModelForCausalLM
# Automatically loads the base model and attaches the adapter
model = AutoModelForCausalLM.from_pretrained("klcsp/gemma7b-lora-alpaca-11-v1")
```

To load an adapter onto an existing model, use [load_adapter()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.load_adapter).

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("google/gemma-7b")
model.load_adapter("klcsp/gemma7b-lora-alpaca-11-v1")
```

For large models, load a quantized version in 8-bit or 4-bit precision with [bitsandbytes](./quantization/bitsandbytes) to save memory. Add `device_map="auto"` to distribute the model across available hardware.

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "klcsp/gemma7b-lora-alpaca-11-v1",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="auto",
)
```
## Managing multiple adapters

A model can hold multiple adapters at once. Add adapters with unique names, and switch between them as needed.

```py
from peft import LoraConfig

model.add_adapter(LoraConfig(r=8, lora_alpha=32), adapter_name="adapter_1")
model.add_adapter(LoraConfig(r=16, lora_alpha=64), adapter_name="adapter_2")
```

Use [set_adapter()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.set_adapter) to activate a specific adapter. The other adapters are disabled but remain in memory.

```py
model.set_adapter("adapter_2")
```

[enable_adapters()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.enable_adapters) enables all attached adapters, and [disable_adapters()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.disable_adapters) disables all of them.

```py
## Hotswapping adapters

Loading a new adapter each time you serve a request allocates new memory. If the model is compiled with `torch.compile`, each new adapter triggers recompilation. Hotswapping replaces adapter weights in-place, avoiding both issues. Only LoRA adapters are supported.

Pass `hotswap=True` when loading a LoRA adapter to swap its weights into an existing adapter slot. Set `adapter_name` to the name of the adapter to replace (`"default"` is the default adapter name).

```py
model = AutoModel.from_pretrained(...)
### torch.compile

For compiled models, call [enable_peft_hotswap()](/docs/transformers/v5.6.2/en/main_classes/peft#transformers.integrations.PeftAdapterMixin.enable_peft_hotswap) *before* loading the first adapter and before compiling.

```py
model = AutoModel.from_pretrained(...)
max_rank = ...  # highest rank among all LoRAs you'll load
model.enable_peft_hotswap(target_rank=max_rank)
model.load_adapter(adapter_path_1, adapter_name="default")
model = torch.compile(model, ...)
output_1 = model(...)
# Hotswap without recompilation
model.load_adapter(adapter_path_2, adapter_name="default")
output_2 = model(...)
```

The `target_rank` argument sets the maximum rank among all LoRA adapters you'll load. If you have adapters with rank 8 and rank 16, pass `target_rank=16`. The default is 128.

After calling `enable_peft_hotswap`, all subsequent `load_adapter` calls hotswap by default. Pass `hotswap=False` explicitly to disable hotswapping.

Recompilation may still occur if the hotswapped adapter targets more layers than the initial adapter. Load the adapter that targets the most layers first to avoid recompilation.

> [!TIP]
> Wrap your code in `with torch._dynamo.config.patch(error_on_recompile=True)` to detect unexpected recompilation. If you detect recompilation despite following the steps above, open an issue with [PEFT](https://github.com/huggingface/peft/issues) with a reproducible example.
## Next steps

- The PEFT [documentation](https://huggingface.co/docs/peft/index) covers the full range of PEFT methods and options.
- The PEFT [hotswapping reference](https://huggingface.co/docs/peft/main/en/package_reference/hotswap#peft.utils.hotswap.hotswap_adapter) details limitations and edge cases.
- A [blog post](https://huggingface.co/blog/lora-fast) benchmarks how `torch.compile` with hotswapping improves runtime.
## Set up

To share a model to the Hub, you need a Hugging Face [account](https://hf.co/join). Create a [User Access Token](https://hf.co/docs/hub/security-tokens#user-access-tokens) (stored in the [cache](./installation#cache-directory) by default) and login to your account from either the command line or notebook.

```bash
hf auth login
```

```py
from huggingface_hub import notebook_login

notebook_login()
```
## Repository features

Each model repository features versioning, commit history, and diff visualization.

Versioning is based on [Git](https://git-scm.com/) and [Git Large File Storage (LFS)](https://git-lfs.github.com/), and it enables revisions, a way to specify a model version with a commit hash, tag or branch.

For example, use the `revision` parameter in [from_pretrained()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) to load a specific model version from a commit hash.

```py
model = AutoModel.from_pretrained(
    "julien-c/EsperBERTo-small", revision="4c77982"
)
```

Model repositories also support [gating](https://hf.co/docs/hub/models-gated) to control who can access a model. Gating is common for allowing a select group of users to preview a research model before it's made public.

A model repository also includes an inference [widget](https://hf.co/docs/hub/models-widgets) for users to directly interact with a model on the Hub.
## Uploading a model

There are several ways to upload a model to the Hub depending on your workflow preference. You can push a model with [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer), call [push_to_hub()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.utils.PushToHubMixin.push_to_hub) directly on a model, or use the Hub web interface.
### Trainer

[Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) can push a model directly to the Hub after training. Set `push_to_hub=True` in [TrainingArguments](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.TrainingArguments) and pass it to [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer). Once training is complete, call [push_to_hub()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.push_to_hub) to upload the model.

[push_to_hub()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.push_to_hub) automatically adds useful information like training hyperparameters and results to the model card.
```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.push_to_hub()
```
### PushToHubMixin

The [PushToHubMixin](/docs/transformers/v5.6.2/en/main_classes/model#transformers.utils.PushToHubMixin) provides functionality for pushing a model or tokenizer to the Hub.

Call [push_to_hub()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.utils.PushToHubMixin.push_to_hub) directly on a model to upload it to the Hub. It creates a repository under your namespace with the model name specified in [push_to_hub()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.utils.PushToHubMixin.push_to_hub).

```py
model.push_to_hub("my-awesome-model")
```

Other objects like a tokenizer are also pushed to the Hub in the same way.

```py
tokenizer.push_to_hub("my-awesome-model")
```

Your Hugging Face profile should now display the newly created model repository. Navigate to the **Files** tab to see all the uploaded files.

Refer to the [Upload files to the Hub](https://hf.co/docs/hub/how-to-upstream) guide for more information about pushing files to the Hub.
### Hub web interface

The Hub web interface is a no-code approach for uploading a model.

1. Create a new repository by selecting [**New Model**](https://huggingface.co/new).

    

Add some information about your model:

- Select the **owner** of the repository. This can be yourself or any of the organizations you belong to.
- Pick a name for your model, which will also be the repository name.
- Choose whether your model is public or private.
- Set the license usage.

2. Click on **Create model** to create the model repository.

3. Select the **Files** tab and click on the **Add file** button to drag-and-drop a file to your repository. Add a commit message and click on **Commit changes to main** to commit the file.
## Model card

[Model cards](https://hf.co/docs/hub/model-cards#model-cards) inform users about a models performance, limitations, potential biases, and ethical considerations. It is highly recommended to add a model card to your repository!

A model card is a `README.md` file in your repository. Add this file by:

- manually creating and uploading a `README.md` file
- clicking on the **Edit model card** button in the repository

Take a look at the Llama 3.1 [model card](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) for an example of what to include on a model card.

Learn more about other model card metadata (carbon emissions, license, link to paper, etc.) available in the [Model Cards](https://hf.co/docs/hub/model-cards#model-cards) guide.
# Text generation

Text generation is the most popular application for large language models (LLMs). A LLM is trained to generate the next word (token) given some initial text (prompt) along with its own generated outputs up to a predefined length or when it reaches an end-of-sequence (`EOS`) token.

In Transformers, the [generate()](/docs/transformers/v5.6.2/en/main_classes/text_generation#transformers.GenerationMixin.generate) API handles text generation, and it is available for all models with generative capabilities. This guide will show you the basics of text generation with [generate()](/docs/transformers/v5.6.2/en/main_classes/text_generation#transformers.GenerationMixin.generate) and some common pitfalls to avoid.

> [!TIP]
> For the following commands, please make sure [`transformers serve` is running](https://huggingface.co/docs/transformers/main/en/serving).
>
> ```shell
> transformers chat Qwen/Qwen2.5-0.5B-Instruct
> ```
## Default generate

Before you begin, it's helpful to install [bitsandbytes](https://hf.co/docs/bitsandbytes/index) to quantize really large models to reduce their memory usage.

```bash
!pip install -U transformers bitsandbytes
```

Bitsandbytes supports multiple backends in addition to CUDA-based GPUs. Refer to the multi-backend installation [guide](https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend) to learn more.

Load a LLM with [from_pretrained()](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) and add the following two parameters to reduce the memory requirements.
- `device_map="auto"` enables Accelerates' [Big Model Inference](./models#big-model-inference) feature for automatically initiating the model skeleton and loading and dispatching the model weights across all available devices, starting with the fastest device (GPU).
- `quantization_config` is a configuration object that defines the quantization settings. This examples uses bitsandbytes as the quantization backend (see the [Quantization](./quantization/overview) section for more available backends) and it loads the model in [4-bits](./quantization/bitsandbytes).

```py
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto", quantization_config=quantization_config)
```
Tokenize your input, and set the `padding_side()` parameter to `"left"` because a LLM is not trained to continue generation from padding tokens. The tokenizer returns the input ids and attention mask.

> [!TIP]
> Process more than one prompt at a time by passing a list of strings to the tokenizer. Batch the inputs to improve throughput at a small cost to latency and memory.

```py
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to(model.device)
```

Pass the inputs to [generate()](/docs/transformers/v5.6.2/en/main_classes/text_generation#transformers.GenerationMixin.generate) to generate tokens, and [batch_decode()](/docs/transformers/v5.6.2/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.batch_decode) the generated tokens back to text.