Repository: huggingface/trl (language: python)

Issue:
AttributeError: property 'tokenizer' of 'DPOTrainer' object has no setter
### System Info

- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.11.9
- PyTorch version: 2.4.1
- CUDA device: NVIDIA H100 80GB HBM3
- Transformers version: 4.46.0.dev0
- Accelerate version: 0.34.2
- Accelerate config: not found
- Datasets version: 3.0.0
- HF Hub version: 0.24.7
- TRL version: 0.12.0.dev0+07cebf3
- bitsandbytes version: 0.41.1
- DeepSpeed version: 0.15.1
- Diffusers version: 0.30.3
- Liger-Kernel version: 0.3.0
- LLM-Blender version: 0.0.2
- OpenAI version: 1.46.0
- PEFT version: 0.12.0

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset
import tempfile
from trl import DPOTrainer, DPOConfig

model_id = "HuggingFaceM4/tiny-random-LlamaForCausalLM"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

with tempfile.TemporaryDirectory() as tmp_dir:
    training_args = DPOConfig(output_dir=tmp_dir)
    dummy_dataset = load_dataset("trl-internal-testing/zen", "standard_preference")
    trainer = DPOTrainer(model=model, args=training_args, tokenizer=tokenizer, train_dataset=dummy_dataset["train"])
```


```
[2024-10-03 09:03:00,224] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
/fsx/qgallouedec/transformers/src/transformers/generation/configuration_utils.py:579: UserWarning: `pad_token_id` should be positive but got -1. This will cause errors when batch generating, if there is padding. Please set `pad_token_id` explicitly as `model.generation_config.pad_token_id=PAD_TOKEN_ID` to avoid errors in generation
  warnings.warn(
Traceback (most recent call last):
  File "/fsx/qgallouedec/transformers/../trl/dfg.py", line 13, in <module>
    trainer = DPOTrainer(model=model, args=training_args, tokenizer=tokenizer, train_dataset=dummy_dataset["train"])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.11/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/trl/trl/trainer/dpo_trainer.py", line 635, in __init__
    self.tokenizer = tokenizer
    ^^^^^^^^^^^^^^
AttributeError: property 'tokenizer' of 'DPOTrainer' object has no setter
```



### Expected behavior

To work, like before.


Discussion / hints:
Origin of error, this change: https://github.com/huggingface/transformers/pull/32385
`git bisect` is a wonderfull tool.
This bug is linked to the fact that `tokenizer` will no longer be an argument of trainer, but instead, `processing_class`.

Suggested migration plan:
- Do the same change, eg;
  ```diff
    trainer = RewardTrainer(
        model=model,
        args=training_args,
  -     tokenizer=tokenizer,
  +     processing_class=tokenizer,
        train_dataset=dataset,
        peft_config=peft_config,
    )
  ```
- Ensure backward compatibility only for `SFTTrainer` and `DPOTrainer` via:

```python
def __init__(
    ...
        tokenizer: Optional[PreTrainedTokenizerBase] = None,
        processing_class: Optional[
            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
        ] = None,
    ...
):
    if tokenizer is not None:
      if processing_class is not None:
          raise ValueError(
              "You cannot specify both `tokenizer` and `processing_class` at the same time. Please use `processing_class`."
          )
      warnings.warn(
          "`tokenizer` is now deprecated and will be removed in the future, please use `processing_class` instead.",
          FutureWarning,
      )
      processing_class = tokenizer
```

yes looks like a good solution

Fix patch:
diff --git a/README.md b/README.md
index c692fef0a3..96e07c84e1 100644
--- a/README.md
+++ b/README.md
@@ -133,7 +133,7 @@ training_args = RewardConfig(output_dir="Qwen2.5-0.5B-Reward", per_device_train_
 trainer = RewardTrainer(
     args=training_args,
     model=model,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     train_dataset=dataset,
 )
 trainer.train()
@@ -166,7 +166,7 @@ dataset = dataset.map(lambda x: tokenizer(x["prompt"]), remove_columns="prompt")
 training_args = RLOOConfig(output_dir="Qwen2.5-0.5B-RL")
 trainer = RLOOTrainer(
     config=training_args,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     policy=policy,
     ref_policy=ref_policy,
     reward_model=reward_model,
@@ -189,7 +189,7 @@ model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
 tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
 dataset = load_dataset("trl-lib/Capybara-Preferences", split="train")
 training_args = DPOConfig(output_dir="Qwen2.5-0.5B-DPO")
-trainer = DPOTrainer(model=model, args=training_args, train_dataset=dataset, tokenizer=tokenizer)
+trainer = DPOTrainer(model=model, args=training_args, train_dataset=dataset, processing_class=tokenizer)
 trainer.train()
 ```
 
diff --git a/docs/source/bco_trainer.mdx b/docs/source/bco_trainer.mdx
index c200d91cbc..035b2d1d1d 100644
--- a/docs/source/bco_trainer.mdx
+++ b/docs/source/bco_trainer.mdx
@@ -32,7 +32,7 @@ bco_trainer = BCOTrainer(
     model_ref,
     args=training_args,
     train_dataset=train_dataset,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
 )
 ```
 After this one can then call:
@@ -75,7 +75,7 @@ bco_trainer = BCOTrainer(
     model_ref,
     args=training_args,
     train_dataset=train_dataset,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     embedding_func=embedding_func,
     embedding_tokenizer=self.embedding_tokenizer,
 )
diff --git a/docs/source/cpo_trainer.mdx b/docs/source/cpo_trainer.mdx
index 6ebb3b84ca..7d8d1511ab 100644
--- a/docs/source/cpo_trainer.mdx
+++ b/docs/source/cpo_trainer.mdx
@@ -32,7 +32,7 @@ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
 train_dataset = load_dataset("trl-lib/Capybara-Preferences", split="train")
 
 training_args = CPOConfig(output_dir="Qwen2-0.5B-CPO", logging_steps=10)
-trainer = CPOTrainer(model=model, args=training_args, tokenizer=tokenizer, train_dataset=train_dataset)
+trainer = CPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)
 trainer.train()
 ```
 
diff --git a/docs/source/dpo_trainer.mdx b/docs/source/dpo_trainer.mdx
index 01f3b3e097..54377f246a 100644
--- a/docs/source/dpo_trainer.mdx
+++ b/docs/source/dpo_trainer.mdx
@@ -47,7 +47,7 @@ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
 train_dataset = load_dataset("trl-lib/Capybara-Preferences", split="train")
 
 training_args = DPOConfig(output_dir="Qwen2-0.5B-DPO", logging_steps=10)
-trainer = DPOTrainer(model=model, args=training_args, tokenizer=tokenizer, train_dataset=train_dataset)
+trainer = DPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)
 trainer.train()
 ```
 
@@ -100,8 +100,8 @@ Additionally, unlike standard text-based models where a `tokenizer` is used, for
       model,
       args=training_args,
       train_dataset=train_dataset,
--     tokenizer=tokenizer,
-+     tokenizer=processor,
+-     processing_class=tokenizer,
++     processing_class=processor,
 )
 ```
 
@@ -194,7 +194,7 @@ First install `unsloth` according to the [official documentation](https://github
 
 - training_args = DPOConfig(output_dir="Qwen2-0.5B-DPO", logging_steps=10)
 + training_args = DPOConfig(output_dir="Qwen2-0.5B-DPO", logging_steps=10, bf16=True)
-  trainer = DPOTrainer(model=model, args=training_args, tokenizer=tokenizer, train_dataset=train_dataset)
+  trainer = DPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)
   trainer.train()
 
 ```
diff --git a/docs/source/gkd_trainer.md b/docs/source/gkd_trainer.md
index 7e5ab9e6e6..e56a0f8fbc 100644
--- a/docs/source/gkd_trainer.md
+++ b/docs/source/gkd_trainer.md
@@ -74,7 +74,7 @@ trainer = GKDTrainer(
     model=model,
     teacher_model=teacher_model,
     args=training_args,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     train_dataset=train_dataset,
     eval_dataset=eval_dataset,
 )
diff --git a/docs/source/kto_trainer.mdx b/docs/source/kto_trainer.mdx
index 91c6ea69b1..04b8c16aa5 100644
--- a/docs/source/kto_trainer.mdx
+++ b/docs/source/kto_trainer.mdx
@@ -83,7 +83,7 @@ kto_trainer = KTOTrainer(
     ref_model,
     args=training_args,
     train_dataset=train_dataset,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
 )
 ```
 After this one can then call:
diff --git a/docs/source/nash_md_trainer.md b/docs/source/nash_md_trainer.md
index e7d6fb679b..9397533b14 100644
--- a/docs/source/nash_md_trainer.md
+++ b/docs/source/nash_md_trainer.md
@@ -41,7 +41,7 @@ trainer = NashMDTrainer(
     model=model,
     reward_model=reward_model,
     args=training_args,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     train_dataset=train_dataset,
 )
 trainer.train()
diff --git a/docs/source/online_dpo_trainer.md b/docs/source/online_dpo_trainer.md
index bbb2fe1b1c..de38e11915 100644
--- a/docs/source/online_dpo_trainer.md
+++ b/docs/source/online_dpo_trainer.md
@@ -40,7 +40,7 @@ train_dataset = load_dataset("trl-lib/ultrafeedback-prompt", split="train")
 
 training_args = OnlineDPOConfig(output_dir="online-dpo-qwen2", logging_steps=10)
 trainer = OnlineDPOTrainer(
-    model=model, reward_model=reward_model, args=training_args, tokenizer=tokenizer, train_dataset=train_dataset
+    model=model, reward_model=reward_model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset
 )
 trainer.train()
 ```
diff --git a/docs/source/orpo_trainer.md b/docs/source/orpo_trainer.md
index c717eff454..f21c320ebd 100644
--- a/docs/source/orpo_trainer.md
+++ b/docs/source/orpo_trainer.md
@@ -66,7 +66,7 @@ orpo_trainer = ORPOTrainer(
     model,
     args=training_args,
     train_dataset=train_dataset,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
 )
 ```
 After this one can then call:
diff --git a/docs/source/reward_trainer.mdx b/docs/source/reward_trainer.mdx
index 3ac3de3261..5784308d2c 100644
--- a/docs/source/reward_trainer.mdx
+++ b/docs/source/reward_trainer.mdx
@@ -41,7 +41,7 @@ peft_config = LoraConfig(
 trainer = RewardTrainer(
     model=model,
     args=training_args,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     train_dataset=dataset,
     peft_config=peft_config,
 )
diff --git a/docs/source/sft_trainer.mdx b/docs/source/sft_trainer.mdx
index 39e9e9b638..d3d1a9b221 100644
--- a/docs/source/sft_trainer.mdx
+++ b/docs/source/sft_trainer.mdx
@@ -744,7 +744,7 @@ trainer = SFTTrainer(
     args=training_args,
     data_collator=collate_fn,
     train_dataset=train_dataset,
-    tokenizer=processor.tokenizer,
+    processing_class=processor.tokenizer,
 )
 ```
 
diff --git a/docs/source/xpo_trainer.mdx b/docs/source/xpo_trainer.mdx
index 3d355b4dca..cd6692618f 100644
--- a/docs/source/xpo_trainer.mdx
+++ b/docs/source/xpo_trainer.mdx
@@ -41,7 +41,7 @@ trainer = XPOTrainer(
     model=model,
     reward_model=reward_model,
     args=training_args,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     train_dataset=train_dataset,
 )
 trainer.train()
diff --git a/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py b/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
index 7637367381..8530cd66c4 100644
--- a/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
+++ b/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
@@ -237,7 +237,7 @@ def return_prompt_and_responses(samples) -> Dict[str, str]:
         beta=script_args.beta,
         train_dataset=train_dataset,
         eval_dataset=eval_dataset,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=peft_config,
         max_prompt_length=script_args.max_prompt_length,
         max_length=script_args.max_length,
diff --git a/examples/research_projects/stack_llama_2/scripts/sft_llama2.py b/examples/research_projects/stack_llama_2/scripts/sft_llama2.py
index 97fd9720aa..ef34a67f0c 100644
--- a/examples/research_projects/stack_llama_2/scripts/sft_llama2.py
+++ b/examples/research_projects/stack_llama_2/scripts/sft_llama2.py
@@ -187,7 +187,7 @@ def create_datasets(tokenizer, args, seed=None):
     peft_config=peft_config,
     max_seq_length=None,
     formatting_func=prepare_sample_text,
-    tokenizer=tokenizer,
+    processing_class=tokenizer,
     args=training_args,
 )
 trainer.train()
diff --git a/examples/scripts/bco.py b/examples/scripts/bco.py
index d00b039c21..ebad74c9d0 100644
--- a/examples/scripts/bco.py
+++ b/examples/scripts/bco.py
@@ -152,7 +152,7 @@ def mean_pooling(model_output, attention_mask):
         args=training_args,
         train_dataset=dataset["train"],
         eval_dataset=dataset["test"],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_args),
         embedding_func=embedding_func,
         embedding_tokenizer=embedding_tokenizer,
diff --git a/examples/scripts/cpo.py b/examples/scripts/cpo.py
index 96192cd472..7acbce743a 100644
--- a/examples/scripts/cpo.py
+++ b/examples/scripts/cpo.py
@@ -100,7 +100,7 @@ class ScriptArguments:
         args=training_args,
         train_dataset=dataset["train"],
         eval_dataset=dataset["test"],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_config),
     )
 
diff --git a/examples/scripts/dpo.py b/examples/scripts/dpo.py
index 0669bcc093..685e3f0a6e 100644
--- a/examples/scripts/dpo.py
+++ b/examples/scripts/dpo.py
@@ -121,7 +121,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=peft_config,
     )
 
diff --git a/examples/scripts/dpo_online.py b/examples/scripts/dpo_online.py
index 73abbcd898..48024e3ff1 100644
--- a/examples/scripts/dpo_online.py
+++ b/examples/scripts/dpo_online.py
@@ -107,7 +107,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_config),
     )
     generation_config = GenerationConfig(
diff --git a/examples/scripts/dpo_vlm.py b/examples/scripts/dpo_vlm.py
index 18b3a23303..453d322875 100644
--- a/examples/scripts/dpo_vlm.py
+++ b/examples/scripts/dpo_vlm.py
@@ -126,7 +126,7 @@ def process(row):
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=processor,
+        processing_class=processor,
         peft_config=peft_config,
     )
 
diff --git a/examples/scripts/gkd.py b/examples/scripts/gkd.py
index 0a097f6138..67727fdad5 100644
--- a/examples/scripts/gkd.py
+++ b/examples/scripts/gkd.py
@@ -122,7 +122,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_config),
     )
     completions_callback = LogCompletionsCallback(trainer, trainer.generation_config, num_prompts=8)
diff --git a/examples/scripts/kto.py b/examples/scripts/kto.py
index aefdc812af..af00b6816e 100644
--- a/examples/scripts/kto.py
+++ b/examples/scripts/kto.py
@@ -122,7 +122,7 @@ def format_dataset(example):
         args=training_args,
         train_dataset=dataset["train"],
         eval_dataset=dataset["test"],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_args),
     )
 
diff --git a/examples/scripts/nash_md.py b/examples/scripts/nash_md.py
index b9dd544103..1b3b63504e 100644
--- a/examples/scripts/nash_md.py
+++ b/examples/scripts/nash_md.py
@@ -110,7 +110,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
     )
     generation_config = GenerationConfig(
         max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature
diff --git a/examples/scripts/orpo.py b/examples/scripts/orpo.py
index 521f86c129..e87eb20a68 100644
--- a/examples/scripts/orpo.py
+++ b/examples/scripts/orpo.py
@@ -112,7 +112,7 @@ def process(row):
         args=training_args,
         train_dataset=dataset["train"],
         eval_dataset=dataset["test"],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_config),
     )
 
diff --git a/examples/scripts/ppo/ppo.py b/examples/scripts/ppo/ppo.py
index f257816544..a71ed138d1 100644
--- a/examples/scripts/ppo/ppo.py
+++ b/examples/scripts/ppo/ppo.py
@@ -120,7 +120,7 @@ def tokenize(element):
     ################
     trainer = PPOv2Trainer(
         config=training_args,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         policy=policy,
         ref_policy=ref_policy,
         reward_model=reward_model,
diff --git a/examples/scripts/ppo/ppo_tldr.py b/examples/scripts/ppo/ppo_tldr.py
index 58968046fb..14f12e680e 100644
--- a/examples/scripts/ppo/ppo_tldr.py
+++ b/examples/scripts/ppo/ppo_tldr.py
@@ -125,7 +125,7 @@ def tokenize(element):
     ################
     trainer = PPOv2Trainer(
         config=training_args,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         policy=policy,
         ref_policy=ref_policy,
         reward_model=reward_model,
diff --git a/examples/scripts/reward_modeling.py b/examples/scripts/reward_modeling.py
index 3fae956119..b686d9a98d 100644
--- a/examples/scripts/reward_modeling.py
+++ b/examples/scripts/reward_modeling.py
@@ -111,7 +111,7 @@
     ##########
     trainer = RewardTrainer(
         model=model,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
diff --git a/examples/scripts/rloo/rloo.py b/examples/scripts/rloo/rloo.py
index 5f85a9580b..a924d33950 100644
--- a/examples/scripts/rloo/rloo.py
+++ b/examples/scripts/rloo/rloo.py
@@ -121,7 +121,7 @@ def tokenize(element):
     ################
     trainer = RLOOTrainer(
         config=training_args,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         policy=policy,
         ref_policy=ref_policy,
         reward_model=reward_model,
diff --git a/examples/scripts/rloo/rloo_tldr.py b/examples/scripts/rloo/rloo_tldr.py
index 0a86381e0b..5bcff58b55 100644
--- a/examples/scripts/rloo/rloo_tldr.py
+++ b/examples/scripts/rloo/rloo_tldr.py
@@ -125,7 +125,7 @@ def tokenize(element):
     ################
     trainer = RLOOTrainer(
         config=training_args,
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         policy=policy,
         ref_policy=ref_policy,
         reward_model=reward_model,
diff --git a/examples/scripts/sft.py b/examples/scripts/sft.py
index fddf9c91b0..63ac7ecf2b 100644
--- a/examples/scripts/sft.py
+++ b/examples/scripts/sft.py
@@ -95,7 +95,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
         peft_config=get_peft_config(model_config),
     )
 
diff --git a/examples/scripts/sft_vlm.py b/examples/scripts/sft_vlm.py
index 6cecd93a7a..9935c108f5 100644
--- a/examples/scripts/sft_vlm.py
+++ b/examples/scripts/sft_vlm.py
@@ -119,7 +119,7 @@ def collate_fn(examples):
         data_collator=collate_fn,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=processor.tokenizer,
+        processing_class=processor.tokenizer,
         peft_config=get_peft_config(model_config),
     )
 
diff --git a/examples/scripts/xpo.py b/examples/scripts/xpo.py
index 235935e593..77c792cf1f 100644
--- a/examples/scripts/xpo.py
+++ b/examples/scripts/xpo.py
@@ -94,7 +94,7 @@
         args=training_args,
         train_dataset=dataset[script_args.dataset_train_split],
         eval_dataset=dataset[script_args.dataset_test_split],
-        tokenizer=tokenizer,
+        processing_class=tokenizer,
     )
     generation_config = GenerationConfig(
         max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature
diff --git a/trl/trainer/bco_trainer.py b/trl/trainer/bco_trainer.py
index a836e3b928..3054b971d1 100644
--- a/trl/trainer/bco_trainer.py
+++ b/trl/trainer/bco_trainer.py
@@ -34,9 +34,12 @@
 from torch.utils.data import DataLoader, SequentialSampler
 from transformers import (
     AutoModelForCausalLM,
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainingArguments,
     is_sklearn_available,
@@ -281,8 +284,10 @@ class BCOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         data_collator (`transformers.DataCollator`, *optional*, defaults to `None`):
             The data collator to use for training. If None is specified, the default data collator (`DPODataCollatorWithPadding`) will be used
             which will pad the sequences to the maximum length of the sequences in the batch, given a dataset of paired sequences.
@@ -316,7 +321,9 @@ def __init__(
         args: BCOConfig = None,
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         data_collator: Optional[DataCollator] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -476,9 +483,9 @@ def make_inputs_require_grad(module, input, output):
         else:
             self.ref_model = create_reference_model(model)
 
-        if tokenizer is None:
+        if processing_class is None:
             raise ValueError(
-                "max_length or a tokenizer must be specified when using the default DPODataCollatorWithPadding"
+                "max_length or a processing_class must be specified when using the default DPODataCollatorWithPadding"
             )
         if args.max_length is None:
             warnings.warn(
@@ -513,7 +520,7 @@ def make_inputs_require_grad(module, input, output):
 
         if data_collator is None:
             data_collator = DPODataCollatorWithPadding(
-                pad_token_id=tokenizer.pad_token_id,
+                pad_token_id=processing_class.pad_token_id,
                 label_pad_token_id=args.label_pad_token_id,
                 is_encoder_decoder=self.is_encoder_decoder,
             )
@@ -539,11 +546,10 @@ def make_inputs_require_grad(module, input, output):
         self.max_length = max_length
         self.generate_during_eval = args.generate_during_eval
         self.label_pad_token_id = args.label_pad_token_id
-        self.padding_value = args.padding_value if args.padding_value is not None else tokenizer.pad_token_id
+        self.padding_value = args.padding_value if args.padding_value is not None else processing_class.pad_token_id
         self.max_prompt_length = max_prompt_length
         self.truncation_mode = args.truncation_mode
         self.max_completion_length = max_completion_length
-        self.tokenizer = tokenizer
         self.precompute_ref_log_probs = args.precompute_ref_log_probs
 
         # Since ref_logs are precomputed on the first call to get_train/eval_dataloader
@@ -565,11 +571,13 @@ def make_inputs_require_grad(module, input, output):
         with PartialState().local_main_process_first():
             # Apply the chat template if needed
             train_dataset = train_dataset.map(
-                maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                maybe_apply_chat_template, fn_kwargs={"tokenizer": processing_class}, num_proc=args.dataset_num_proc
             )
             if eval_dataset is not None:
                 eval_dataset = eval_dataset.map(
-                    maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                    maybe_apply_chat_template,
+                    fn_kwargs={"tokenizer": processing_class},
+                    num_proc=args.dataset_num_proc,
                 )
             # Shuffle the datasets
             train_dataset = train_dataset.shuffle(seed=args.data_seed)
@@ -579,7 +587,7 @@ def make_inputs_require_grad(module, input, output):
             train_dataset = train_dataset.map(
                 _tokenize,
                 batched=True,
-                fn_kwargs={"tokenizer": self.tokenizer, "embedding_tokenizer": self.embedding_tokenizer},
+                fn_kwargs={"tokenizer": processing_class, "embedding_tokenizer": self.embedding_tokenizer},
                 num_proc=args.dataset_num_proc,
                 desc="Tokenizing train dataset",
             )
@@ -588,7 +596,7 @@ def make_inputs_require_grad(module, input, output):
             fn_kwargs = {
                 "prefix": "",
                 "is_encoder_decoder": self.is_encoder_decoder,
-                "tokenizer": self.tokenizer,
+                "tokenizer": processing_class,
                 "max_length": self.max_length,
                 "truncation_mode": self.truncation_mode,
                 "label_pad_token_id": self.label_pad_token_id,
@@ -606,7 +614,7 @@ def make_inputs_require_grad(module, input, output):
                 # Tokenize
                 eval_dataset = eval_dataset.map(
                     _tokenize,
-                    fn_kwargs={"tokenizer": self.tokenizer, "embedding_tokenizer": self.embedding_tokenizer},
+                    fn_kwargs={"tokenizer": processing_class, "embedding_tokenizer": self.embedding_tokenizer},
                     batched=True,
                     num_proc=args.dataset_num_proc,
                     desc="Tokenizing eval dataset",
@@ -616,7 +624,7 @@ def make_inputs_require_grad(module, input, output):
                 fn_kwargs = {
                     "prefix": "",
                     "is_encoder_decoder": self.is_encoder_decoder,
-                    "tokenizer": self.tokenizer,
+                    "tokenizer": processing_class,
                     "max_length": self.max_length,
                     "truncation_mode": self.truncation_mode,
                     "label_pad_token_id": self.label_pad_token_id,
@@ -646,7 +654,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -734,11 +742,11 @@ def _get_chosen_prob(self, prompt_embeddings: torch.FloatTensor) -> torch.FloatT
 
     def _vectorize_prompt(self, input_ids: torch.LongTensor, attention_mask: torch.LongTensor) -> torch.FloatTensor:
         """
-        Replaces tokenizer.pad_token_id to embedding_tokenizer.pad_token_id
+        Replaces processing_class.pad_token_id to embedding_tokenizer.pad_token_id
         and applies self.embedding_func
         """
         input_ids = torch.where(
-            input_ids == self.tokenizer.pad_token_id,
+            input_ids == self.processing_class.pad_token_id,
             self.embedding_tokenizer.pad_token_id,
             input_ids,
         )
@@ -1288,7 +1296,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                 attention_mask=batch["prompt_attention_mask"],
                 max_length=self.max_length,
                 do_sample=True,
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
             )
 
             # if reference_output in batch use that otherwise use the reference model
@@ -1302,7 +1310,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                             attention_mask=batch["prompt_attention_mask"],
                             max_length=self.max_length,
                             do_sample=True,
-                            pad_token_id=self.tokenizer.pad_token_id,
+                            pad_token_id=self.processing_class.pad_token_id,
                         )
                 else:
                     reference_output = self.ref_model.generate(
@@ -1310,14 +1318,14 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                         attention_mask=batch["prompt_attention_mask"],
                         max_length=self.max_length,
                         do_sample=True,
-                        pad_token_id=self.tokenizer.pad_token_id,
+                        pad_token_id=self.processing_class.pad_token_id,
                     )
 
-        policy_output = pad_to_length(policy_output, self.max_length, self.tokenizer.pad_token_id)
-        policy_output_decoded = self.tokenizer.batch_decode(policy_output, skip_special_tokens=True)
+        policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)
+        policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)
 
-        reference_output = pad_to_length(reference_output, self.max_length, self.tokenizer.pad_token_id)
-        reference_output_decoded = self.tokenizer.batch_decode(reference_output, skip_special_tokens=True)
+        reference_output = pad_to_length(reference_output, self.max_length, self.processing_class.pad_token_id)
+        reference_output_decoded = self.processing_class.batch_decode(reference_output, skip_special_tokens=True)
 
         return policy_output_decoded, reference_output_decoded
 
diff --git a/trl/trainer/cpo_trainer.py b/trl/trainer/cpo_trainer.py
index 929e137062..ebd70bf93f 100644
--- a/trl/trainer/cpo_trainer.py
+++ b/trl/trainer/cpo_trainer.py
@@ -32,9 +32,12 @@
 from torch.utils.data import DataLoader
 from transformers import (
     AutoModelForCausalLM,
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     is_wandb_available,
 )
@@ -79,8 +82,10 @@ class CPOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         model_init (`Callable[[], transformers.PreTrainedModel]`):
             The model initializer to use for training. If None is specified, the default model initializer will be used.
         callbacks (`List[transformers.TrainerCallback]`):
@@ -105,7 +110,9 @@ def __init__(
         data_collator: Optional[DataCollator] = None,
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
         optimizers: Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),
@@ -212,8 +219,8 @@ def make_inputs_require_grad(module, input, output):
             self.decoder_start_token_id = model.config.decoder_start_token_id
             self.pad_token_id = model.config.pad_token_id
 
-        if tokenizer is None:
-            raise ValueError("tokenizer must be specified to tokenize a CPO dataset.")
+        if processing_class is None:
+            raise ValueError("processing_class must be specified to tokenize a CPO dataset.")
         if args.max_length is None:
             warnings.warn(
                 "`max_length` is not set in the CPOConfig's init"
@@ -245,7 +252,7 @@ def make_inputs_require_grad(module, input, output):
 
         if data_collator is None:
             data_collator = DPODataCollatorWithPadding(
-                pad_token_id=tokenizer.pad_token_id,
+                pad_token_id=processing_class.pad_token_id,
                 label_pad_token_id=args.label_pad_token_id,
                 is_encoder_decoder=self.is_encoder_decoder,
             )
@@ -269,11 +276,11 @@ def make_inputs_require_grad(module, input, output):
         self.max_length = max_length
         self.generate_during_eval = args.generate_during_eval
         self.label_pad_token_id = args.label_pad_token_id
-        self.padding_value = args.padding_value if args.padding_value is not None else tokenizer.pad_token_id
+        self.padding_value = args.padding_value if args.padding_value is not None else processing_class.pad_token_id
         self.max_prompt_length = max_prompt_length
         self.truncation_mode = args.truncation_mode
         self.max_completion_length = max_completion_length
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
 
         if args.loss_type in ["hinge", "ipo"] and args.label_smoothing > 0:
             warnings.warn(
@@ -306,12 +313,14 @@ def make_inputs_require_grad(module, input, output):
             # Extract the prompt if needed, and apply the chat template if needed
             train_dataset = train_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)
             train_dataset = train_dataset.map(
-                maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                maybe_apply_chat_template, fn_kwargs={"tokenizer": processing_class}, num_proc=args.dataset_num_proc
             )
             if eval_dataset is not None:
                 eval_dataset = eval_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)
                 eval_dataset = eval_dataset.map(
-                    maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                    maybe_apply_chat_template,
+                    fn_kwargs={"tokenizer": processing_class},
+                    num_proc=args.dataset_num_proc,
                 )
 
             # tokenize the dataset
@@ -325,7 +334,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -350,8 +359,8 @@ def build_tokenized_answer(self, prompt, answer):
             https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
         """
 
-        full_tokenized = self.tokenizer(prompt + answer, add_special_tokens=False)
-        prompt_input_ids = self.tokenizer(prompt, add_special_tokens=False)["input_ids"]
+        full_tokenized = self.processing_class(prompt + answer, add_special_tokens=False)
+        prompt_input_ids = self.processing_class(prompt, add_special_tokens=False)["input_ids"]
 
         answer_input_ids = full_tokenized["input_ids"][len(prompt_input_ids) :]
         answer_attention_mask = full_tokenized["attention_mask"][len(prompt_input_ids) :]
@@ -416,7 +425,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             if not isinstance(prompt, str):
                 raise ValueError(f"prompt should be an str but got {type(prompt)}")
-            prompt_tokens = self.tokenizer(prompt, add_special_tokens=False)
+            prompt_tokens = self.processing_class(prompt, add_special_tokens=False)
             prompt_tokens = {f"prompt_{k}": v for k, v in prompt_tokens.items()}
 
             if not isinstance(chosen, str):
@@ -452,7 +461,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             # add BOS token to head of prompt. Avoid adding if it's already there
             prompt_tokens, chosen_tokens, rejected_tokens = add_bos_token_if_needed(
-                self.tokenizer.bos_token_id,
+                self.processing_class.bos_token_id,
                 prompt_len_input_ids,
                 prompt_tokens,
                 chosen_prompt_len_input_ids,
@@ -463,7 +472,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             # add EOS token to end of answer. Avoid adding if it's already there
             chosen_tokens, rejected_tokens = add_eos_token_if_needed(
-                self.tokenizer.eos_token_id, chosen_tokens, rejected_tokens
+                self.processing_class.eos_token_id, chosen_tokens, rejected_tokens
             )
 
             longer_response_length = max(len(chosen_tokens["input_ids"]), len(rejected_tokens["input_ids"]))
@@ -513,13 +522,13 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
                     batch[f"{k}{type_key}"] = tokens
 
         else:
-            chosen_tokens = self.tokenizer(
+            chosen_tokens = self.processing_class(
                 chosen, truncation=True, max_length=self.max_completion_length, add_special_tokens=True
             )
-            rejected_tokens = self.tokenizer(
+            rejected_tokens = self.processing_class(
                 rejected, truncation=True, max_length=self.max_completion_length, add_special_tokens=True
             )
-            prompt_tokens = self.tokenizer(
+            prompt_tokens = self.processing_class(
                 prompt, truncation=True, max_length=self.max_prompt_length, add_special_tokens=True
             )
 
@@ -845,11 +854,11 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                 attention_mask=batch["prompt_attention_mask"],
                 max_length=self.max_length,
                 do_sample=True,
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
             )
 
-        policy_output = pad_to_length(policy_output, self.max_length, self.tokenizer.pad_token_id)
-        policy_output_decoded = self.tokenizer.batch_decode(policy_output, skip_special_tokens=True)
+        policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)
+        policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)
 
         return policy_output_decoded
 
diff --git a/trl/trainer/dpo_trainer.py b/trl/trainer/dpo_trainer.py
index 55095ae5da..f92a5cc98e 100644
--- a/trl/trainer/dpo_trainer.py
+++ b/trl/trainer/dpo_trainer.py
@@ -33,9 +33,12 @@
 from torch.utils.data import DataLoader
 from transformers import (
     AutoModelForCausalLM,
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     is_wandb_available,
 )
@@ -43,6 +46,7 @@
 from transformers.trainer_callback import TrainerCallback
 from transformers.trainer_utils import EvalLoopOutput
 from transformers.utils import is_peft_available
+from transformers.utils.deprecation import deprecate_kwarg
 
 from ..data_utils import maybe_apply_chat_template, maybe_extract_prompt
 from ..models import PreTrainedModelWrapper, create_reference_model
@@ -373,8 +377,11 @@ class DPOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
+            This supercedes the `tokenizer` argument, which is now deprecated.
         model_init (`Callable[[], transformers.PreTrainedModel]`):
             The model initializer to use for training. If None is specified, the default model initializer will be used.
         callbacks (`List[transformers.TrainerCallback]`):
@@ -418,6 +425,7 @@ class DPOTrainer(Trainer):
         ],
         custom_message="Deprecated positional argument(s) used in DPOTrainer, please use the DPOConfig to set these arguments instead.",
     )
+    @deprecate_kwarg("tokenizer", new_name="processing_class", version="0.14.0", raise_if_both_names=True)
     def __init__(
         self,
         model: Optional[Union[PreTrainedModel, nn.Module, str]] = None,
@@ -432,7 +440,9 @@ def __init__(
         truncation_mode: str = "keep_end",
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
         optimizers: Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),
@@ -629,10 +639,10 @@ def make_inputs_require_grad(module, input, output):
             self.is_vision_model = False
 
         if self.is_vision_model:
-            self.processor = tokenizer
-            self.tokenizer = tokenizer.tokenizer  # tokenizer is actually a processor at this point
+            self.processor = processing_class
+            self.processing_class = self.processor.tokenizer  # tokenizer is actually a processor at this point
         else:
-            self.tokenizer = tokenizer
+            self.processing_class = processing_class
 
         self.is_peft_model = is_peft_available() and isinstance(model, PeftModel)
         if model_adapter_name is not None:
@@ -670,8 +680,8 @@ def make_inputs_require_grad(module, input, output):
         else:
             self.ref_model = create_reference_model(model)
 
-        if tokenizer is None:
-            raise ValueError("tokenizer must be specified to tokenize a DPO dataset.")
+        if processing_class is None:
+            raise ValueError("processing_class must be specified to tokenize a DPO dataset.")
 
         if max_length is not None:
             warnings.warn(
@@ -719,7 +729,7 @@ def make_inputs_require_grad(module, input, output):
             args.label_pad_token_id = label_pad_token_id
         if data_collator is None:
             data_collator = DPODataCollatorWithPadding(
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
                 label_pad_token_id=args.label_pad_token_id,
                 is_encoder_decoder=self.is_encoder_decoder,
             )
@@ -755,7 +765,7 @@ def make_inputs_require_grad(module, input, output):
                 "You passed `padding_value` to the DPOTrainer, the value you passed will override the one in the `DPOConfig`."
             )
             args.padding_value = padding_value
-        self.padding_value = args.padding_value if padding_value is not None else self.tokenizer.pad_token_id
+        self.padding_value = args.padding_value if padding_value is not None else self.processing_class.pad_token_id
         self.max_prompt_length = args.max_prompt_length
         if truncation_mode != "keep_end":
             warnings.warn(
@@ -819,17 +829,19 @@ def make_inputs_require_grad(module, input, output):
             # Extract the prompt if needed, and apply the chat template if needed
             train_dataset = train_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)
             train_dataset = train_dataset.map(
-                maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                maybe_apply_chat_template, fn_kwargs={"tokenizer": processing_class}, num_proc=args.dataset_num_proc
             )
             if eval_dataset is not None:
                 eval_dataset = eval_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)
                 eval_dataset = eval_dataset.map(
-                    maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer}, num_proc=args.dataset_num_proc
+                    maybe_apply_chat_template,
+                    fn_kwargs={"tokenizer": processing_class},
+                    num_proc=args.dataset_num_proc,
                 )
 
             # tokenize the dataset, lower writer batch size to avoid OOM (frequent in vision models)
             fn_kwargs = {
-                "tokenizer": self.tokenizer,
+                "tokenizer": self.processing_class,
                 "args": args,
                 "processor": self.processor if self.is_vision_model else None,
                 "model": model if self.is_encoder_decoder else None,
@@ -858,7 +870,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -1547,7 +1559,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                 attention_mask=batch["prompt_attention_mask"],
                 max_length=self.max_length,
                 do_sample=True,
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
             )
 
             # if reference_output in batch use that otherwise use the reference model
@@ -1561,7 +1573,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                             attention_mask=batch["prompt_attention_mask"],
                             max_length=self.max_length,
                             do_sample=True,
-                            pad_token_id=self.tokenizer.pad_token_id,
+                            pad_token_id=self.processing_class.pad_token_id,
                         )
                 else:
                     reference_output = self.ref_model.generate(
@@ -1569,14 +1581,14 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                         attention_mask=batch["prompt_attention_mask"],
                         max_length=self.max_length,
                         do_sample=True,
-                        pad_token_id=self.tokenizer.pad_token_id,
+                        pad_token_id=self.processing_class.pad_token_id,
                     )
 
-        policy_output = pad_to_length(policy_output, self.max_length, self.tokenizer.pad_token_id)
-        policy_output_decoded = self.tokenizer.batch_decode(policy_output, skip_special_tokens=True)
+        policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)
+        policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)
 
-        reference_output = pad_to_length(reference_output, self.max_length, self.tokenizer.pad_token_id)
-        reference_output_decoded = self.tokenizer.batch_decode(reference_output, skip_special_tokens=True)
+        reference_output = pad_to_length(reference_output, self.max_length, self.processing_class.pad_token_id)
+        reference_output_decoded = self.processing_class.batch_decode(reference_output, skip_special_tokens=True)
 
         return policy_output_decoded, reference_output_decoded
 
diff --git a/trl/trainer/gkd_trainer.py b/trl/trainer/gkd_trainer.py
index 1dc96b4135..d9c20308cf 100644
--- a/trl/trainer/gkd_trainer.py
+++ b/trl/trainer/gkd_trainer.py
@@ -16,13 +16,27 @@
 import textwrap
 import warnings
 from copy import deepcopy
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from accelerate.utils import is_deepspeed_available
-from transformers import AutoModelForCausalLM, GenerationConfig, PreTrainedModel, is_wandb_available
+from datasets import Dataset
+from transformers import (
+    AutoModelForCausalLM,
+    BaseImageProcessor,
+    DataCollator,
+    FeatureExtractionMixin,
+    GenerationConfig,
+    PreTrainedModel,
+    PreTrainedTokenizerBase,
+    ProcessorMixin,
+    is_wandb_available,
+)
+from transformers.trainer_callback import TrainerCallback
+from transformers.trainer_utils import EvalPrediction
+from transformers.utils import is_peft_available
 
 from ..import_utils import is_liger_kernel_available
 from ..models import PreTrainedModelWrapper
@@ -38,6 +52,9 @@
 if is_liger_kernel_available():
     from liger_kernel.transformers import AutoLigerKernelForCausalLM
 
+if is_peft_available():
+    from peft import PeftConfig
+
 if is_wandb_available():
     import wandb
 
@@ -47,16 +64,42 @@ class GKDTrainer(SFTTrainer):
 
     def __init__(
         self,
-        teacher_model: Union[PreTrainedModel, nn.Module, str],
+        model: Optional[Union[PreTrainedModel, nn.Module, str]] = None,
+        teacher_model: Union[PreTrainedModel, nn.Module, str] = None,
         args: Optional[GKDConfig] = None,
-        *sft_args,
-        **kwargs,
+        data_collator: Optional[DataCollator] = None,  # type: ignore
+        train_dataset: Optional[Dataset] = None,
+        eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
+        model_init: Optional[Callable[[], PreTrainedModel]] = None,
+        compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
+        callbacks: Optional[List[TrainerCallback]] = None,
+        optimizers: Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),
+        preprocess_logits_for_metrics: Optional[Callable[[torch.Tensor, torch.Tensor], torch.Tensor]] = None,
+        peft_config: Optional["PeftConfig"] = None,
+        formatting_func: Optional[Callable] = None,
     ):
         # add remove_unused_columns=False to the the dataclass args
         args.remove_unused_columns = False
-        kwargs["data_collator"] = DataCollatorForChatML(tokenizer=kwargs["tokenizer"], max_length=args.max_seq_length)
-
-        super().__init__(*sft_args, args=args, **kwargs)
+        data_collator = DataCollatorForChatML(tokenizer=processing_class, max_length=args.max_seq_length)
+
+        super().__init__(
+            model,
+            args=args,
+            data_collator=data_collator,
+            train_dataset=train_dataset,
+            eval_dataset=eval_dataset,
+            processing_class=processing_class,
+            model_init=model_init,
+            compute_metrics=compute_metrics,
+            callbacks=callbacks,
+            optimizers=optimizers,
+            preprocess_logits_for_metrics=preprocess_logits_for_metrics,
+            peft_config=peft_config,
+            formatting_func=formatting_func,
+        )
 
         if args.teacher_model_init_kwargs is None:
             teacher_model_init_kwargs = {}
@@ -100,7 +143,7 @@ def __init__(
             do_sample=True,
             top_k=0,
             use_cache=False if args.gradient_checkpointing else True,
-            pad_token_id=self.tokenizer.pad_token_id,
+            pad_token_id=self.processing_class.pad_token_id,
         )
         # Set custom EOS tokens if they are specified by the model's generation
         # config. This is important for models with the Llama 3 chat template,
@@ -235,7 +278,7 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
         if random.random() <= self.lmbda:
             with unwrap_model_for_generation(model, self.accelerator) as unwrapped_model:
                 new_input_ids, new_attention_mask, new_labels = self.generate_on_policy_outputs(
-                    unwrapped_model, inputs, self.generation_config, self.tokenizer.pad_token_id
+                    unwrapped_model, inputs, self.generation_config, self.processing_class.pad_token_id
                 )
             inputs["input_ids"] = new_input_ids
             inputs["attention_mask"] = new_attention_mask
diff --git a/trl/trainer/iterative_sft_trainer.py b/trl/trainer/iterative_sft_trainer.py
index 0e0d19f08e..1cddd37346 100644
--- a/trl/trainer/iterative_sft_trainer.py
+++ b/trl/trainer/iterative_sft_trainer.py
@@ -19,11 +19,14 @@
 from datasets import Dataset
 from torch.utils.data import DataLoader
 from transformers import (
+    BaseImageProcessor,
     DataCollator,
     DataCollatorForLanguageModeling,
     DataCollatorForSeq2Seq,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainingArguments,
     is_wandb_available,
@@ -53,9 +56,10 @@ class IterativeSFTTrainer(Trainer):
             Check the documentation of `PreTrainedModel` for more details.
         args (`transformers.TrainingArguments`):
             The arguments to use for training.
-        tokenizer (`PreTrainedTokenizerBase`):
-            Tokenizer to be used for encoding the data. Check the documentation of `transformers.PreTrainedTokenizer` and
-            `transformers.PreTrainedTokenizerFast` for more details.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         optimizers (`Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):
             The optimizer and scheduler to use for training.
         data_collator (Union[DataCollatorForLanguageModeling, DataCollatorForSeq2Seq], *optional*):
@@ -80,7 +84,9 @@ def __init__(
         self,
         model: Optional[PreTrainedModel] = None,
         args: Optional[TrainingArguments] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         optimizers: Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (
             None,
             None,
@@ -94,9 +100,9 @@ def __init__(
         optimize_device_cache: Optional[bool] = False,
     ):
         # Step 0: check positional arguments validity
-        if not isinstance(tokenizer, (PreTrainedTokenizerBase)):
+        if not isinstance(processing_class, (PreTrainedTokenizerBase)):
             raise ValueError(
-                f"tokenizer must be a PreTrainedTokenizerBase like a PreTrainedTokenizer or a PreTrainedTokenizerFast, got {type(tokenizer)}"
+                f"processing_class must be a PreTrainedTokenizerBase like a PreTrainedTokenizer or a PreTrainedTokenizerFast, got {type(processing_class)}"
             )
         if not isinstance(model, PreTrainedModel):
             raise ValueError(f"model must be a PreTrainedModel, got {type(model)}")
@@ -113,7 +119,7 @@ def __init__(
         self.is_encoder_decoder = getattr(model.config, "is_encoder_decoder", False)
         self.is_peft_model = is_peft_available() and isinstance(model, PeftModel)
 
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
 
         if data_collator is None:
             if self.is_encoder_decoder:
@@ -121,10 +127,12 @@ def __init__(
                     "No data collator is provided. Using 'DataCollatorForSeq2Seq' with"
                     "'labels_pad_token_id' set to '-100' and 'pad_to_multiple_of' set to 8."
                 )
-                self.data_collator = DataCollatorForSeq2Seq(tokenizer, label_pad_token_id=-100, pad_to_multiple_of=8)
+                self.data_collator = DataCollatorForSeq2Seq(
+                    processing_class, label_pad_token_id=-100, pad_to_multiple_of=8
+                )
             else:
                 warnings.warn("No data collator is provided. Using 'DataCollatorForLanguageModeling'")
-                self.data_collator = DataCollatorForLanguageModeling(self.tokenizer, mlm=False)
+                self.data_collator = DataCollatorForLanguageModeling(self.processing_class, mlm=False)
         else:
             self.data_collator = data_collator
 
@@ -137,7 +145,7 @@ def __init__(
             args=args,
             data_collator=self.data_collator,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             compute_metrics=compute_metrics,
             optimizers=optimizers,
             preprocess_logits_for_metrics=preprocess_logits_for_metrics,
@@ -154,7 +162,7 @@ def __init__(
             self.model, self.optimizer, self.lr_scheduler
         )
 
-        self.tokenizer.truncation_side = "left" if self.truncation_mode == "keep_end" else "right"
+        self.processing_class.truncation_side = "left" if self.truncation_mode == "keep_end" else "right"
 
         if not hasattr(self, "accelerator"):
             raise AttributeError(
@@ -177,7 +185,7 @@ def prepare_model_inputs(self, input_ids: torch.Tensor, attention_mask: torch.Te
 
             input_data.pop("decoder_input_ids", None)  # This is directly computed inside the model
 
-            input_data["labels"][input_data["labels"] == self.tokenizer.pad_token_id] = -100
+            input_data["labels"][input_data["labels"] == self.processing_class.pad_token_id] = -100
 
         else:
             input_data = self.data_collator(
@@ -298,14 +306,14 @@ def step(
         )
 
         if texts is not None:
-            model_inputs = self.tokenizer(
+            model_inputs = self.processing_class(
                 texts, max_length=self.max_length, truncation=True, padding=True, return_tensors="pt"
             )
 
             input_ids, attention_mask = model_inputs["input_ids"], model_inputs["attention_mask"]
 
         if texts_labels is not None:
-            labels = self.tokenizer(
+            labels = self.processing_class(
                 texts, max_length=self.max_length, truncation=True, padding=True, return_tensors="pt"
             )["input_ids"]
 
diff --git a/trl/trainer/kto_trainer.py b/trl/trainer/kto_trainer.py
index b670db9c6e..5d7e7769ea 100644
--- a/trl/trainer/kto_trainer.py
+++ b/trl/trainer/kto_trainer.py
@@ -34,9 +34,12 @@
 from torch.utils.data import DataLoader, SequentialSampler
 from transformers import (
     AutoModelForCausalLM,
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainerCallback,
     TrainingArguments,
@@ -271,8 +274,10 @@ class KTOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         data_collator (`transformers.DataCollator`, *optional*, defaults to `None`):
             The data collator to use for training. If None is specified, the default data collator (`DPODataCollatorWithPadding`) will be used
             which will pad the sequences to the maximum length of the sequences in the batch, given a dataset of paired sequences.
@@ -306,7 +311,9 @@ def __init__(
         args: KTOConfig = None,
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         data_collator: Optional[DataCollator] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -459,9 +466,9 @@ def make_inputs_require_grad(module, input, output):
         else:
             self.ref_model = create_reference_model(model)
 
-        if tokenizer is None:
+        if processing_class is None:
             raise ValueError(
-                "max_length or a tokenizer must be specified when using the default DPODataCollatorWithPadding"
+                "max_length or a processing_class must be specified when using the default DPODataCollatorWithPadding"
             )
         if args.max_length is None:
             warnings.warn(
@@ -496,7 +503,7 @@ def make_inputs_require_grad(module, input, output):
 
         if data_collator is None:
             data_collator = DPODataCollatorWithPadding(
-                pad_token_id=tokenizer.pad_token_id,
+                pad_token_id=processing_class.pad_token_id,
                 label_pad_token_id=args.label_pad_token_id,
                 is_encoder_decoder=self.is_encoder_decoder,
             )
@@ -523,11 +530,11 @@ def make_inputs_require_grad(module, input, output):
         self.max_length = max_length
         self.generate_during_eval = args.generate_during_eval
         self.label_pad_token_id = args.label_pad_token_id
-        self.padding_value = args.padding_value if args.padding_value is not None else tokenizer.pad_token_id
+        self.padding_value = args.padding_value if args.padding_value is not None else processing_class.pad_token_id
         self.max_prompt_length = max_prompt_length
         self.truncation_mode = args.truncation_mode
         self.max_completion_length = max_completion_length
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
         self.precompute_ref_log_probs = args.precompute_ref_log_probs
 
         # Not all losses require a KL calculation
@@ -559,7 +566,7 @@ def make_inputs_require_grad(module, input, output):
             train_dataset = train_dataset.map(
                 _tokenize,
                 batched=True,
-                fn_kwargs={"tokenizer": self.tokenizer},
+                fn_kwargs={"tokenizer": self.processing_class},
                 num_proc=args.dataset_num_proc,
                 desc="Tokenizing train dataset",
             )
@@ -567,7 +574,7 @@ def make_inputs_require_grad(module, input, output):
             fn_kwargs = {
                 "prefix": "",
                 "is_encoder_decoder": self.is_encoder_decoder,
-                "tokenizer": self.tokenizer,
+                "tokenizer": self.processing_class,
                 "max_length": self.max_length,
                 "truncation_mode": self.truncation_mode,
                 "label_pad_token_id": self.label_pad_token_id,
@@ -586,7 +593,7 @@ def make_inputs_require_grad(module, input, output):
             if eval_dataset is not None:
                 eval_dataset = eval_dataset.map(
                     _tokenize,
-                    fn_kwargs={"tokenizer": self.tokenizer},
+                    fn_kwargs={"tokenizer": self.processing_class},
                     batched=True,
                     num_proc=args.dataset_num_proc,
                     desc="Tokenizing eval dataset",
@@ -685,7 +692,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -1265,7 +1272,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                 attention_mask=batch["prompt_attention_mask"],
                 max_length=self.max_length,
                 do_sample=True,
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
             )
 
             # if reference_output in batch use that otherwise use the reference model
@@ -1279,7 +1286,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                             attention_mask=batch["prompt_attention_mask"],
                             max_length=self.max_length,
                             do_sample=True,
-                            pad_token_id=self.tokenizer.pad_token_id,
+                            pad_token_id=self.processing_class.pad_token_id,
                         )
                 else:
                     reference_output = self.ref_model.generate(
@@ -1287,14 +1294,14 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                         attention_mask=batch["prompt_attention_mask"],
                         max_length=self.max_length,
                         do_sample=True,
-                        pad_token_id=self.tokenizer.pad_token_id,
+                        pad_token_id=self.processing_class.pad_token_id,
                     )
 
-        policy_output = pad_to_length(policy_output, self.max_length, self.tokenizer.pad_token_id)
-        policy_output_decoded = self.tokenizer.batch_decode(policy_output, skip_special_tokens=True)
+        policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)
+        policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)
 
-        reference_output = pad_to_length(reference_output, self.max_length, self.tokenizer.pad_token_id)
-        reference_output_decoded = self.tokenizer.batch_decode(reference_output, skip_special_tokens=True)
+        reference_output = pad_to_length(reference_output, self.max_length, self.processing_class.pad_token_id)
+        reference_output_decoded = self.processing_class.batch_decode(reference_output, skip_special_tokens=True)
 
         return policy_output_decoded, reference_output_decoded
 
diff --git a/trl/trainer/nash_md_trainer.py b/trl/trainer/nash_md_trainer.py
index b15c0d5c59..36f48a48da 100644
--- a/trl/trainer/nash_md_trainer.py
+++ b/trl/trainer/nash_md_trainer.py
@@ -20,8 +20,15 @@
 import torch.nn as nn
 import torch.nn.functional as F
 from datasets import Dataset, IterableDataset
-from transformers import PreTrainedTokenizerBase, TrainerCallback, is_wandb_available
-from transformers.modeling_utils import PreTrainedModel
+from transformers import (
+    BaseImageProcessor,
+    FeatureExtractionMixin,
+    PreTrainedModel,
+    PreTrainedTokenizerBase,
+    ProcessorMixin,
+    TrainerCallback,
+    is_wandb_available,
+)
 from transformers.trainer_utils import EvalPrediction
 from transformers.training_args import OptimizerNames
 from transformers.utils import is_apex_available
@@ -65,8 +72,10 @@ class NashMDTrainer(OnlineDPOTrainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         peft_config (`Dict`):
             The peft config to use for training.
         compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*):
@@ -91,7 +100,9 @@ def __init__(
         data_collator: Optional[Callable] = None,
         train_dataset: Optional[Union[Dataset, IterableDataset]] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         peft_config: Optional[Dict] = None,
         compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -106,7 +117,7 @@ def __init__(
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             peft_config=peft_config,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -175,7 +186,7 @@ def _process_completions(self, model_output, mixture_output, prompts):
         # Process model completions
         model_completion_ids = model_output[:, context_length:]
         model_completion_ids, model_completion_mask = truncate_right(
-            model_completion_ids, self.tokenizer.eos_token_id, self.tokenizer.pad_token_id
+            model_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id
         )
         model_data = {
             "input_ids": torch.cat((prompts["input_ids"], model_completion_ids), dim=1),
@@ -185,7 +196,7 @@ def _process_completions(self, model_output, mixture_output, prompts):
         # Process reference model completions
         mixture_completion_ids = mixture_output[:, context_length:]
         mixture_completion_ids, mixture_completion_mask = truncate_right(
-            mixture_completion_ids, self.tokenizer.eos_token_id, self.tokenizer.pad_token_id
+            mixture_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id
         )
         mixture_data = {
             "input_ids": torch.cat((prompts["input_ids"], mixture_completion_ids), dim=1),
@@ -197,16 +208,16 @@ def _process_completions(self, model_output, mixture_output, prompts):
     def _compute_rewards(self, model_data, mixture_data, context_length):
         with torch.no_grad():
             _, model_scores, _ = get_reward(
-                self.reward_model, model_data["input_ids"], self.tokenizer.pad_token_id, context_length
+                self.reward_model, model_data["input_ids"], self.processing_class.pad_token_id, context_length
             )
             _, mixture_scores, _ = get_reward(
-                self.reward_model, mixture_data["input_ids"], self.tokenizer.pad_token_id, context_length
+                self.reward_model, mixture_data["input_ids"], self.processing_class.pad_token_id, context_length
             )
 
         # Apply EOS penalty if needed
         if self.args.missing_eos_penalty is not None:
-            model_contain_eos = torch.any(model_data["input_ids"] == self.tokenizer.eos_token_id, dim=-1)
-            mixture_contain_eos = torch.any(mixture_data["input_ids"] == self.tokenizer.eos_token_id, dim=-1)
+            model_contain_eos = torch.any(model_data["input_ids"] == self.processing_class.eos_token_id, dim=-1)
+            mixture_contain_eos = torch.any(mixture_data["input_ids"] == self.processing_class.eos_token_id, dim=-1)
             model_scores[~model_contain_eos] -= self.args.missing_eos_penalty
             mixture_scores[~mixture_contain_eos] -= self.args.missing_eos_penalty
 
@@ -308,8 +319,8 @@ def gather_mean(tensor):
         self.stats["rewards/accuracies"].append(gather_mean(accuracy))
 
         # Log EOS token statistics
-        model_eos = (model_data["input_ids"][:, context_length:] == self.tokenizer.eos_token_id).any(dim=1)
-        mixture_eos = (mixture_data["input_ids"][:, context_length:] == self.tokenizer.eos_token_id).any(dim=1)
+        model_eos = (model_data["input_ids"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)
+        mixture_eos = (mixture_data["input_ids"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)
         self.stats["val/model_contain_eos_token"].append(gather_mean(model_eos.float()))
         self.stats["val/ref_contain_eos_token"].append(gather_mean(mixture_eos.float()))
 
@@ -323,8 +334,8 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
         # Apply chat template and tokenize the input
         batch_size = len(next(iter(inputs.values())))
         inputs = [{k: v[i] for k, v in inputs.items()} for i in range(batch_size)]
-        inputs = [maybe_apply_chat_template(x, self.tokenizer) for x in inputs]
-        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.tokenizer) for x in inputs]
+        inputs = [maybe_apply_chat_template(x, self.processing_class) for x in inputs]
+        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.processing_class) for x in inputs]
         inputs = self.data_collator(inputs)
 
         # need the prompt_ only
diff --git a/trl/trainer/online_dpo_trainer.py b/trl/trainer/online_dpo_trainer.py
index f495f5af1c..95c1f826da 100644
--- a/trl/trainer/online_dpo_trainer.py
+++ b/trl/trainer/online_dpo_trainer.py
@@ -27,15 +27,18 @@
 from packaging import version
 from torch.utils.data import DataLoader, IterableDataset
 from transformers import (
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     GenerationConfig,
+    PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainerCallback,
     is_apex_available,
     is_wandb_available,
 )
-from transformers.modeling_utils import PreTrainedModel
 from transformers.trainer_utils import EvalPrediction, seed_worker
 from transformers.training_args import OptimizerNames
 from transformers.utils import is_peft_available, is_sagemaker_mp_enabled, logging
@@ -100,8 +103,10 @@ class OnlineDPOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         peft_config (`Dict`):
             The peft config to use for training.
         compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*):
@@ -127,7 +132,9 @@ def __init__(
         data_collator: Optional[DataCollator] = None,
         train_dataset: Optional[Union[Dataset, IterableDataset, "datasets.Dataset"]] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset], "datasets.Dataset"]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         peft_config: Optional[Dict] = None,
         compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -158,9 +165,9 @@ def __init__(
         if args is None:
             raise ValueError("`args` must be provided.")
 
-        # Check that the tokenizer is provided
-        if tokenizer is None:
-            raise ValueError("`tokenizer` must be provided.")
+        # Check that the processing_class is provided
+        if processing_class is None:
+            raise ValueError("`processing_class` must be provided.")
 
         # Convert to PEFT model if peft_config is provided
         if peft_config is not None:
@@ -202,7 +209,7 @@ def __init__(
 
         # Define the collator is not provided
         if data_collator is None:
-            data_collator = DPODataCollatorWithPadding(pad_token_id=tokenizer.pad_token_id)
+            data_collator = DPODataCollatorWithPadding(pad_token_id=processing_class.pad_token_id)
 
         self.stats = {
             "objective/kl": [],
@@ -236,7 +243,7 @@ def __init__(
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
             optimizers=optimizers,
@@ -366,8 +373,8 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
         # We do this on-the-fly to enable the use of reward models and policies with different tokenizers / chat templates.
         batch_size = len(next(iter(inputs.values())))
         inputs = [{k: v[i] for k, v in inputs.items()} for i in range(batch_size)]
-        inputs = [maybe_apply_chat_template(x, self.tokenizer) for x in inputs]
-        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.tokenizer) for x in inputs]
+        inputs = [maybe_apply_chat_template(x, self.processing_class) for x in inputs]
+        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.processing_class) for x in inputs]
         inputs = self.data_collator(inputs)
 
         # Sample 2 completations per prompt of size `max_new_tokens` from the model
@@ -385,7 +392,7 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
 
         completion_ids = output[:, context_length:]
         completion_ids, completion_mask = truncate_right(
-            completion_ids, self.tokenizer.eos_token_id, self.tokenizer.pad_token_id
+            completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id
         )
         prompt_completion_ids = torch.cat((prompt_ids, completion_ids), dim=1)
         prompt_completion_mask = torch.cat((prompt_mask, completion_mask), dim=1)
@@ -414,12 +421,12 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
 
             # Get the reward from the reward model
             _, scores, _ = get_reward(
-                self.reward_model, prompt_completion_ids, self.tokenizer.pad_token_id, context_length
+                self.reward_model, prompt_completion_ids, self.processing_class.pad_token_id, context_length
             )
 
         # Filter completion. Ensure that the sample contains stop_token_id
         # Completions not passing that filter will receive a lower score.
-        contain_eos_token = torch.any(completion_ids == self.tokenizer.eos_token_id, dim=-1)
+        contain_eos_token = torch.any(completion_ids == self.processing_class.eos_token_id, dim=-1)
         if self.args.missing_eos_penalty is not None:
             scores[~contain_eos_token] -= self.args.missing_eos_penalty
 
diff --git a/trl/trainer/orpo_trainer.py b/trl/trainer/orpo_trainer.py
index 7aa991ad93..8a41758d02 100644
--- a/trl/trainer/orpo_trainer.py
+++ b/trl/trainer/orpo_trainer.py
@@ -35,9 +35,12 @@
 from torch.utils.data import DataLoader
 from transformers import (
     AutoModelForCausalLM,
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     is_torch_xla_available,
     is_wandb_available,
@@ -89,8 +92,10 @@ class ORPOTrainer(Trainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         model_init (`Callable[[], transformers.PreTrainedModel]`):
             The model initializer to use for training. If None is specified, the default model initializer will be used.
         callbacks (`List[transformers.TrainerCallback]`):
@@ -115,7 +120,9 @@ def __init__(
         data_collator: Optional[DataCollator] = None,
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
         optimizers: Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),
@@ -222,8 +229,8 @@ def make_inputs_require_grad(module, input, output):
             self.decoder_start_token_id = model.config.decoder_start_token_id
             self.pad_token_id = model.config.pad_token_id
 
-        if tokenizer is None:
-            raise ValueError("tokenizer must be specified to tokenize a ORPO dataset.")
+        if processing_class is None:
+            raise ValueError("processing_class must be specified to tokenize a ORPO dataset.")
         if args.max_length is None:
             warnings.warn(
                 "`max_length` is not set in the ORPOConfig's init"
@@ -255,7 +262,7 @@ def make_inputs_require_grad(module, input, output):
 
         if data_collator is None:
             data_collator = DPODataCollatorWithPadding(
-                pad_token_id=tokenizer.pad_token_id,
+                pad_token_id=processing_class.pad_token_id,
                 label_pad_token_id=args.label_pad_token_id,
                 is_encoder_decoder=self.is_encoder_decoder,
             )
@@ -279,10 +286,10 @@ def make_inputs_require_grad(module, input, output):
         self.max_length = max_length
         self.generate_during_eval = args.generate_during_eval
         self.label_pad_token_id = args.label_pad_token_id
-        self.padding_value = args.padding_value if args.padding_value is not None else tokenizer.pad_token_id
+        self.padding_value = args.padding_value if args.padding_value is not None else processing_class.pad_token_id
         self.max_prompt_length = max_prompt_length
         self.truncation_mode = args.truncation_mode
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
 
         self.beta = args.beta
         self.aux_loss_enabled = getattr(model.config, "output_router_logits", False)
@@ -303,7 +310,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -359,8 +366,8 @@ def build_tokenized_answer(self, prompt, answer):
             https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
         """
 
-        full_tokenized = self.tokenizer(prompt + answer, add_special_tokens=False)
-        prompt_input_ids = self.tokenizer(prompt, add_special_tokens=False)["input_ids"]
+        full_tokenized = self.processing_class(prompt + answer, add_special_tokens=False)
+        prompt_input_ids = self.processing_class(prompt, add_special_tokens=False)["input_ids"]
 
         answer_input_ids = full_tokenized["input_ids"][len(prompt_input_ids) :]
         answer_attention_mask = full_tokenized["attention_mask"][len(prompt_input_ids) :]
@@ -425,7 +432,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             if not isinstance(prompt, str):
                 raise ValueError(f"prompt should be an str but got {type(prompt)}")
-            prompt_tokens = self.tokenizer(prompt, add_special_tokens=False)
+            prompt_tokens = self.processing_class(prompt, add_special_tokens=False)
             prompt_tokens = {f"prompt_{k}": v for k, v in prompt_tokens.items()}
 
             if not isinstance(chosen, str):
@@ -461,7 +468,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             # add BOS token to head of prompt. Avoid adding if it's already there
             prompt_tokens, chosen_tokens, rejected_tokens = add_bos_token_if_needed(
-                self.tokenizer.bos_token_id,
+                self.processing_class.bos_token_id,
                 prompt_len_input_ids,
                 prompt_tokens,
                 chosen_prompt_len_input_ids,
@@ -472,7 +479,7 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
 
             # add EOS token to end of answer. Avoid adding if it's already there
             chosen_tokens, rejected_tokens = add_eos_token_if_needed(
-                self.tokenizer.eos_token_id, chosen_tokens, rejected_tokens
+                self.processing_class.eos_token_id, chosen_tokens, rejected_tokens
             )
 
             longer_response_length = max(len(chosen_tokens["input_ids"]), len(rejected_tokens["input_ids"]))
@@ -522,13 +529,13 @@ def tokenize_row(self, feature, model: Optional[Union[PreTrainedModel, nn.Module
                     batch[f"{k}{type_key}"] = tokens
 
         else:
-            chosen_tokens = self.tokenizer(
+            chosen_tokens = self.processing_class(
                 chosen, truncation=True, max_length=self.max_completion_length, add_special_tokens=True
             )
-            rejected_tokens = self.tokenizer(
+            rejected_tokens = self.processing_class(
                 rejected, truncation=True, max_length=self.max_completion_length, add_special_tokens=True
             )
-            prompt_tokens = self.tokenizer(
+            prompt_tokens = self.processing_class(
                 prompt, truncation=True, max_length=self.max_prompt_length, add_special_tokens=True
             )
 
@@ -855,11 +862,11 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
                 attention_mask=batch["prompt_attention_mask"],
                 max_length=self.max_length,
                 do_sample=True,
-                pad_token_id=self.tokenizer.pad_token_id,
+                pad_token_id=self.processing_class.pad_token_id,
             )
 
-        policy_output = pad_to_length(policy_output, self.max_length, self.tokenizer.pad_token_id)
-        policy_output_decoded = self.tokenizer.batch_decode(policy_output, skip_special_tokens=True)
+        policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)
+        policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)
 
         return policy_output_decoded
 
diff --git a/trl/trainer/ppov2_trainer.py b/trl/trainer/ppov2_trainer.py
index e6fb9d70ef..2114cf60de 100644
--- a/trl/trainer/ppov2_trainer.py
+++ b/trl/trainer/ppov2_trainer.py
@@ -30,9 +30,12 @@
 from datasets import Dataset
 from torch.utils.data import DataLoader
 from transformers import (
+    BaseImageProcessor,
     DataCollatorWithPadding,
+    FeatureExtractionMixin,
     GenerationConfig,
-    PreTrainedTokenizer,
+    PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainerCallback,
     TrainerControl,
@@ -90,7 +93,9 @@ class PPOv2Trainer(Trainer):
     def __init__(
         self,
         config: PPOv2Config,
-        tokenizer: PreTrainedTokenizer,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ],
         policy: nn.Module,
         ref_policy: nn.Module,
         reward_model: nn.Module,
@@ -110,7 +115,7 @@ def __init__(
 
         self.args = config
         args = config
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
         self.policy = policy
 
         self.policy.generation_config.eos_token_id = (
@@ -169,7 +174,7 @@ def __init__(
         for module in [policy, ref_policy, value_model, reward_model]:
             disable_dropout_in_model(module)
         if args.stop_token and args.stop_token == "eos":
-            args.stop_token_id = tokenizer.eos_token_id
+            args.stop_token_id = processing_class.eos_token_id
         self.model = PolicyAndValueWrapper(policy, value_model)
         self.model.config = policy.config  # needed for pushing to hub
         self.create_optimizer_and_scheduler(
@@ -182,7 +187,7 @@ def __init__(
         default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)
         self.callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks
         self.callback_handler = CallbackHandler(
-            self.callbacks, self.model, self.tokenizer, self.optimizer, self.lr_scheduler
+            self.callbacks, self.model, self.processing_class, self.optimizer, self.lr_scheduler
         )
         self.add_callback(PrinterCallback if self.args.disable_tqdm else DEFAULT_PROGRESS_CALLBACK)
         self.control = TrainerControl()
@@ -215,7 +220,7 @@ def __init__(
             self.train_dataset,
             batch_size=self.local_dataloader_batch_size,
             shuffle=True,
-            collate_fn=DataCollatorWithPadding(tokenizer),
+            collate_fn=DataCollatorWithPadding(self.processing_class),
             drop_last=True,  # needed; otherwise the last batch will be of ragged shape
         )
         # sync random states for DataLoader(shuffle=True) before `accelerator.prepare`
@@ -227,7 +232,7 @@ def __init__(
         self.eval_dataloader = DataLoader(
             self.eval_dataset,
             batch_size=args.per_device_eval_batch_size,
-            collate_fn=DataCollatorWithPadding(self.tokenizer),
+            collate_fn=DataCollatorWithPadding(self.processing_class),
             drop_last=True,
         )  # no need to shuffle eval dataset
         self.eval_dataloader = accelerator.prepare(self.eval_dataloader)
@@ -271,7 +276,7 @@ def train(self):
         model = self.model
         ref_policy = self.ref_policy
         reward_model = self.reward_model
-        tokenizer = self.tokenizer
+        processing_class = self.processing_class
         dataloader = self.dataloader
         device = accelerator.device
 
@@ -346,7 +351,7 @@ def repeat_generator():
                         unwrapped_model.policy,
                         queries,
                         args.local_rollout_forward_batch_size,
-                        tokenizer.pad_token_id,
+                        processing_class.pad_token_id,
                         generation_config,
                     )
 
@@ -360,7 +365,7 @@ def repeat_generator():
                     del logits, all_logprob
                     torch.cuda.empty_cache()
 
-                    ref_output = forward(ref_policy, query_response, tokenizer.pad_token_id)
+                    ref_output = forward(ref_policy, query_response, processing_class.pad_token_id)
                     ref_logits = ref_output.logits[:, context_length - 1 : -1]
                     ref_logits /= args.temperature + 1e-7
                     ref_all_logprob = F.log_softmax(ref_logits, dim=-1)
@@ -372,19 +377,19 @@ def repeat_generator():
                     postprocessed_response = response
                     if args.stop_token_id is not None:  # handle the edge case when stop_token_id exists but is 0
                         postprocessed_response = truncate_response(
-                            args.stop_token_id, tokenizer.pad_token_id, response
+                            args.stop_token_id, processing_class.pad_token_id, response
                         )
 
                     # Response Processing 2. run reward model on the truncated responses
                     postprocessed_query_response = torch.cat((query, postprocessed_response), 1)
-                    sequence_length = first_true_indices(postprocessed_response == tokenizer.pad_token_id) - 1
+                    sequence_length = first_true_indices(postprocessed_response == processing_class.pad_token_id) - 1
                     unwrapped_value_model = accelerator.unwrap_model(model).value_model
                     full_value, _, _ = get_reward(
-                        unwrapped_value_model, query_response, tokenizer.pad_token_id, context_length
+                        unwrapped_value_model, query_response, processing_class.pad_token_id, context_length
                     )
                     value = full_value[:, context_length - 1 : -1].squeeze(-1)
                     _, score, _ = get_reward(
-                        reward_model, postprocessed_query_response, tokenizer.pad_token_id, context_length
+                        reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length
                     )
 
                     responses.append(response)
@@ -407,7 +412,7 @@ def repeat_generator():
 
                 # Response Processing 3. Filter completion. Ensure that the sample contains stop_token_id
                 # Completions not passing that filter will receive a lower score.
-                contain_eos_token = torch.any(postprocessed_responses == self.tokenizer.eos_token_id, dim=-1)
+                contain_eos_token = torch.any(postprocessed_responses == self.processing_class.eos_token_id, dim=-1)
                 if self.args.missing_eos_penalty is not None:
                     scores[~contain_eos_token] -= self.args.missing_eos_penalty
                 # accelerator.print(f"{scores=}, {(contain_eos_token.sum() / len(contain_eos_token))=}")
@@ -468,7 +473,7 @@ def repeat_generator():
                             mb_return = returns[micro_batch_inds]
                             mb_values = values[micro_batch_inds]
 
-                            output, vpred_temp = forward(model, mb_query_responses, tokenizer.pad_token_id)
+                            output, vpred_temp = forward(model, mb_query_responses, processing_class.pad_token_id)
                             logits = output.logits[:, context_length - 1 : -1]
                             logits /= args.temperature + 1e-7
                             new_all_logprobs = F.log_softmax(logits, dim=-1)
@@ -551,7 +556,7 @@ def repeat_generator():
                 metrics["policy/entropy_avg"] = self.accelerator.gather(entropy_stats).mean().item()
                 metrics["val/ratio"] = self.accelerator.gather(ratio_stats).mean().item()
                 metrics["val/ratio_var"] = self.accelerator.gather(ratio_stats).var().item()
-                metrics["val/num_eos_tokens"] = (responses == tokenizer.eos_token_id).sum().item()
+                metrics["val/num_eos_tokens"] = (responses == processing_class.eos_token_id).sum().item()
                 metrics["lr"] = self.lr_scheduler.get_last_lr()[0]
                 metrics["episode"] = self.state.episode
                 self.state.epoch = self.state.episode / self.train_dataset_len  # used by self.log
@@ -599,7 +604,7 @@ def repeat_generator():
 
     def generate_completions(self, sampling: bool = False):
         args = self.args
-        tokenizer = self.tokenizer
+        processing_class = self.processing_class
         generation_config = GenerationConfig(
             max_new_tokens=self.args.response_length,
             temperature=(0.01 + 1e-7),
@@ -618,21 +623,25 @@ def generate_completions(self, sampling: bool = False):
                         unwrapped_model.policy,
                         query,
                         query.shape[0],
-                        tokenizer.pad_token_id,
+                        processing_class.pad_token_id,
                         generation_config,
                     )
                     response = query_response[:, context_length:]
                     postprocessed_response = response
                     if args.stop_token_id is not None:  # handle the edge case when stop_token_id exists but is 0
                         postprocessed_response = truncate_response(
-                            args.stop_token_id, tokenizer.pad_token_id, response
+                            args.stop_token_id, processing_class.pad_token_id, response
                         )
-                    table["query"].extend(gather_object(tokenizer.batch_decode(query, skip_special_tokens=True)))
-                    table["model response"].extend(gather_object(tokenizer.batch_decode(postprocessed_response)))
+                    table["query"].extend(
+                        gather_object(processing_class.batch_decode(query, skip_special_tokens=True))
+                    )
+                    table["model response"].extend(
+                        gather_object(processing_class.batch_decode(postprocessed_response))
+                    )
 
                     postprocessed_query_response = torch.cat((query, postprocessed_response), 1)
                     _, score, _ = get_reward(
-                        self.reward_model, postprocessed_query_response, tokenizer.pad_token_id, context_length
+                        self.reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length
                     )
                     table["score"].extend(self.accelerator.gather(score).float().cpu().numpy())
 
diff --git a/trl/trainer/reward_trainer.py b/trl/trainer/reward_trainer.py
index c1cf590372..64f873ef34 100644
--- a/trl/trainer/reward_trainer.py
+++ b/trl/trainer/reward_trainer.py
@@ -25,9 +25,12 @@
 from accelerate.utils import gather_object
 from datasets import Dataset
 from transformers import (
+    BaseImageProcessor,
     DataCollator,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainingArguments,
     is_wandb_available,
@@ -84,7 +87,9 @@ def __init__(
         data_collator: Optional[DataCollator] = None,
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -111,8 +116,10 @@ def __init__(
                 The dataset to use for training.
             eval_dataset (`datasets.Dataset`):
                 The dataset to use for evaluation.
-            tokenizer (`transformers.PreTrainedTokenizerBase`):
-                The tokenizer to use for training. This argument is required if you want to use the default data collator.
+            processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+                Processing class used to process the data. If provided, will be used to automatically process the inputs
+                for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+                reuse the fine-tuned model.
             model_init (`Callable[[], transformers.PreTrainedModel]`):
                 The model initializer to use for training. If None is specified, the default model initializer will be used.
             compute_metrics (`Callable[[transformers.EvalPrediction], Dict]`, *optional* defaults to `compute_accuracy`):
@@ -175,12 +182,14 @@ def __init__(
             compute_metrics = compute_accuracy
 
         if data_collator is None:
-            if tokenizer is None:
-                raise ValueError("A tokenizer must be specified when using the default RewardDataCollatorWithPadding")
+            if processing_class is None:
+                raise ValueError(
+                    "A processing_class must be specified when using the default RewardDataCollatorWithPadding"
+                )
             if max_length is None:
                 max_length = 512 if type(args) is TrainingArguments or args.max_length is None else args.max_length
 
-            data_collator = RewardDataCollatorWithPadding(tokenizer)
+            data_collator = RewardDataCollatorWithPadding(processing_class)
 
             if args.remove_unused_columns:
                 try:  # for bc before https://github.com/huggingface/transformers/pull/25435
@@ -200,8 +209,8 @@ def __init__(
 
         if "input_ids_chosen" not in train_dataset.column_names:
             with PartialState().local_main_process_first():
-                fn_kwargs = {"tokenizer": tokenizer}
-                train_dataset = train_dataset.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer})
+                fn_kwargs = {"tokenizer": processing_class}
+                train_dataset = train_dataset.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": processing_class})
                 train_dataset = train_dataset.map(
                     _tokenize,
                     batched=True,
@@ -216,7 +225,9 @@ def __init__(
                     num_proc=args.dataset_num_proc,
                 )
                 if eval_dataset is not None:
-                    eval_dataset = eval_dataset.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer})
+                    eval_dataset = eval_dataset.map(
+                        maybe_apply_chat_template, fn_kwargs={"tokenizer": processing_class}
+                    )
                     eval_dataset = eval_dataset.map(
                         _tokenize,
                         fn_kwargs=fn_kwargs,
@@ -238,7 +249,7 @@ def __init__(
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -337,8 +348,8 @@ def visualize_samples(self, num_print_samples: int):
         table = defaultdict(list)
         for _, inputs in enumerate(eval_dataloader):
             _, logits, _ = self.prediction_step(self.model, inputs, prediction_loss_only=False)
-            chosen_text = decode_and_strip_padding(inputs["input_ids_chosen"], self.tokenizer)
-            rejected_text = decode_and_strip_padding(inputs["input_ids_rejected"], self.tokenizer)
+            chosen_text = decode_and_strip_padding(inputs["input_ids_chosen"], self.processing_class)
+            rejected_text = decode_and_strip_padding(inputs["input_ids_rejected"], self.processing_class)
             table["chosen_text"].extend(gather_object(chosen_text))
             table["rejected_text"].extend(gather_object(rejected_text))
             table["logits"].extend(
diff --git a/trl/trainer/rloo_trainer.py b/trl/trainer/rloo_trainer.py
index 516eeb4dfb..2532fa9167 100644
--- a/trl/trainer/rloo_trainer.py
+++ b/trl/trainer/rloo_trainer.py
@@ -30,9 +30,12 @@
 from datasets import Dataset
 from torch.utils.data import DataLoader
 from transformers import (
+    BaseImageProcessor,
     DataCollatorWithPadding,
+    FeatureExtractionMixin,
     GenerationConfig,
-    PreTrainedTokenizer,
+    PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     TrainerCallback,
     TrainerControl,
@@ -71,7 +74,9 @@ class RLOOTrainer(Trainer):
     def __init__(
         self,
         config: RLOOConfig,
-        tokenizer: PreTrainedTokenizer,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ],
         policy: nn.Module,
         ref_policy: nn.Module,
         reward_model: nn.Module,
@@ -90,7 +95,7 @@ def __init__(
 
         self.args = config
         args = config
-        self.tokenizer = tokenizer
+        self.processing_class = processing_class
         self.policy = policy
 
         self.policy.generation_config.eos_token_id = (
@@ -144,7 +149,7 @@ def __init__(
         for module in [policy, ref_policy, reward_model]:
             disable_dropout_in_model(module)
         if args.stop_token and args.stop_token == "eos":
-            args.stop_token_id = tokenizer.eos_token_id
+            args.stop_token_id = self.processing_class.eos_token_id
         self.model = policy
         self.create_optimizer_and_scheduler(
             num_training_steps=args.num_total_batches
@@ -160,7 +165,7 @@ def __init__(
         default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)
         self.callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks
         self.callback_handler = CallbackHandler(
-            self.callbacks, self.model, self.tokenizer, self.optimizer, self.lr_scheduler
+            self.callbacks, self.model, self.processing_class, self.optimizer, self.lr_scheduler
         )
         self.add_callback(PrinterCallback if self.args.disable_tqdm else DEFAULT_PROGRESS_CALLBACK)
         self.control = TrainerControl()
@@ -187,7 +192,7 @@ def __init__(
             self.train_dataset,
             batch_size=self.local_dataloader_batch_size,
             shuffle=True,
-            collate_fn=DataCollatorWithPadding(tokenizer),
+            collate_fn=DataCollatorWithPadding(self.processing_class),
             drop_last=True,  # needed; otherwise the last batch will be of ragged shape
         )
         # sync random states for DataLoader(shuffle=True) before `accelerator.prepare`
@@ -199,7 +204,7 @@ def __init__(
         self.eval_dataloader = DataLoader(
             self.eval_dataset,
             batch_size=args.per_device_eval_batch_size,
-            collate_fn=DataCollatorWithPadding(self.tokenizer),
+            collate_fn=DataCollatorWithPadding(self.processing_class),
             drop_last=True,
         )  # no need to shuffle eval dataset
         self.eval_dataloader = accelerator.prepare(self.eval_dataloader)
@@ -230,7 +235,7 @@ def train(self):
         self.model_wrapped = self.model
         ref_policy = self.ref_policy
         reward_model = self.reward_model
-        tokenizer = self.tokenizer
+        processing_class = self.processing_class
         dataloader = self.dataloader
         device = accelerator.device
 
@@ -301,7 +306,7 @@ def repeat_generator():
                         unwrapped_model,
                         queries,
                         args.local_rollout_forward_batch_size,
-                        tokenizer.pad_token_id,
+                        processing_class.pad_token_id,
                         generation_config,
                     )
 
@@ -315,7 +320,7 @@ def repeat_generator():
                     del logits, all_logprob
                     torch.cuda.empty_cache()
 
-                    ref_output = forward(ref_policy, query_response, tokenizer.pad_token_id)
+                    ref_output = forward(ref_policy, query_response, processing_class.pad_token_id)
                     ref_logits = ref_output.logits[:, context_length - 1 : -1]
                     ref_logits /= args.temperature + 1e-7
                     ref_all_logprob = F.log_softmax(ref_logits, dim=-1)
@@ -327,14 +332,14 @@ def repeat_generator():
                     postprocessed_response = response
                     if args.stop_token_id is not None:  # handle the edge case when stop_token_id exists but is 0
                         postprocessed_response = truncate_response(
-                            args.stop_token_id, tokenizer.pad_token_id, response
+                            args.stop_token_id, processing_class.pad_token_id, response
                         )
 
                     # Response Processing 2. run reward model on the truncated responses
                     postprocessed_query_response = torch.cat((query, postprocessed_response), 1)
-                    sequence_length = first_true_indices(postprocessed_response == tokenizer.pad_token_id) - 1
+                    sequence_length = first_true_indices(postprocessed_response == processing_class.pad_token_id) - 1
                     _, score, _ = get_reward(
-                        reward_model, postprocessed_query_response, tokenizer.pad_token_id, context_length
+                        reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length
                     )
 
                     responses.append(response)
@@ -356,7 +361,7 @@ def repeat_generator():
                 # Response Processing 3. filter response. Ensure that the sample contains stop_token_id
                 # responses not passing that filter will receive a low (fixed) score
                 # only query humans on responses that pass that filter
-                contain_eos_token = torch.any(postprocessed_responses == tokenizer.eos_token_id, dim=-1)
+                contain_eos_token = torch.any(postprocessed_responses == processing_class.eos_token_id, dim=-1)
                 if args.missing_eos_penalty is not None:
                     scores[~contain_eos_token] -= self.args.missing_eos_penalty
                 # accelerator.print(f"{scores=}, {(contain_eos_token.sum() / len(contain_eos_token))=}")
@@ -396,7 +401,7 @@ def repeat_generator():
                             mb_query_responses = query_responses[micro_batch_inds]
                             mb_logprobs = logprobs[micro_batch_inds]
 
-                            output = forward(model, mb_query_responses, tokenizer.pad_token_id)
+                            output = forward(model, mb_query_responses, processing_class.pad_token_id)
                             logits = output.logits[:, context_length - 1 : -1]
                             logits /= args.temperature + 1e-7
                             new_all_logprobs = F.log_softmax(logits, dim=-1)
@@ -462,7 +467,7 @@ def repeat_generator():
                 metrics["policy/entropy_avg"] = self.accelerator.gather(entropy_stats).mean().item()
                 metrics["val/ratio"] = self.accelerator.gather(ratio_stats).mean().item()
                 metrics["val/ratio_var"] = self.accelerator.gather(ratio_stats).var().item()
-                metrics["val/num_eos_tokens"] = (responses == tokenizer.eos_token_id).sum().item()
+                metrics["val/num_eos_tokens"] = (responses == processing_class.eos_token_id).sum().item()
                 metrics["lr"] = self.lr_scheduler.get_last_lr()[0]
                 metrics["episode"] = self.state.episode
                 self.state.epoch = self.state.episode / self.train_dataset_len  # used by self.log
@@ -489,7 +494,7 @@ def repeat_generator():
 
     def generate_completions(self, sampling: bool = False):
         args = self.args
-        tokenizer = self.tokenizer
+        processing_class = self.processing_class
         generation_config = GenerationConfig(
             max_new_tokens=self.args.response_length,
             temperature=(0.01 + 1e-7),
@@ -508,21 +513,25 @@ def generate_completions(self, sampling: bool = False):
                         unwrapped_model,
                         query,
                         query.shape[0],
-                        tokenizer.pad_token_id,
+                        processing_class.pad_token_id,
                         generation_config,
                     )
                     response = query_response[:, context_length:]
                     postprocessed_response = response
                     if args.stop_token_id is not None:  # handle the edge case when stop_token_id exists but is 0
                         postprocessed_response = truncate_response(
-                            args.stop_token_id, tokenizer.pad_token_id, response
+                            args.stop_token_id, processing_class.pad_token_id, response
                         )
-                    table["query"].extend(gather_object(tokenizer.batch_decode(query, skip_special_tokens=True)))
-                    table["model response"].extend(gather_object(tokenizer.batch_decode(postprocessed_response)))
+                    table["query"].extend(
+                        gather_object(processing_class.batch_decode(query, skip_special_tokens=True))
+                    )
+                    table["model response"].extend(
+                        gather_object(processing_class.batch_decode(postprocessed_response))
+                    )
 
                     postprocessed_query_response = torch.cat((query, postprocessed_response), 1)
                     _, score, _ = get_reward(
-                        self.reward_model, postprocessed_query_response, tokenizer.pad_token_id, context_length
+                        self.reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length
                     )
                     table["score"].extend(self.accelerator.gather(score).float().cpu().numpy())
 
diff --git a/trl/trainer/sft_trainer.py b/trl/trainer/sft_trainer.py
index b18615e57c..6875f95a7b 100644
--- a/trl/trainer/sft_trainer.py
+++ b/trl/trainer/sft_trainer.py
@@ -29,16 +29,20 @@
 from transformers import (
     AutoModelForCausalLM,
     AutoTokenizer,
+    BaseImageProcessor,
     DataCollator,
     DataCollatorForLanguageModeling,
+    FeatureExtractionMixin,
     PreTrainedModel,
     PreTrainedTokenizerBase,
+    ProcessorMixin,
     Trainer,
     is_wandb_available,
 )
 from transformers.trainer_callback import TrainerCallback
 from transformers.trainer_utils import EvalPrediction
 from transformers.utils import is_peft_available
+from transformers.utils.deprecation import deprecate_kwarg
 
 from ..extras.dataset_formatting import get_formatting_func_from_dataset
 from ..import_utils import is_liger_kernel_available
@@ -82,8 +86,11 @@ class SFTTrainer(Trainer):
             The dataset to use for training. We recommend users to use `trl.trainer.ConstantLengthDataset` to create their dataset.
         eval_dataset (Optional[Union[`datasets.Dataset`, Dict[`str`, `datasets.Dataset`]]]):
             The dataset to use for evaluation. We recommend users to use `trl.trainer.ConstantLengthDataset` to create their dataset.
-        tokenizer (`Optional[transformers.PreTrainedTokenizer]`):
-            The tokenizer to use for training. If not specified, the tokenizer associated to the model will be used.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
+            This supercedes the `tokenizer` argument, which is now deprecated.
         model_init (`Callable[[], transformers.PreTrainedModel]`):
             The model initializer to use for training. If None is specified, the default model initializer will be used.
         compute_metrics (`Callable[[transformers.EvalPrediction], Dict]`, *optional* defaults to None):
@@ -120,6 +127,7 @@ class SFTTrainer(Trainer):
         ],
         custom_message="Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.",
     )
+    @deprecate_kwarg("tokenizer", new_name="processing_class", version="0.14.0", raise_if_both_names=True)
     def __init__(
         self,
         model: Optional[Union[PreTrainedModel, nn.Module, str]] = None,
@@ -127,7 +135,9 @@ def __init__(
         data_collator: Optional[DataCollator] = None,  # type: ignore
         train_dataset: Optional[Dataset] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         model_init: Optional[Callable[[], PreTrainedModel]] = None,
         compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -284,10 +294,10 @@ def make_inputs_require_grad(module, input, output):
                 ):
                     peft_module_casting_to_bf16(model)
 
-        if tokenizer is None:
-            tokenizer = AutoTokenizer.from_pretrained(model.config._name_or_path)
-            if getattr(tokenizer, "pad_token", None) is None:
-                tokenizer.pad_token = tokenizer.eos_token
+        if processing_class is None:
+            processing_class = AutoTokenizer.from_pretrained(model.config._name_or_path)
+            if getattr(processing_class, "pad_token", None) is None:
+                processing_class.pad_token = processing_class.eos_token
 
         if max_seq_length is not None:
             warnings.warn(
@@ -297,7 +307,7 @@ def make_inputs_require_grad(module, input, output):
 
         if args.max_seq_length is None:
             # to overcome some issues with broken tokenizers
-            args.max_seq_length = min(tokenizer.model_max_length, 1024)
+            args.max_seq_length = min(processing_class.model_max_length, 1024)
 
             warnings.warn(
                 f"You didn't pass a `max_seq_length` argument to the SFTTrainer, this will default to {args.max_seq_length}"
@@ -334,14 +344,14 @@ def make_inputs_require_grad(module, input, output):
         if formatting_func is None:
             # check if dataset has ChatML format or instruction format and is supported
             # if not stays None
-            formatting_func = get_formatting_func_from_dataset(train_dataset, tokenizer)
+            formatting_func = get_formatting_func_from_dataset(train_dataset, processing_class)
             # if a template is detected, we don't need to add special tokens again
             if formatting_func is not None:
                 args.dataset_kwargs["add_special_tokens"] = False
 
         if not args.packing:
             if data_collator is None:
-                data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
+                data_collator = DataCollatorForLanguageModeling(tokenizer=processing_class, mlm=False)
 
         if num_of_sequences is not None:
             warnings.warn(
@@ -360,7 +370,7 @@ def make_inputs_require_grad(module, input, output):
             if train_dataset is not None:
                 train_dataset = self._prepare_dataset(
                     train_dataset,
-                    tokenizer,
+                    processing_class,
                     args.packing,
                     args.dataset_text_field,
                     args.max_seq_length,
@@ -379,7 +389,7 @@ def make_inputs_require_grad(module, input, output):
                 for _eval_dataset_name, _eval_dataset in _eval_datasets.items():
                     _eval_datasets[_eval_dataset_name] = self._prepare_dataset(
                         _eval_dataset,
-                        tokenizer,
+                        processing_class,
                         eval_packing,
                         args.dataset_text_field,
                         args.max_seq_length,
@@ -392,10 +402,10 @@ def make_inputs_require_grad(module, input, output):
                 if not _multiple:
                     eval_dataset = _eval_datasets["singleton"]
 
-        if tokenizer.padding_side is not None and tokenizer.padding_side != "right":
+        if processing_class.padding_side is not None and processing_class.padding_side != "right":
             warnings.warn(
-                "You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to "
-                "overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code."
+                "You passed a processing_class with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to "
+                "overflow issues when training a model in half-precision. You might consider adding `processing_class.padding_side = 'right'` to your code."
             )
 
         super().__init__(
@@ -404,7 +414,7 @@ def make_inputs_require_grad(module, input, output):
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             model_init=model_init,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -443,7 +453,7 @@ def push_to_hub(
     def _prepare_dataset(
         self,
         dataset,
-        tokenizer,
+        processing_class,
         packing,
         dataset_text_field: str,
         max_seq_length,
@@ -483,7 +493,7 @@ def _prepare_dataset(
 
         if not packing:
             return self._prepare_non_packed_dataloader(
-                tokenizer,
+                processing_class,
                 dataset,
                 dataset_text_field,
                 max_seq_length,
@@ -494,7 +504,7 @@ def _prepare_dataset(
 
         else:
             return self._prepare_packed_dataloader(
-                tokenizer,
+                processing_class,
                 dataset,
                 dataset_text_field,
                 max_seq_length,
@@ -507,7 +517,7 @@ def _prepare_dataset(
 
     def _prepare_non_packed_dataloader(
         self,
-        tokenizer,
+        processing_class,
         dataset,
         dataset_text_field: str,
         max_seq_length,
@@ -517,7 +527,7 @@ def _prepare_non_packed_dataloader(
     ):
         # Inspired from: https://huggingface.co/learn/nlp-course/chapter7/6?fw=pt
         def tokenize(element):
-            outputs = tokenizer(
+            outputs = processing_class(
                 element[dataset_text_field] if formatting_func is None else formatting_func(element),
                 add_special_tokens=add_special_tokens,
                 truncation=True,
@@ -560,7 +570,7 @@ def tokenize(element):
 
     def _prepare_packed_dataloader(
         self,
-        tokenizer,
+        processing_class,
         dataset,
         dataset_text_field: str,
         max_seq_length,
@@ -570,11 +580,11 @@ def _prepare_packed_dataloader(
         append_concat_token=True,
         add_special_tokens=True,
     ):
-        if tokenizer is None:
-            raise ValueError("You need to pass a tokenizer with `SFTTrainer`.")
+        if processing_class is None:
+            raise ValueError("You need to pass a processing_class with `SFTTrainer`.")
 
         constant_length_iterator = ConstantLengthDataset(
-            tokenizer,
+            processing_class,
             dataset,
             dataset_text_field=None if formatting_func is not None else dataset_text_field,
             formatting_func=formatting_func,
@@ -582,7 +592,7 @@ def _prepare_packed_dataloader(
             infinite=False,
             num_of_sequences=num_of_sequences,
             chars_per_token=chars_per_token,
-            eos_token_id=tokenizer.eos_token_id,
+            eos_token_id=processing_class.eos_token_id,
             append_concat_token=append_concat_token,
             add_special_tokens=add_special_tokens,
         )
diff --git a/trl/trainer/xpo_trainer.py b/trl/trainer/xpo_trainer.py
index 4fe778fb64..581acbf3db 100644
--- a/trl/trainer/xpo_trainer.py
+++ b/trl/trainer/xpo_trainer.py
@@ -20,8 +20,16 @@
 import torch.nn as nn
 import torch.nn.functional as F
 from datasets import Dataset, IterableDataset
-from transformers import PreTrainedTokenizerBase, TrainerCallback, is_apex_available, is_wandb_available
-from transformers.modeling_utils import PreTrainedModel
+from transformers import (
+    BaseImageProcessor,
+    FeatureExtractionMixin,
+    PreTrainedModel,
+    PreTrainedTokenizerBase,
+    ProcessorMixin,
+    TrainerCallback,
+    is_apex_available,
+    is_wandb_available,
+)
 from transformers.trainer_utils import EvalPrediction
 from transformers.training_args import OptimizerNames
 
@@ -63,8 +71,10 @@ class XPOTrainer(OnlineDPOTrainer):
             The dataset to use for training.
         eval_dataset (`datasets.Dataset`):
             The dataset to use for evaluation.
-        tokenizer (`transformers.PreTrainedTokenizerBase`):
-            The tokenizer to use for training. This argument is required if you want to use the default data collator.
+        processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*):
+            Processing class used to process the data. If provided, will be used to automatically process the inputs
+            for the model, and it will be saved along the model to make it easier to rerun an interrupted training or
+            reuse the fine-tuned model.
         peft_config (`Dict`):
             The peft config to use for training.
         compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*):
@@ -89,7 +99,9 @@ def __init__(
         data_collator: Optional[Callable] = None,
         train_dataset: Optional[Union[Dataset, IterableDataset]] = None,
         eval_dataset: Optional[Union[Dataset, Dict[str, Dataset]]] = None,
-        tokenizer: Optional[PreTrainedTokenizerBase] = None,
+        processing_class: Optional[
+            Union[PreTrainedTokenizerBase, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin]
+        ] = None,
         peft_config: Optional[Dict] = None,
         compute_metrics: Optional[Callable[[EvalPrediction], Dict]] = None,
         callbacks: Optional[List[TrainerCallback]] = None,
@@ -104,7 +116,7 @@ def __init__(
             data_collator=data_collator,
             train_dataset=train_dataset,
             eval_dataset=eval_dataset,
-            tokenizer=tokenizer,
+            processing_class=processing_class,
             peft_config=peft_config,
             compute_metrics=compute_metrics,
             callbacks=callbacks,
@@ -171,7 +183,7 @@ def _process_completions(self, model_output, ref_output, prompts):
         # Process model completions
         model_completion_ids = model_output[:, context_length:]
         model_completion_ids, model_completion_mask = truncate_right(
-            model_completion_ids, self.tokenizer.eos_token_id, self.tokenizer.pad_token_id
+            model_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id
         )
         model_data = {
             "input_ids": torch.cat((prompts["input_ids"], model_completion_ids), dim=1),
@@ -181,7 +193,7 @@ def _process_completions(self, model_output, ref_output, prompts):
         # Process reference model completions
         ref_completion_ids = ref_output[:, context_length:]
         ref_completion_ids, ref_completion_mask = truncate_right(
-            ref_completion_ids, self.tokenizer.eos_token_id, self.tokenizer.pad_token_id
+            ref_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id
         )
         ref_data = {
             "input_ids": torch.cat((prompts["input_ids"], ref_completion_ids), dim=1),
@@ -193,16 +205,16 @@ def _process_completions(self, model_output, ref_output, prompts):
     def _compute_rewards(self, model_data, ref_data, context_length):
         with torch.no_grad():
             _, model_scores, _ = get_reward(
-                self.reward_model, model_data["input_ids"], self.tokenizer.pad_token_id, context_length
+                self.reward_model, model_data["input_ids"], self.processing_class.pad_token_id, context_length
             )
             _, ref_scores, _ = get_reward(
-                self.reward_model, ref_data["input_ids"], self.tokenizer.pad_token_id, context_length
+                self.reward_model, ref_data["input_ids"], self.processing_class.pad_token_id, context_length
             )
 
         # Apply EOS penalty if needed
         if self.args.missing_eos_penalty is not None:
-            model_contain_eos = torch.any(model_data["input_ids"] == self.tokenizer.eos_token_id, dim=-1)
-            ref_contain_eos = torch.any(ref_data["input_ids"] == self.tokenizer.eos_token_id, dim=-1)
+            model_contain_eos = torch.any(model_data["input_ids"] == self.processing_class.eos_token_id, dim=-1)
+            ref_contain_eos = torch.any(ref_data["input_ids"] == self.processing_class.eos_token_id, dim=-1)
             model_scores[~model_contain_eos] -= self.args.missing_eos_penalty
             ref_scores[~ref_contain_eos] -= self.args.missing_eos_penalty
 
@@ -356,8 +368,8 @@ def gather_mean(tensor):
         self.stats["rewards/accuracies"].append(gather_mean(accuracy.mean()))
 
         # Log EOS token statistics
-        model_eos = (model_data["input_ids"][:, context_length:] == self.tokenizer.eos_token_id).any(dim=1)
-        ref_eos = (ref_data["input_ids"][:, context_length:] == self.tokenizer.eos_token_id).any(dim=1)
+        model_eos = (model_data["input_ids"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)
+        ref_eos = (ref_data["input_ids"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)
         self.stats["val/model_contain_eos_token"].append(gather_mean(model_eos.float()))
         self.stats["val/ref_contain_eos_token"].append(gather_mean(ref_eos.float()))
 
@@ -371,8 +383,8 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
         # Apply chat template and tokenize the input
         batch_size = len(next(iter(inputs.values())))
         inputs = [{k: v[i] for k, v in inputs.items()} for i in range(batch_size)]
-        inputs = [maybe_apply_chat_template(x, self.tokenizer) for x in inputs]
-        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.tokenizer) for x in inputs]
+        inputs = [maybe_apply_chat_template(x, self.processing_class) for x in inputs]
+        inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.processing_class) for x in inputs]
         inputs = self.data_collator(inputs)
 
         # need the prompt_ only


Test patch:
diff --git a/tests/slow/test_dpo_slow.py b/tests/slow/test_dpo_slow.py
index 96e7af086d..e7d25c3d1c 100644
--- a/tests/slow/test_dpo_slow.py
+++ b/tests/slow/test_dpo_slow.py
@@ -85,7 +85,7 @@ def test_dpo_bare_model(self, model_id, loss_type, pre_compute_logits):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.dataset,
                 eval_dataset=self.dataset,
             )
@@ -142,7 +142,7 @@ def test_dpo_peft_model(self, model_id, loss_type, pre_compute_logits, gradient_
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.dataset,
                 eval_dataset=self.dataset,
                 peft_config=self.peft_config,
@@ -206,7 +206,7 @@ def test_dpo_peft_model_qlora(self, model_id, loss_type, pre_compute_logits, gra
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.dataset,
                 eval_dataset=self.dataset,
                 peft_config=self.peft_config,
diff --git a/tests/slow/test_sft_slow.py b/tests/slow/test_sft_slow.py
index 4de673d756..5035de4848 100644
--- a/tests/slow/test_sft_slow.py
+++ b/tests/slow/test_sft_slow.py
@@ -108,7 +108,7 @@ def test_sft_trainer_transformers(self, model_name, packing):
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
             )
@@ -142,7 +142,7 @@ def test_sft_trainer_peft(self, model_name, packing):
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
                 peft_config=self.peft_config,
@@ -178,7 +178,7 @@ def test_sft_trainer_transformers_mp(self, model_name, packing):
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
             )
@@ -213,7 +213,7 @@ def test_sft_trainer_transformers_mp_gc(self, model_name, packing, gradient_chec
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
             )
@@ -249,7 +249,7 @@ def test_sft_trainer_transformers_mp_gc_peft(self, model_name, packing, gradient
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
                 peft_config=self.peft_config,
@@ -292,7 +292,7 @@ def test_sft_trainer_transformers_mp_gc_device_map(
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
             )
@@ -331,7 +331,7 @@ def test_sft_trainer_transformers_mp_gc_peft_qlora(self, model_name, packing, gr
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=self.train_dataset,
                 eval_dataset=self.eval_dataset,
                 peft_config=self.peft_config,
@@ -375,7 +375,7 @@ def test_sft_trainer_with_chat_format_qlora(self, model_name, packing):
             trainer = SFTTrainer(
                 model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=train_dataset,
                 peft_config=self.peft_config,
             )
diff --git a/tests/test_bco_trainer.py b/tests/test_bco_trainer.py
index 6aea7e993c..6c86c714e5 100644
--- a/tests/test_bco_trainer.py
+++ b/tests/test_bco_trainer.py
@@ -85,7 +85,7 @@ def test_bco_trainer(self, name, pre_compute, eval_dataset, config_name):
                 model=model,
                 ref_model=ref_model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"] if eval_dataset else None,
             )
@@ -119,7 +119,7 @@ def test_bco_trainer_with_ref_model_is_model(self):
                     model=self.model,
                     ref_model=self.model,  # ref_model can't be the same as model
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                 )
 
@@ -142,7 +142,7 @@ def test_tokenize_and_process_tokens(self):
                 model=self.model,
                 ref_model=self.ref_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -204,7 +204,7 @@ def test_bco_trainer_without_providing_ref_model(self):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -249,7 +249,7 @@ def embed_prompt(input_ids, attention_mask, model):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 embedding_func=embedding_func,
@@ -299,7 +299,7 @@ def test_bco_trainer_without_providing_ref_model_with_lora(self):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -345,7 +345,7 @@ def test_bco_trainer_generate_during_eval_no_wandb(self):
                     model=self.model,
                     ref_model=None,
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                 )
@@ -385,7 +385,7 @@ def test_bco_lora_save(self):
                 model=model_peft,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
diff --git a/tests/test_cpo_trainer.py b/tests/test_cpo_trainer.py
index 15f39a630b..7bbccbb248 100644
--- a/tests/test_cpo_trainer.py
+++ b/tests/test_cpo_trainer.py
@@ -75,7 +75,7 @@ def test_cpo_trainer(self, name, loss_type, config_name):
             trainer = CPOTrainer(
                 model=model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -132,7 +132,7 @@ def test_cpo_trainer_with_lora(self, config_name):
             trainer = CPOTrainer(
                 model=self.model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
diff --git a/tests/test_dpo_trainer.py b/tests/test_dpo_trainer.py
index 8d857c84ba..da36ce0f86 100644
--- a/tests/test_dpo_trainer.py
+++ b/tests/test_dpo_trainer.py
@@ -263,7 +263,7 @@ def test_dpo_trainer(self, name, loss_type, pre_compute):
                 model=model,
                 ref_model=ref_model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -309,7 +309,7 @@ def test_dpo_trainer_without_providing_ref_model(self, rpo_alpha, _):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -343,7 +343,7 @@ def test_dpo_trainer_with_ref_model_is_model(self):
                     model=self.model,
                     ref_model=self.model,  # ref_model can't be the same as model
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                 )
 
@@ -379,7 +379,7 @@ def test_dpo_trainer_without_providing_ref_model_with_lora(self):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -428,7 +428,7 @@ def test_dpo_trainer_padding_token_is_none(self):
                     model=self.model,
                     ref_model=None,
                     args=training_args,
-                    tokenizer=tokenizer,
+                    processing_class=tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                 )
@@ -465,7 +465,7 @@ def test_dpo_trainer_w_dataset_num_proc(self):
                     model=self.model,
                     ref_model=None,
                     args=training_args,
-                    tokenizer=tokenizer,
+                    processing_class=tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                 )
@@ -494,9 +494,8 @@ def test_tr_dpo_trainer(self):
             trainer = DPOTrainer(
                 model=self.model,
                 ref_model=self.ref_model,
-                beta=0.1,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -542,7 +541,7 @@ def test_dpo_trainer_generate_during_eval_no_wandb(self):
                     model=self.model,
                     ref_model=None,
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                 )
@@ -584,7 +583,7 @@ def test_dpo_lora_save(self):
                 model=model_peft,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -644,7 +643,7 @@ def test_dpo_lora_bf16_autocast_llama(self):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -726,7 +725,7 @@ def test_dpo_lora_bf16_autocast(self, name, loss_type, pre_compute, gen_during_e
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -776,7 +775,7 @@ def test_dpo_lora_tags(self):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -813,7 +812,7 @@ def test_dpo_tags(self):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -861,7 +860,7 @@ def test_dpo_lora_force_use_ref(self):
                     model=model_peft,
                     ref_model=ref_model,
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                     peft_config=lora_config,
@@ -884,7 +883,7 @@ def test_dpo_lora_force_use_ref(self):
                 model=model_peft,
                 ref_model=ref_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -909,7 +908,7 @@ def test_dpo_trainer_torch_dtype(self):
             trainer = DPOTrainer(
                 model=self.model_id,
                 ref_model=self.model_id,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 args=training_args,
                 train_dataset=dummy_dataset["train"],
             )
@@ -932,7 +931,7 @@ def test_dpo_trainer_torch_dtype(self):
             ):
                 _ = DPOTrainer(
                     model=self.model_id,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     args=training_args,
                     train_dataset=dummy_dataset["train"],
                 )
@@ -953,7 +952,7 @@ def test_dpo_trainer_torch_dtype(self):
                 _ = DPOTrainer(
                     model=self.model_id,
                     ref_model=self.model_id,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     args=training_args,
                     train_dataset=dummy_dataset["train"],
                 )
@@ -986,7 +985,7 @@ def test_dpo_loss_alpha_div_f(self):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -1029,7 +1028,7 @@ def test_dpo_loss_js_div_f(self):
                 model=model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -1061,7 +1060,7 @@ def test_vdpo_trainer(self, model_id):
                 [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Is this bus in the USA?"}]}],
                 [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Give a thorough description of the image."}]}],
                 [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Who are the people in the image?"}]}],
-                [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What ise written?"}]}],
+                [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is written?"}]}],
             ],
             "chosen": [
                 [{"role": "assistant", "content": [{"type": "text", "text": "The image features a modern, multi-colored train."}]}],
@@ -1075,7 +1074,7 @@ def test_vdpo_trainer(self, model_id):
                 [{"role": "assistant", "content": [{"type": "text", "text": "No, it's not in the USA."}]}],
                 [{"role": "assistant", "content": [{"type": "text", "text": "The image features a forest path surrounded by trees."}]}],
                 [{"role": "assistant", "content": [{"type": "text", "text": "In the image, there are two individuals."}]}],
-                [{"role": "assistant", "content": [{"text": '"ccpb".', "type": "text"}]}],
+                [{"role": "assistant", "content": [{"type": "text", "text": '"ccpb".'}]}],
             ],
             "images": [
                 [Image.fromarray(np.random.randint(0, 255, (92, 33, 3), dtype=np.uint8))],
@@ -1107,7 +1106,7 @@ def test_vdpo_trainer(self, model_id):
                 model=model,
                 ref_model=ref_model,
                 args=training_args,
-                tokenizer=processor,
+                processing_class=processor,
                 train_dataset=dataset,
                 eval_dataset=dataset,
             )
diff --git a/tests/test_gkd_trainer.py b/tests/test_gkd_trainer.py
index 5df6a30c99..3330acbbea 100644
--- a/tests/test_gkd_trainer.py
+++ b/tests/test_gkd_trainer.py
@@ -231,7 +231,7 @@ def test_gkd_trainer(self):
                 args=training_args,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
             )
 
             trainer.train()
@@ -251,7 +251,7 @@ def test_generation_config_init(self):
                 args=training_args,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
             )
 
             self.assertEqual(trainer.generation_config.pad_token_id, self.tokenizer.eos_token_id)
diff --git a/tests/test_iterative_sft_trainer.py b/tests/test_iterative_sft_trainer.py
index 295d82bf69..da3978466c 100644
--- a/tests/test_iterative_sft_trainer.py
+++ b/tests/test_iterative_sft_trainer.py
@@ -107,7 +107,7 @@ def test_iterative_step_from_tensor(self, model_name, input_name):
                 learning_rate=1e-3,
                 report_to="none",
             )
-            iterative_trainer = IterativeSFTTrainer(model=model, args=training_args, tokenizer=tokenizer)
+            iterative_trainer = IterativeSFTTrainer(model=model, args=training_args, processing_class=tokenizer)
             iterative_trainer.optimizer.zero_grad = partial(iterative_trainer.optimizer.zero_grad, set_to_none=False)
 
             iterative_trainer.step(**inputs)
diff --git a/tests/test_kto_trainer.py b/tests/test_kto_trainer.py
index ae43bd0c85..6154aeeecf 100644
--- a/tests/test_kto_trainer.py
+++ b/tests/test_kto_trainer.py
@@ -83,7 +83,7 @@ def test_kto_trainer(self, name, loss_type, pre_compute, eval_dataset):
                 model=model,
                 ref_model=ref_model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"] if eval_dataset else None,
             )
@@ -117,7 +117,7 @@ def test_kto_trainer_with_ref_model_is_model(self):
                     model=self.model,
                     ref_model=self.model,  # ref_model can't be the same as model
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                 )
 
@@ -141,7 +141,7 @@ def test_tokenize_and_process_tokens(self):
                 model=self.model,
                 ref_model=self.ref_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -226,7 +226,7 @@ def test_kto_trainer_without_providing_ref_model(self):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -275,7 +275,7 @@ def test_kto_trainer_without_providing_ref_model_with_lora(self):
                 model=self.model,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -322,7 +322,7 @@ def test_kto_trainer_generate_during_eval_no_wandb(self):
                     model=self.model,
                     ref_model=None,
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                     eval_dataset=dummy_dataset["test"],
                 )
@@ -363,7 +363,7 @@ def test_kto_lora_save(self):
                 model=model_peft,
                 ref_model=None,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
diff --git a/tests/test_nash_md_trainer.py b/tests/test_nash_md_trainer.py
index 470e3eacc5..aff4d9e2cd 100644
--- a/tests/test_nash_md_trainer.py
+++ b/tests/test_nash_md_trainer.py
@@ -56,7 +56,7 @@ def test_nash_md_trainer_training(self, config_name):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -84,7 +84,7 @@ def test_training_with_peft(self):
                 model=self.model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -114,7 +114,7 @@ def test_training_with_peft_and_ref_model(self):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -145,7 +145,7 @@ def test_training_with_peft_model_and_peft_config(self):
                 model=model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_train_config,
diff --git a/tests/test_online_dpo_trainer.py b/tests/test_online_dpo_trainer.py
index 22def9003e..acabbbafa4 100644
--- a/tests/test_online_dpo_trainer.py
+++ b/tests/test_online_dpo_trainer.py
@@ -53,7 +53,7 @@ def test_training(self, config_name):
                 model=self.model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -79,7 +79,7 @@ def test_training_with_ref_model(self):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -104,7 +104,7 @@ def test_ref_model_is_model(self):
                     model=self.model,
                     ref_model=self.model,  # ref_model can't be the same as model
                     args=training_args,
-                    tokenizer=self.tokenizer,
+                    processing_class=self.tokenizer,
                     train_dataset=dummy_dataset["train"],
                 )
 
@@ -126,7 +126,7 @@ def test_training_with_peft(self):
                 model=self.model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -156,7 +156,7 @@ def test_training_with_peft_and_ref_model(self):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -188,7 +188,7 @@ def test_training_with_peft_model_and_peft_config(self):
                 model=model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_train_config,
diff --git a/tests/test_orpo_trainer.py b/tests/test_orpo_trainer.py
index 35785e95f7..2e1dee06cb 100644
--- a/tests/test_orpo_trainer.py
+++ b/tests/test_orpo_trainer.py
@@ -63,7 +63,7 @@ def test_orpo_trainer(self, name):
             trainer = ORPOTrainer(
                 model=model,
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -111,7 +111,7 @@ def test_orpo_trainer_with_lora(self):
             trainer = ORPOTrainer(
                 model=self.model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
diff --git a/tests/test_reward_trainer.py b/tests/test_reward_trainer.py
index 2a9ce8cea8..4e03dd6ff9 100644
--- a/tests/test_reward_trainer.py
+++ b/tests/test_reward_trainer.py
@@ -46,7 +46,7 @@ def test_preprocessing_conversational(self):
             dummy_dataset = load_dataset("trl-internal-testing/zen", "conversational_preference", split="train")
             training_args = RewardConfig(output_dir=tmp_dir, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=self.tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset
             )
             dummy_dataset = dummy_dataset.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": self.tokenizer})
             dummy_dataset = dummy_dataset.map(_tokenize, batched=True, fn_kwargs={"tokenizer": self.tokenizer})
@@ -59,7 +59,7 @@ def test_preprocessing_standard(self):
             dummy_dataset = load_dataset("trl-internal-testing/zen", "standard_preference", split="train")
             training_args = RewardConfig(output_dir=tmp_dir, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=tokenizer, train_dataset=dummy_dataset
             )
             dummy_dataset = dummy_dataset.map(_tokenize, batched=True, fn_kwargs={"tokenizer": tokenizer})
             self.assertDictEqual(trainer.train_dataset[:], dummy_dataset[:])
@@ -69,7 +69,7 @@ def test_train_full(self):
             dummy_dataset = load_dataset("trl-internal-testing/zen", "conversational_preference", split="train")
             training_args = RewardConfig(output_dir=tmp_dir, max_steps=3, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=self.tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset
             )
             previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}
             trainer.train()
@@ -89,7 +89,7 @@ def test_train_full_pretokenized(self):
             dummy_dataset = dummy_dataset.map(_tokenize, batched=True, fn_kwargs={"tokenizer": self.tokenizer})
             training_args = RewardConfig(output_dir=tmp_dir, max_steps=3, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=self.tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset
             )
             previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}
             trainer.train()
@@ -117,7 +117,7 @@ def test_train_lora(self):
             trainer = RewardTrainer(
                 model=self.model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset,
                 peft_config=peft_config,
             )
@@ -165,7 +165,7 @@ def test_train_lora_pretokenized(self):
             trainer = RewardTrainer(
                 model=self.model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset,
                 peft_config=peft_config,
             )
@@ -218,7 +218,7 @@ def test_margin(self):
             dummy_dataset = Dataset.from_dict(dummy_dataset_dict)
             training_args = RewardConfig(output_dir=tmp_dir, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=self.tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset
             )
 
             batch = [dummy_dataset[0]]
@@ -237,6 +237,6 @@ def test_tags(self):
             dummy_dataset = load_dataset("trl-internal-testing/zen", "conversational_preference", split="train")
             training_args = RewardConfig(output_dir=tmp_dir, report_to="none")
             trainer = RewardTrainer(
-                model=self.model, args=training_args, tokenizer=self.tokenizer, train_dataset=dummy_dataset
+                model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset
             )
             self.assertEqual(trainer.model.model_tags, trainer._tag_names)
diff --git a/tests/test_trainers_args.py b/tests/test_trainers_args.py
index 781d9d833e..c7994b5564 100644
--- a/tests/test_trainers_args.py
+++ b/tests/test_trainers_args.py
@@ -68,7 +68,7 @@ def test_bco(self):
                 max_density_ratio=20.0,
             )
             trainer = BCOTrainer(
-                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, tokenizer=tokenizer
+                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, processing_class=tokenizer
             )
             self.assertEqual(trainer.args.max_length, 256)
             self.assertEqual(trainer.args.max_prompt_length, 64)
@@ -110,7 +110,7 @@ def test_cpo(self):
                 model_init_kwargs={"trust_remote_code": True},
                 dataset_num_proc=4,
             )
-            trainer = CPOTrainer(model="gpt2", args=training_args, train_dataset=dataset, tokenizer=tokenizer)
+            trainer = CPOTrainer(model="gpt2", args=training_args, train_dataset=dataset, processing_class=tokenizer)
             self.assertEqual(trainer.args.max_length, 256)
             self.assertEqual(trainer.args.max_prompt_length, 64)
             self.assertEqual(trainer.args.max_completion_length, 64)
@@ -162,7 +162,7 @@ def test_dpo(self):
                 rpo_alpha=0.5,
             )
             trainer = DPOTrainer(
-                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, tokenizer=tokenizer
+                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, processing_class=tokenizer
             )
             self.assertEqual(trainer.args.beta, 0.5)
             self.assertEqual(trainer.args.label_smoothing, 0.5)
@@ -214,7 +214,7 @@ def test_kto(self):
                 dataset_num_proc=4,
             )
             trainer = KTOTrainer(
-                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, tokenizer=tokenizer
+                model="gpt2", ref_model="gpt2", args=training_args, train_dataset=dataset, processing_class=tokenizer
             )
             self.assertEqual(trainer.args.max_length, 256)
             self.assertEqual(trainer.args.max_prompt_length, 64)
@@ -246,7 +246,7 @@ def test_nash_md(self, mixtures_coef_list):
             tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-14m")
             trainer = NashMDTrainer(
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 model=model,
                 ref_model=ref_model,
                 reward_model=reward_model,
@@ -273,7 +273,7 @@ def test_online_dpo(self, beta_list):
             tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-14m")
             trainer = OnlineDPOTrainer(
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 model=model,
                 ref_model=ref_model,
                 reward_model=reward_model,
@@ -306,7 +306,7 @@ def test_orpo(self):
                 dataset_num_proc=4,
             )
 
-            trainer = ORPOTrainer(model="gpt2", args=training_args, train_dataset=dataset, tokenizer=tokenizer)
+            trainer = ORPOTrainer(model="gpt2", args=training_args, train_dataset=dataset, processing_class=tokenizer)
             self.assertEqual(trainer.args.max_length, 256)
             self.assertEqual(trainer.args.max_prompt_length, 64)
             self.assertEqual(trainer.args.max_completion_length, 64)
@@ -329,7 +329,7 @@ def test_reward(self):
                 model=model,
                 args=training_args,
                 train_dataset=dataset,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
             )
             self.assertEqual(trainer.args.max_length, 256)
             self.assertEqual(trainer.args.dataset_num_proc, 4)
@@ -380,7 +380,7 @@ def test_xpo(self, alpha_list):
             tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-14m")
             trainer = XPOTrainer(
                 args=training_args,
-                tokenizer=tokenizer,
+                processing_class=tokenizer,
                 model=model,
                 ref_model=ref_model,
                 reward_model=reward_model,
diff --git a/tests/test_xpo_trainer.py b/tests/test_xpo_trainer.py
index 952ecdcbc7..734d30ba4d 100644
--- a/tests/test_xpo_trainer.py
+++ b/tests/test_xpo_trainer.py
@@ -56,7 +56,7 @@ def test_xpo_trainer_training(self, config_name):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
             )
@@ -84,7 +84,7 @@ def test_training_with_peft(self):
                 model=self.model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -114,7 +114,7 @@ def test_training_with_peft_and_ref_model(self):
                 ref_model=self.ref_model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_config,
@@ -145,7 +145,7 @@ def test_training_with_peft_model_and_peft_config(self):
                 model=model,
                 reward_model=self.reward_model,
                 args=training_args,
-                tokenizer=self.tokenizer,
+                processing_class=self.tokenizer,
                 train_dataset=dummy_dataset["train"],
                 eval_dataset=dummy_dataset["test"],
                 peft_config=lora_train_config,