philipp-zettl/vrom-hf-docs huggingface dataset:
tasks (`str` or `list[str]`, *optional*) : One or several task identifiers, to be included in the metadata of the model card.

dataset_tags (`str` or `list[str]`, *optional*) : One or several dataset tags, to be included in the metadata of the model card.

dataset (`str` or `list[str]`, *optional*) : One or several dataset identifiers, to be included in the metadata of the model card.

dataset_args (`str` or `list[str]`, *optional*) : One or several dataset arguments, to be included in the metadata of the model card.
#### create_optimizer[[transformers.Trainer.create_optimizer]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L1146)

Setup the optimizer.

We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
Trainer's init through `optimizers`, or subclass and override this method in a subclass.

**Returns:**

``torch.optim.Optimizer``

The optimizer instance.
#### create_optimizer_and_scheduler[[transformers.Trainer.create_optimizer_and_scheduler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L1135)

Setup the optimizer and the learning rate scheduler.

We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
Trainer's init through `optimizers`, or subclass and override this method (or `create_optimizer` and/or
`create_scheduler`) in a subclass.
#### create_scheduler[[transformers.Trainer.create_scheduler]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L1222)

Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or
passed as an argument.

**Parameters:**

num_training_steps (int) : The number of training steps to do.

**Returns:**

``torch.optim.lr_scheduler.LRScheduler``

The learning rate scheduler instance.
#### evaluate[[transformers.Trainer.evaluate]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L2515)

Run evaluation and returns metrics.

The calling script will be responsible for providing a method to compute metrics, as they are task-dependent
(pass it to the init `compute_metrics` argument).

You can also subclass and override this method to inject custom behavior.

**Parameters:**
eval_dataset (`Dataset` | dict[str, `Dataset`], *optional*) : Pass a dataset if you wish to override `self.eval_dataset`. If it is a [Dataset](https://huggingface.co/docs/datasets/v4.8.4/en/package_reference/main_classes#datasets.Dataset), columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset, prepending the dictionary key to the metric name. Datasets must implement the `__len__` method.    If you pass a dictionary with names of datasets as keys and datasets as values, evaluate will run separate evaluations on each dataset. This can be useful to monitor how training affects other datasets or simply to get a more fine-grained evaluation. When used with `load_best_model_at_end`, make sure `metric_for_best_model` references exactly one of the datasets. If you, for example, pass in `{"data1": data1, "data2": data2}` for two datasets `data1` and `data2`, you could specify `metric_for_best_model="eval_data1_loss"` for using the loss on `data1` and `metric_for_best_model="eval_data2_loss"` for the loss on `data2`.
ignore_keys (`list[str]`, *optional*) : A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.

metric_key_prefix (`str`, *optional*, defaults to `"eval"`) : An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "eval_bleu" if the prefix is "eval" (default)

**Returns:**

A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The
dictionary also contains the epoch number which comes from the training state.
#### floating_point_ops[[transformers.Trainer.floating_point_ops]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L3886)

For models that inherit from [PreTrainedModel](/docs/transformers/v5.6.2/en/main_classes/model#transformers.PreTrainedModel), uses that method to compute the number of floating point
operations for every backward + forward pass. If using another model, either implement such a method in the
model or subclass and override this method.

**Parameters:**

inputs (`dict[str, torch.Tensor | Any]`) : The inputs and targets of the model.

**Returns:**

``int``

The number of floating-point operations.
#### get_decay_parameter_names[[transformers.Trainer.get_decay_parameter_names]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L1283)

Get all parameter names that weight decay will be applied to.

This function filters out parameters in two ways:
1. By layer type (instances of layers specified in ALL_LAYERNORM_LAYERS)
2. By parameter name patterns (containing 'bias', or variation of 'norm')
#### get_eval_dataloader[[transformers.Trainer.get_eval_dataloader]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L885)

Returns the evaluation `~torch.utils.data.DataLoader`.

Subclass and override this method if you want to inject some custom behavior.

**Parameters:**

eval_dataset (`str` or `torch.utils.data.Dataset`, *optional*) : If a `str`, will use `self.eval_dataset[eval_dataset]` as the evaluation dataset. If a `Dataset`, will override `self.eval_dataset` and must implement `__len__`. If it is a [Dataset](https://huggingface.co/docs/datasets/v4.8.4/en/package_reference/main_classes#datasets.Dataset), columns not accepted by the `model.forward()` method are automatically removed.
#### get_optimizer_cls_and_kwargs[[transformers.Trainer.get_optimizer_cls_and_kwargs]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L1252)

Returns the optimizer class and optimizer parameters based on the training arguments.

**Parameters:**

args (`transformers.training_args.TrainingArguments`) : The training arguments for the training session.

model (`PreTrainedModel`, *optional*) : The model being trained. Required for some optimizers (GaLore, Apollo, LOMO).

**Returns:**

A tuple containing the optimizer class and a dictionary of optimizer keyword arguments.
#### get_optimizer_group[[transformers.Trainer.get_optimizer_group]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer_pt_utils.py#L992)

Returns optimizer group for a parameter if given, else returns all optimizer groups for params.

**Parameters:**

param (`str` or `torch.nn.parameter.Parameter`, *optional*) : The parameter for which optimizer group needs to be returned.
#### get_test_dataloader[[transformers.Trainer.get_test_dataloader]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L924)

Returns the test `~torch.utils.data.DataLoader`.

Subclass and override this method if you want to inject some custom behavior.

**Parameters:**

test_dataset (`torch.utils.data.Dataset`, *optional*) : The test dataset to use. If it is a [Dataset](https://huggingface.co/docs/datasets/v4.8.4/en/package_reference/main_classes#datasets.Dataset), columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`.
#### get_total_train_batch_size[[transformers.Trainer.get_total_train_batch_size]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L2360)

Calculates total batch size (micro_batch * grad_accum * dp_world_size).

Accounts for all parallelism dimensions: TP, CP, and SP.

Formula: dp_world_size = world_size // (tp_size * cp_size * sp_size)

Where:
- TP (Tensor Parallelism): Model layers split across GPUs
- CP (Context Parallelism): Sequences split using Ring Attention (FSDP2)
- SP (Sequence Parallelism): Sequences split using ALST/Ulysses (DeepSpeed)

All dimensions are separate and multiplicative: world_size = dp_size * tp_size * cp_size * sp_size
#### get_train_dataloader[[transformers.Trainer.get_train_dataloader]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L865)

Returns the training `~torch.utils.data.DataLoader`.

Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed
training if necessary) otherwise.

Subclass and override this method if you want to inject some custom behavior.
#### hyperparameter_search[[transformers.Trainer.hyperparameter_search]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L4160)

Launch a hyperparameter search using `optuna` or `Ray Tune`. The optimized quantity is determined
by `compute_objective`, which defaults to a function returning the evaluation loss when no metric is provided,
the sum of all metrics otherwise.

To use this method, you need to have provided a `model_init` when initializing your [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer): we need to
reinitialize the model at each new run. This is incompatible with the `optimizers` argument, so you need to
subclass [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) and override the method [create_optimizer_and_scheduler()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.create_optimizer_and_scheduler) for custom
optimizer/scheduler.

**Parameters:**
hp_space (`Callable[["optuna.Trial"], dict[str, float]]`, *optional*) : A function that defines the hyperparameter search space. Will default to `default_hp_space_optuna()` or `default_hp_space_ray()` depending on your backend.

compute_objective (`Callable[[dict[str, float]], float]`, *optional*) : A function computing the objective to minimize or maximize from the metrics returned by the `evaluate` method. Will default to `default_compute_objective()`.

n_trials (`int`, *optional*, defaults to 100) : The number of trial runs to test.
direction (`str` or `list[str]`, *optional*, defaults to `"minimize"`) : If it's single objective optimization, direction is `str`, can be `"minimize"` or `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. If it's multi objectives optimization, direction is `list[str]`, can be List of `"minimize"` and `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics.

backend (`str` or `~training_utils.HPSearchBackend`, *optional*) : The backend to use for hyperparameter search. Will default to optuna or Ray Tune, depending on which one is installed. If all are installed, will default to optuna.

hp_name (`Callable[["optuna.Trial"], str]]`, *optional*) : A function that defines the trial/run name. Will default to None.
kwargs (`dict[str, Any]`, *optional*) : Additional keyword arguments for each backend:  - `optuna`: parameters from [optuna.study.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html) and also the parameters `timeout`, `n_jobs` and `gc_after_trial` from [optuna.study.Study.optimize](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize) - `ray`: parameters from [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run). If `resources_per_trial` is not set in the `kwargs`, it defaults to 1 CPU core and 1 GPU (if available). If `progress_reporter` is not set in the `kwargs`, [ray.tune.CLIReporter](https://docs.ray.io/en/latest/tune/api/doc/ray.tune.CLIReporter.html) is used.

**Returns:**

`[`trainer_utils.BestRun` or `list[trainer_utils.BestRun]`]`
#### log[[transformers.Trainer.log]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L3851)

Log `logs` on the various objects watching training.

Subclass and override this method to inject custom behavior.

**Parameters:**

logs (`dict[str, float]`) : The values to log.

start_time (`Optional[float]`) : The start of training.
#### log_metrics[[transformers.Trainer.log_metrics]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer_pt_utils.py#L830)

Log metrics in a specially formatted way.

Under distributed environment this is done only for a process with rank 0.

Notes on memory reports:

In order to get memory usage report you need to install `psutil`. You can do that with `pip install psutil`.

Now when this method is run, you will see a report that will include:

```
init_mem_cpu_alloc_delta   =     1301MB
init_mem_cpu_peaked_delta  =      154MB
init_mem_gpu_alloc_delta   =      230MB
init_mem_gpu_peaked_delta  =        0MB
train_mem_cpu_alloc_delta  =     1345MB
train_mem_cpu_peaked_delta =        0MB
train_mem_gpu_alloc_delta  =      693MB
train_mem_gpu_peaked_delta =        7MB
```

**Understanding the reports:**
- the first segment, e.g., `train__`, tells you which stage the metrics are for. Reports starting with `init_`
  will be added to the first stage that gets run. So that if only evaluation is run, the memory usage for the
  `__init__` will be reported along with the `eval_` metrics.
- the third segment, is either `cpu` or `gpu`, tells you whether it's the general RAM or the gpu0 memory
  metric.
- `*_alloc_delta` - is the difference in the used/allocated memory counter between the end and the start of the
  stage - it can be negative if a function released more memory than it allocated.
- `*_peaked_delta` - is any extra memory that was consumed and then freed - relative to the current allocated
  memory counter - it is never negative. When you look at the metrics of any stage you add up `alloc_delta` +
  `peaked_delta` and you know how much memory was needed to complete that stage.
The reporting happens only for process of rank 0 and gpu 0 (if there is a gpu). Typically this is enough since the
main process does the bulk of work, but it could be not quite so if model parallel is used and then other GPUs may
use a different amount of gpu memory. This is also not the same under DataParallel where gpu0 may require much more
memory than the rest since it stores the gradient and optimizer states for all participating GPUs. Perhaps in the
future these reports will evolve to measure those too.

The CPU RAM metric measures RSS (Resident Set Size) includes both the memory which is unique to the process and the
memory shared with other processes. It is important to note that it does not include swapped out memory, so the
reports could be imprecise.
The CPU peak memory is measured using a sampling thread. Due to python's GIL it may miss some of the peak memory if
that thread didn't get a chance to run when the highest memory was used. Therefore this report can be less than
reality. Using `tracemalloc` would have reported the exact peak memory, but it doesn't report memory allocations
outside of python. So if some C++ CUDA extension allocated its own memory it won't be reported. And therefore it
was dropped in favor of the memory sampling approach, which reads the current process memory usage.

The GPU allocated and peak memory reporting is done with `torch.cuda.memory_allocated()` and
`torch.cuda.max_memory_allocated()`. This metric reports only "deltas" for pytorch-specific allocations, as
`torch.cuda` memory management system doesn't track any memory allocated outside of pytorch. For example, the very
first cuda call typically loads CUDA kernels, which may take from 0.5 to 2GB of GPU memory.
Note that this tracker doesn't account for memory allocations outside of [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer)'s `__init__`, `train`,
`evaluate` and `predict` calls.

Because `evaluation` calls may happen during `train`, we can't handle nested invocations because
`torch.cuda.max_memory_allocated` is a single counter, so if it gets reset by a nested eval call, `train`'s tracker
will report incorrect info. If this [pytorch issue](https://github.com/pytorch/pytorch/issues/16266) gets resolved
it will be possible to change this class to be re-entrant. Until then we will only track the outer level of
`train`, `evaluate` and `predict` methods. Which means that if `eval` is called during `train`, it's the latter
that will account for its memory usage and that of the former.
This also means that if any other tool that is used along the [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) calls
`torch.cuda.reset_peak_memory_stats`, the gpu peak memory stats could be invalid. And the [Trainer](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer) will disrupt
the normal behavior of any such tools that rely on calling `torch.cuda.reset_peak_memory_stats` themselves.

For best performance you may want to consider turning the memory profiling off for production runs.

**Parameters:**

split (`str`) : Mode/split name: one of `train`, `eval`, `test`

metrics (`dict[str, float]`) : The metrics returned from train/evaluate/predictmetrics: metrics dict
#### metrics_format[[transformers.Trainer.metrics_format]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer_pt_utils.py#L803)

Reformat Trainer metrics values to a human-readable format.

**Parameters:**

metrics (`dict[str, float]`) : The metrics returned from train/evaluate/predict

**Returns:**

`metrics (`dict[str, float]`)`

The reformatted metrics
#### pop_callback[[transformers.Trainer.pop_callback]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L4361)

Remove a callback from the current list of [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback) and returns it.

If the callback is not found, returns `None` (and no error is raised).

**Parameters:**

callback (`type` or [`~transformers.TrainerCallback]`) : A [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback) class or an instance of a [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback). In the first case, will pop the first member of that class found in the list of callbacks.

**Returns:**

`[TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback)`

The callback removed, if found.
#### predict[[transformers.Trainer.predict]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L2822)

Run prediction and returns predictions and potential metrics.

Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method
will also return metrics, like in `evaluate()`.

If your predictions or labels have different sequence length (for instance because you're doing dynamic padding
in a token classification task) the predictions will be padded (on the right) to allow for concatenation into
one array. The padding index is -100.

Returns: *NamedTuple* A namedtuple with the following keys:

- predictions (`np.ndarray`): The predictions on `test_dataset`.
- label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some).
- metrics (`dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained
  labels).

**Parameters:**
test_dataset (`Dataset`) : Dataset to run the predictions on. If it is an `datasets.Dataset`, columns not accepted by the `model.forward()` method are automatically removed. Has to implement the method `__len__`

ignore_keys (`list[str]`, *optional*) : A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.

metric_key_prefix (`str`, *optional*, defaults to `"test"`) : An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "test_bleu" if the prefix is "test" (default)
#### prediction_step[[transformers.Trainer.prediction_step]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L2883)

Perform an evaluation step on `model` using `inputs`.

Subclass and override to inject custom behavior.

**Parameters:**

model (`nn.Module`) : The model to evaluate.

inputs (`dict[str, torch.Tensor | Any]`) : The inputs and targets of the model.  The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments.

prediction_loss_only (`bool`) : Whether or not to return the loss only.

ignore_keys (`list[str]`, *optional*) : A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.

**Returns:**

`tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]`

A tuple with the loss,
logits and labels (each being optional).
#### push_to_hub[[transformers.Trainer.push_to_hub]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L3999)

Upload `self.model` and `self.processing_class` to the 🤗 model hub on the repo `self.args.hub_model_id`.

**Parameters:**

commit_message (`str`, *optional*, defaults to `"End of training"`) : Message to commit while pushing.

blocking (`bool`, *optional*, defaults to `True`) : Whether the function should return only when the `git push` has finished.

token (`str`, *optional*, defaults to `None`) : Token with write permission to overwrite Trainer's original args.

revision (`str`, *optional*) : The git revision to commit from. Defaults to the head of the "main" branch.

kwargs (`dict[str, Any]`, *optional*) : Additional keyword arguments passed along to [create_model_card()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.create_model_card).

**Returns:**
#### remove_callback[[transformers.Trainer.remove_callback]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L4377)

Remove a callback from the current list of [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback).

**Parameters:**

callback (`type` or [`~transformers.TrainerCallback]`) : A [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback) class or an instance of a [TrainerCallback](/docs/transformers/v5.6.2/en/main_classes/callback#transformers.TrainerCallback). In the first case, will remove the first member of that class found in the list of callbacks.
#### save_metrics[[transformers.Trainer.save_metrics]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer_pt_utils.py#L921)

Save metrics into a json file for that split, e.g. `train_results.json`.

Under distributed environment this is done only for a process with rank 0.

To understand the metrics please read the docstring of [log_metrics()](/docs/transformers/v5.6.2/en/main_classes/trainer#transformers.Trainer.log_metrics). The only difference is that raw
unformatted numbers are saved in the current method.

**Parameters:**

split (`str`) : Mode/split name: one of `train`, `eval`, `test`, `all`

metrics (`dict[str, float]`) : The metrics returned from train/evaluate/predict

combined (`bool`, *optional*, defaults to `True`) : Creates combined metrics by updating `all_results.json` with metrics of this call
#### set_initial_training_values[[transformers.Trainer.set_initial_training_values]]

[Source](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/trainer.py#L2290)

Calculates and returns the following values:
- `num_train_epochs`
- `num_update_steps_per_epoch`
- `num_examples`
- `num_train_samples`
- `total_train_batch_size`
- `steps_in_epoch` (total batches per epoch)
- `max_steps`