Keras callbacks

Transformers documentation

Natural Language Processing

Performance and scalability

Reinforcement learning models

When training a Transformers model with Keras, there are some library-specific callbacks available to automate common tasks:

## [](#transformers.KerasMetricCallback)KerasMetricCallback

### class transformers.KerasMetricCallback

[](#transformers.KerasMetricCallback)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/keras_callbacks.py#L21)

( metric\_fn: typing.Callableeval\_dataset: typing.Union\[tensorflow.python.data.ops.dataset\_ops.DatasetV2, numpy.ndarray, tensorflow.python.framework.ops.Tensor, tuple, dict\]output\_cols: typing.Optional\[typing.List\[str\]\] = Nonelabel\_cols: typing.Optional\[typing.List\[str\]\] = Nonebatch\_size: typing.Optional\[int\] = Nonepredict\_with\_generate: bool = Falseuse\_xla\_generation: bool = Falsegenerate\_kwargs: typing.Optional\[dict\] = None )

Callback to compute metrics at the end of every epoch. Unlike normal Keras metrics, these do not need to be compilable by TF. It is particularly useful for common NLP metrics like BLEU and ROUGE that require string operations or generation loops that cannot be compiled. Predictions (or generations) will be computed on the `eval_dataset` before being passed to the `metric_fn` in `np.ndarray` format. The `metric_fn` should compute metrics and return a dict mapping metric names to metric values.

We provide an example of a suitable metric\_fn that computes ROUGE scores for a summarization model below. Note that this example skips some post-processing for readability and simplicity, and should probably not be used as-is!

```
from datasets import load_metric

rouge_metric = load_metric("rouge")


def rouge_fn(predictions, labels):
    decoded_predictions = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
    result = rouge_metric.compute(predictions=decoded_predictions, references=decoded_labels)
    return {key: value.mid.fmeasure * 100 for key, value in result.items()}```

The above function will return a dict containing values which will be logged like any other Keras metric:

```
{'rouge1': 37.4199, 'rouge2': 13.9768, 'rougeL': 34.361, 'rougeLsum': 35.0781```

## [](#transformers.PushToHubCallback)PushToHubCallback

### class transformers.PushToHubCallback

[](#transformers.PushToHubCallback)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/keras_callbacks.py#L268)

( output\_dir: typing.Union\[str, pathlib.Path\]save\_strategy: typing.Union\[str, transformers.trainer\_utils.IntervalStrategy\] = 'epoch'save\_steps: typing.Optional\[int\] = Nonetokenizer: typing.Optional\[transformers.tokenization\_utils\_base.PreTrainedTokenizerBase\] = Nonehub\_model\_id: typing.Optional\[str\] = Nonehub\_token: typing.Optional\[str\] = Nonecheckpoint: bool = False\*\*model\_card\_args )

Callback that will save and push the model to the Hub regularly. By default, it pushes once per epoch, but this can be changed with the `save_strategy` argument. Pushed models can be accessed like any other model on the hub, such as with the `from_pretrained` method.

```
from transformers.keras_callbacks import PushToHubCallback

push_to_hub_callback = PushToHubCallback(
    output_dir="./model_save",
    tokenizer=tokenizer,
    hub_model_id="gpt5-7xlarge",
)

model.fit(train_dataset, callbacks=[push_to_hub_callback])```