metadata

language: de
tags:
  - german
  - deutsch

Creators

St. S. (company)
Philip May (T-Systems onsite)
Ph. S. (company)

Evaluation

see https://github.com/GermanT5/german-t5-eval

Tips for training on GPUs

This model is too big to fit on a normal 16GB GPU in FP32 mode. For various reasons, T5 models cannot be trained in FP16 mode. However, mixed precision training is not yet supported on many GPUs. For example, it does not work on V100 GPUs. On A100, however, it does.

That is why we suggest to use DeepSpeed for training. In particular, we recommend the ZeRO-3 Example auto configuration.

ZeRO-Offload pushes the boundary of the maximum model size that can be trained efficiently using minimal GPU resources, by exploiting computational and memory resources on both GPUs and their host CPUs. see ZeRO-Offload