text: 
  
  TITLE
  Support for Training with BF16
  
  COMMENTS
  31
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # What does this PR do?
  
  As seen in [this pr](https://github.com/huggingface/transformers/pull/10956), there is demand for bf16 compatibility in training of transformers models. The pytorch folks just added [this feature](https://github.com/pytorch/pytorch/pull/61002) to their master branch, so we are now able to work on adding it to this repo.  This pr follows from [this issue](https://github.com/huggingface/transformers/issues/13170).
  
  Fixes https://github.com/huggingface/transformers/issues/13170
  
  ------------------
  (OP edited by @stas00)
  
  Also merged here and adapted changes proposed by @manuelciosici at https://github.com/huggingface/transformers/pull/14448
  
  This PR:
  
  - adds helper utils: `require_torch_bf16` and `is_torch_bf16_available`
  - modifies `invert_attention_mask` and one `forward` in t5 to include bf16 mode switches
  
  HF Trainer: 
  - adds `--bf16` and `--bf16_full_eval` modes - same as fp16 equivalents
  - renames and deprecates `--fp16_backend` and replaces it with `--half_precision_backend` - since we now have more than one half precision mode
  
  Tests:
  - adds `--bf16` and `--bf16_full_eval` tests
  
  @sgugger, @LysandreJik, 
  
  Also tagging @patrickvonplaten, @patil-suraj since once this is merged you can start sending users that have problems with bf16 pre-trained models and have Amphere hardware to use this `--bf16` mode. Deepspeed bf16 support will follow soon.