text: 
  
  TITLE
  Using `bf16` instead of `fp16`
  
  COMMENTS
  3
  
  REACTIONS
  +1: 1
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  As seen in [this pr](https://github.com/huggingface/transformers/pull/10956), there is demand for `bf16` compatibility in training of transformers models.  The pytorch folks just [added this feature](https://github.com/pytorch/pytorch/pull/61002) to their master branch, so we are now able to work on adding it to this repo.
  
  ## Motivation
  
  Related to [this issue](https://github.com/huggingface/transformers/pull/10956) and [this pytorch pr](https://github.com/pytorch/pytorch/pull/61002).  This feature would allow for proper half-precision training of google-trained models, for example any `T5` model.
  
  ## Your contribution
  
  I am currently working on a PR for this [here](https://github.com/JamesDeAntonis/transformers/tree/bf16), and would gladly field any suggestions and contributions.
  
  @stas00 