text: 
  
  TITLE
  [RFC] introduce `config.trained_precision`
  
  COMMENTS
  11
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  As we are discovering that `bf16`-pretrained models don't do well on `fp16` "regime" (and surely vice-versa), and some models are pre-trained in `fp32` and surely won't do well on either `bf16` or `fp16`, and the problem is going to grow as more `bf16`-supporting hardware comes out, I propose we start requiring that the model tells the user which mode it was pretrained under.
  
  So I suggest we add `config.trained_precision` which currently would be one of `fp16`, `bf16`, `fp32`, `unknown`.
  
  I haven't thoughts it through on how to derive this automatically during `save_pretrained`, but when porting checkpoints the porter can figure that out and manually set this in the conversion script.
  
  For example, from what I understood gtp-neo if `bf16` for all but `2.7B` version, which is `fp32`.
  
  @sgugger, @LysandreJik 