text: 
  
  TITLE
  [Deepspeed] add support for bf16 mode
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  This PR:
  
  - adds support for bfloat16 for ZeRO-1, ZeRO-2 and ZeRO-3 to HF/DS integration.
  - most functional tests are now run for bf16 as well - so this PR almost doubles the number of tests. model zoo tests are left with fp16 for now, as both should work the same.
  - docs are updated to document the new feature
  
  Requirements:
  
  1. [x] merged https://github.com/huggingface/transformers/pull/13207
  2. [x] ZeRO-3 support merged https://github.com/microsoft/DeepSpeed/pull/1453
  3. [x] Need new deepspeed release after the above is merged 
  4. [x] update version dependency table
  
  ---------------------------
  
  for users who want to try this early:
  
  1. add `--bf16` to your previous fp16 or fp32 deepspeed command line 
  
  done ;)
  
  @sgugger 