tr11-176B-logs / README.md
stas's picture
add link
f291fe9
metadata
language:
  - ak
  - ar
  - as
  - bm
  - bn
  - ca
  - code
  - en
  - es
  - eu
  - fon
  - fr
  - gu
  - hi
  - id
  - ig
  - ki
  - kn
  - lg
  - ln
  - ml
  - mr
  - ne
  - nso
  - ny
  - or
  - pa
  - pt
  - rn
  - rw
  - sn
  - st
  - sw
  - ta
  - te
  - tn
  - ts
  - tum
  - tw
  - ur
  - vi
  - wo
  - xh
  - yo
  - zh
  - zhs
  - zht
  - zu

BigScience Large Language Model Training

Training a multilingual 176 billion parameters model in the open BigScience Logo

BigScience is a open and collaborative workshop around the study and creation of very large language models gathering more than 1000 researchers around the worlds. You can find more information on the main website at https://bigscience.huggingface.co.

The training of BigScience’s main model started on March 11, 2022 11:42am PST and will continue for 3-4 months on 384 A100 80GB GPUs of the Jean Zay public supercomputer

You can follow the training at https://twitter.com/BigScienceLLM or on the Tensorboards tab above.

More information on the model, dataset, hardware, environmental consideration:

The model

The dataset

The engineering side

Environmental considerations

  • Jean Zay, the supercomputer we are using for model training, is mostly powered by nuclear energy, which is a low carbon energy source.
  • Significant efforts were made to make sure that the computing infrastructure is as efficient as possible — the heat generated by the hardware even gets used for heating buildings on campus!
  • More information:
    • We are currently working on making a precise estimate of the carbon emitted during all of the steps of model training, including intermediate experiments as well as inference.
    • More soon!