bigscience

/

bloom-7b1

like 184

Hi @malteos ! If I am not mistaken it should be global_step156000 :-)

Thanks @ybelkada

That's like 36%, right? (Assuming 220_000_000 train samples with 512 batch size)

Yeah I think so, to go over the train sample you would need ~430.000 steps so yes it corresponds to 36% approximately

and according to https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/smaller_models/tr11f-6B3-ml.slurm the batch size is indeed 512