I can't figure out how to update my PR in this interface, but perhaps there should be a note somewhere indicating the padding is 200 and actual vocab size is 250680. The model config.json says the vocab size is 250,880 but the card says 250,680, which is confusing to newcomers to BLOOM because 256,901,120 embedding parameters / 1024 embedding dim = 250,880, not 250,680.

mathemakitten changed pull request status to closed
BigScience Workshop org

Okay so 250,880 is the dimension in the embedding matrix. However the tokenizer only generates 250680 different tokens. I think the config.json sets the value to 250880 as the embedding matrix had that number of rows.

BigScience Workshop org

@mathemakitten You can click on the PR label/button thingy and it will take you to that branch so you can update the PR in the GUI
image.png

BigScience Workshop org

(or using git command line if that is an option @mathemakitten )

Sign up or log in to comment