LH-Tech-AI CompactAI commited on
Commit
deff1d6
·
1 Parent(s): 9e90466

Update README.md (#1)

Browse files

- Update README.md (bcaf3fa28e1c745e31c834a592e0924b37e9c8cf)


Co-authored-by: LaneFiedler <CompactAI@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -20,7 +20,7 @@ tags:
20
  This is a tiny 20.75M parameter model showing how small models can perform on a little bunch of data.
21
 
22
  ## Training data
23
- We used the first 100 million tokens of the 10BT Sample of Fineweb-Edu to train this model for 5000 steps to a final val loss of 4.1566.
24
 
25
  ## Training specs
26
  - Architecture: nanoGPT
 
20
  This is a tiny 20.75M parameter model showing how small models can perform on a little bunch of data.
21
 
22
  ## Training data
23
+ We used the first 100 million tokens of the 10BT Sample of Fineweb-Edu to train this model for 5000 steps for a final loss of ~4.0 and a val loss of 4.1566.
24
 
25
  ## Training specs
26
  - Architecture: nanoGPT