Update README.md
Browse files
README.md
CHANGED
@@ -44,7 +44,7 @@ The following table shows the hyper-paramters we used in our training process.
|
|
44 |
| Batch Size | 4M |
|
45 |
| Weight Decay | 0.1 |
|
46 |
|
47 |
-
**Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(Math, Coding), and continued training for 50B tokens.
|
48 |
|
49 |
| Hyper-parameters | |
|
50 |
| --------------------- | ----------- |
|
|
|
44 |
| Batch Size | 4M |
|
45 |
| Weight Decay | 0.1 |
|
46 |
|
47 |
+
**Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets (e.g., Math, Coding), and continued training for 50B tokens.
|
48 |
|
49 |
| Hyper-parameters | |
|
50 |
| --------------------- | ----------- |
|