BlinkDL
/

rwkv-4-pile-1b5

Text Generation

Model card Files Files and versions Community

BlinkDL commited on Mar 4, 2023

Commit

69d0ce8

•

1 Parent(s): 7ef50b9

Update README.md

Files changed (1) hide show

README.md +3 -2

README.md CHANGED Viewed

@@ -24,9 +24,10 @@ ctx_len = 1024
 n_layer = 24
 n_embd = 2048
-New checkpoint: RWKV-4-Pile-1B5-20220929-ctx4096.pth : Fine-tuned to ctx_len = 4096. Use it only when your ctxlen is long. Might be slightly weaker for short ctxlens.
-Final checkpoint: RWKV-4-Pile-1B5-20220903-8040.pth : Trained on the Pile for 332B tokens.
 * Pile loss 2.0415
 * LAMBADA ppl 7.04, acc 56.43%
 * PIQA acc 72.36%

 n_layer = 24
 n_embd = 2048
+RWKV-4-Pile-1B5-20220929-ctx4096.pth : Fine-tuned to ctx_len 4096.
+* Likely better when ctxlen > 100. Please test.
+RWKV-4-Pile-1B5-20220903-8040.pth : Trained on the Pile for 332B tokens.
 * Pile loss 2.0415
 * LAMBADA ppl 7.04, acc 56.43%
 * PIQA acc 72.36%