Update README.md
Browse files
README.md
CHANGED
|
@@ -55,6 +55,8 @@ And for the adventurous, Rope 10 : (max context 40960) : WinterGoddess-1.4x-lima
|
|
| 55 |
|
| 56 |
So the linear rope, at least on this model, is flexible, and you can lower it to have the best peplexity for your max context.
|
| 57 |
|
|
|
|
|
|
|
| 58 |
Then, I wonder about applying a NTK rope on the top of it to expend it further, even if it screws with the integrity of numbers in chat).
|
| 59 |
Multiply a linear rope (2, 4, 8, whatever) by 5888 (Alpha 1.6, or RBF 16119.8), 6144 (Alpha 1.8, or RBF 18168.7) and even 7424 (Alpha 2.2, or RBF 22277).
|
| 60 |
This to get a further boost in max context size. Ex with Linear 8 with Alpha 1.8/RBF22277 : 8*7424 = 59392.
|
|
|
|
| 55 |
|
| 56 |
So the linear rope, at least on this model, is flexible, and you can lower it to have the best peplexity for your max context.
|
| 57 |
|
| 58 |
+
All these results are reproducible with lowers deltas between them for Q3_K_S, and I suppose for other quants as well.
|
| 59 |
+
|
| 60 |
Then, I wonder about applying a NTK rope on the top of it to expend it further, even if it screws with the integrity of numbers in chat).
|
| 61 |
Multiply a linear rope (2, 4, 8, whatever) by 5888 (Alpha 1.6, or RBF 16119.8), 6144 (Alpha 1.8, or RBF 18168.7) and even 7424 (Alpha 2.2, or RBF 22277).
|
| 62 |
This to get a further boost in max context size. Ex with Linear 8 with Alpha 1.8/RBF22277 : 8*7424 = 59392.
|