Korbinian Pöppel
commited on
Commit
•
fbd0d01
1
Parent(s):
a7ad556
Make model card more informative.
Browse files- MMLUvsTrainToken.svg +1737 -0
- README.md +22 -1
- plot_tokens_per_sec.svg +1893 -0
MMLUvsTrainToken.svg
ADDED
README.md
CHANGED
@@ -31,4 +31,25 @@ tokenizers = AutoTokenizer.from_pretrained("NX-AI/xLSTM-7b")
|
|
31 |
xlstm(tokenizer("Hello xLSTM, how are you doing?"))
|
32 |
```
|
33 |
|
34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
31 |
xlstm(tokenizer("Hello xLSTM, how are you doing?"))
|
32 |
```
|
33 |
|
34 |
+
## Speed results
|
35 |
+
Generation Speed using `torch.cuda.graph` and `torch.compile` optimizations:
|
36 |
+
![generation speed](plot_tokens_per_sec.svg)
|
37 |
+
|
38 |
+
## Performance
|
39 |
+
![mmlu_train_token](MMLUvsTrainToken.svg)
|
40 |
+
|
41 |
+
Using HuggingFace's `lm_eval`:
|
42 |
+
|
43 |
+
| BBH | MMLU-Pro | Math | MUSR | GPQA | IfEval |
|
44 |
+
|-------|----------|--------|------|------|--------|
|
45 |
+
| 0.381 | 0.242 | 0.036 | 0.379|0.280 | 0.244 |
|
46 |
+
|
47 |
+
Using HuggingFace's `lighteval` in the Leaderboard-v1 settings:
|
48 |
+
|
49 |
+
|Arc-Challenge (25-shot) |MMLU (5-shot) |Hellaswag (10-shot)|Winogrande (5-shot) |TruthfulQA (0-shot) |GSM8k (5-shot) |OpenbookQA (5-shot) | PiQA (5-shot)|
|
50 |
+
|------------------------|--------------|-------------------|--------------------|--------------------|---------------|--------------------|--------------|
|
51 |
+
| 0.584 |0.589 | 0.710 |0.742 | 0.420 | 0.004 | 0.443 | 0.817 |
|
52 |
+
|
53 |
+
## License
|
54 |
+
NXAI Community License (see `LICENSE` file)
|
55 |
+
|
plot_tokens_per_sec.svg
ADDED