Korbinian Pöppel commited on
Commit
fbd0d01
1 Parent(s): a7ad556

Make model card more informative.

Browse files
Files changed (3) hide show
  1. MMLUvsTrainToken.svg +1737 -0
  2. README.md +22 -1
  3. plot_tokens_per_sec.svg +1893 -0
MMLUvsTrainToken.svg ADDED
README.md CHANGED
@@ -31,4 +31,25 @@ tokenizers = AutoTokenizer.from_pretrained("NX-AI/xLSTM-7b")
31
  xlstm(tokenizer("Hello xLSTM, how are you doing?"))
32
  ```
33
 
34
- License: NXAI Community License (see `LICENSE` file)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  xlstm(tokenizer("Hello xLSTM, how are you doing?"))
32
  ```
33
 
34
+ ## Speed results
35
+ Generation Speed using `torch.cuda.graph` and `torch.compile` optimizations:
36
+ ![generation speed](plot_tokens_per_sec.svg)
37
+
38
+ ## Performance
39
+ ![mmlu_train_token](MMLUvsTrainToken.svg)
40
+
41
+ Using HuggingFace's `lm_eval`:
42
+
43
+ | BBH | MMLU-Pro | Math | MUSR | GPQA | IfEval |
44
+ |-------|----------|--------|------|------|--------|
45
+ | 0.381 | 0.242 | 0.036 | 0.379|0.280 | 0.244 |
46
+
47
+ Using HuggingFace's `lighteval` in the Leaderboard-v1 settings:
48
+
49
+ |Arc-Challenge (25-shot) |MMLU (5-shot) |Hellaswag (10-shot)|Winogrande (5-shot) |TruthfulQA (0-shot) |GSM8k (5-shot) |OpenbookQA (5-shot) | PiQA (5-shot)|
50
+ |------------------------|--------------|-------------------|--------------------|--------------------|---------------|--------------------|--------------|
51
+ | 0.584 |0.589 | 0.710 |0.742 | 0.420 | 0.004 | 0.443 | 0.817 |
52
+
53
+ ## License
54
+ NXAI Community License (see `LICENSE` file)
55
+
plot_tokens_per_sec.svg ADDED