File size: 2,944 Bytes
2f33f25 b38238b 2f33f25 b38238b 25bec7f b38238b 213ec81 b38238b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- llm
- dataset combination
- Pretraining
---
# SlimPajama-DC
<center><img src="assets/SlimPajama-DC-logo.png" alt="SlimPajama-DC logo" width="200"/></center>
[**SlimPajama-DC**](https://arxiv.org/abs/2309.10818) is a set of 1.3B parameter language models, distinctively trained on the different combinations of 330B subsets of SlimPajama dataset.
| Details of Dataset Combinations for Different Models |
|------------------------------------------------|
<center><img src="assets/data_combination.png" alt="details of dataset combinations" width="800"/></center>
Despite being trained on a smaller amount of 330B tokens compared to TinyLlama and Olmo's 3 trillion, SlimPajama-DC surpasses TinyLlama and Olmo in some challenging English tasks.
| Our tests comprise: (1) AI2 Reasoning Challenge (25-shot); (2) HellaSwag (10-shot); (3) MMLU (5-shot); (4) TruthfulQA (0-shot) |
|------------------------------------------------|
<center><img src="assets/res1.png" alt="results" width="930"/></center>
‡ represents the RefinedWeb CC.
| Performance on More Benchmarks |
|------------------------------------------------|
<center><img src="assets/res2.png" alt="results" width="830"/></center>
ARC easy and ARC challenge are evaluated using 25-shot. All other evaluation benchmarks are tested on 0-shot. * represents the results are averaged across multiple sub-items inside each benchmark dataset.
# Dataset
Our full dataset is available at [SlimPajama-627B-DC](https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC).
# Model Usage
To load a specific checkpoint, use the revision argument as shown below, for example, `SlimPajama-DC-6`. All the revisions can be seen from the branch dropdown in the "Files and versions" tab. If no revision argument is provided, it will load the default checkpoint `SlimPajama-DC-6`.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"MBZUAI-LLM/SlimPajama-DC",
revision="SlimPajama-DC-6",
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
"MBZUAI-LLM/SlimPajama-DC",
revision="SlimPajama-DC-6",
trust_remote_code=True
)
prompt = 'int add(int x, int y) {'
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
gen_tokens = model.generate(input_ids, do_sample=True, max_length=400)
print("-"*20 + "Output for model" + 20 * '-')
print(tokenizer.batch_decode(gen_tokens)[0])
```
# Citation
**BibTeX:**
```bibtex
@article{shen2023slimpajama,
title={Slimpajama-dc: Understanding data combinations for llm training},
author={Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, Eric Xing},
journal={arXiv preprint arXiv:2309.10818},
year={2023}
}
``` |