---
license: mit
---

<!-- Provide a quick summary of what the model is/does. -->

An unofficial reproduced PRepBN-Llama-350M checkpoints for [SLAB](https://github.com/xinghaochen/SLAB/).

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [https://github.com/xinghaochen/SLAB/]
- **Paper [optional]:** [https://arxiv.org/abs/2405.11582]


## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

https://github.com/xinghaochen/SLAB/tree/main/llama

```
python evaluation.py --ckpt <checkpoint-path>
```

[Results](https://github.com/xinghaochen/SLAB/blob/main/docs/llama.png)


**BibTeX:**

```
@inproceedings{guo2024slab,
  title={SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization},
  author={Guo, Jialong and Chen, Xinghao and Tang, Yehui  and Wang, Yunhe},
  booktitle={International Conference on Machine Learning},
  year={2024}
}
```