|
--- |
|
base_model: |
|
- Qwen/Qwen2.5-7B-Instruct |
|
datasets: |
|
- chtmp223/CLIPPER |
|
language: |
|
- en |
|
license: apache-2.0 |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# Qwen2.5-7B-CLIPPER |
|
Qwen2.5-7B-CLIPPER is a fine-tuned version of https://huggingface.co/Qwen/Qwen2.5-7B-Instruct using supervised finetuning over chtmp223/CLIPPER dataset. |
|
Please check [our paper](https://arxiv.org/abs/2502.14854) for more details on the method. |
|
|
|
## 📒 Model Details |
|
|
|
### Model Description |
|
|
|
- **Language(s) (NLP):** English |
|
- **License:** Apache-2.0 |
|
- **Finetuned from model:** https://huggingface.co/Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [Github repository](https://github.com/chtmp223/CLIPPER). |
|
- **Paper:** [https://arxiv.org/abs/2502.14854](https://arxiv.org/abs/2502.14854) |
|
|
|
|
|
## 💻 Training Details |
|
|
|
### Training Data |
|
|
|
[chtmp223/CLIPPER](https://huggingface.co/datasets/chtmp223/CLIPPER) |
|
|
|
### Training Procedure |
|
|
|
| **Configurations** | **Values** | |
|
|----------------------------------|--------------| |
|
| Hardware (Training and Inference)| 8xA100s | |
|
| Tracking | wandb | |
|
| batch size | 16 | |
|
| gradient_checkpointing | True | |
|
| learning_rate | 1.0e-6 | |
|
| lr_scheduler_type | cosine | |
|
| max_length | 131072 | |
|
| num_train_epochs | 1 |\n| optim | adamw_torch |\n\n#### Software\n\nTraining code is adapted from [https://github.com/Qihoo360/360-LLaMA-Factory/tree/1b5398f539c7d94a530f3f32b53553a3b1928314](https://github.com/Qihoo360/360-LLaMA-Factory/tree/1b5398f539c7d94a530f3f32b53553a3b1928314).\n\n## 🤗 Inference\nInference is done with [vLLM](https://github.com/vllm-project/vllm) on 1 A100-80GB. \n\n## 📜 Citation \n\n```\n@misc{pham2025clippercompressionenableslongcontext,\n title={CLIPPER: Compression enables long-context synthetic data generation}, \n author={Chau Minh Pham and Yapei Chang and Mohit Iyyer},\n year={2025},\n eprint={2502.14854},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2502.14854}, \n}\n``` |
