YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
This is an experimental depth-upscale of Qwen2.5 14B to a total of 21.4B parameters. A total of 24 layers were added (layers 30-41 inclusive each repeated twice) bringing the total to 72 layers.
The added layers had the o_proj and down_proj modules zeroed out prior to retraining as seen in other modern depth upscaling experiments.
The upscaled model was then trained on a mix of about 10M tokens worth of instruct and creative data, with the majority being general instruct training to try to repair those connections.
- Downloads last month
- 19
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.