Request for pre-distillation / base H3 weights
#97
by muchov - opened
Would it be possible to release the pre-distillation / base H3 weights as well?
The currently released distilled checkpoint is useful for inference, but much less suitable for further training. Full fine-tuning or LoRA can cause behavioral drift and partial de-distillation, especially since the teacher model and exact distillation objective are unknown.
Without the original pre-distillation checkpoint, it is difficult to properly fine-tune, continue pretraining, or adapt H3 without destroying some of the behavior introduced during distillation.
Releasing the base weights alongside the distilled ones would make the model much more useful for research and downstream adaptation.