Any plans to release a non-distilled base model for training?
You guys landed like Prometheus giving us the power that the gods were gatekeeping from us and I couldn't be more grateful for this immense gift.
However, there's still one last hurdle we're facing in the community, many of us are finding it extremely difficult to properly fine-tune or train LoRAs on new concepts the model wasn't originally exposed to, the released weights are distilled so training on that is notoriously not good as it produces weak/broken results compared to a non-distilled model.
Ref2VA is truly amazing but doesn’t 100% replace the ability to teach the model concepts that didn’t exist in its training data.
Would MiniMax team consider releasing a non-distilled base checkpoint (or some solution) specifically for training/fine-tuning? Even a limited or research-only release would be hugely appreciated by the community.
Thanks a lot for the great work so far!
Thank you for raising this. An undistilled FL2VA/Ref2VA BF16 checkpoint would be extremely valuable for downstream training.
Several community trainers can now fine-tune H3, but we still do not know the correct objective for preserving its original CFG-distilled behavior. Standard flow-matching training can produce a decreasing loss while gradually changing prompt adherence, temporal dynamics, motion speed, modality balance, or the model’s expected CFG-free inference behavior.
If releasing the undistilled checkpoint is not possible, could the MiniMax team publish at least the training contract or pseudocode for the original guidance distillation?
In particular:
- Was the released checkpoint distilled from a separate undistilled teacher, or from an earlier version of the same architecture?
- What teacher CFG scale, or distribution of scales, was used?
- Was guidance constant across timesteps, or scheduled according to video/audio sigma?
- Were video and audio distilled jointly, and how were their losses balanced?
- Was the unconditional branch based on an empty prompt, conditioning dropout, or another representation?
- Was the final sparse-attention training stage performed before, during, or after guidance distillation?
- What objective do you recommend for downstream LoRA and partial/full fine-tuning?
- Could you release either:
- the undistilled teacher/base checkpoint,
- a training-only de-distillation adapter,
- or a frozen-teacher/reference-loss recipe that preserves the released model’s behavior?
The community currently has contrastive guidance objectives, base-preservation losses, de-distillation training adapters, and frozen-base drift losses, but these are approximations and may not reproduce the original MiniMax training procedure.
Even a short technical note with the loss equations and recommended downstream fine-tuning settings would prevent a great deal of duplicated experimentation.
PS. We’re wasting a huge amount of time and money looking for solutions.