TeichAI/Minimax-M2.7-Opus-4.6-Distill-Lora

Minimax-m2.7 x Claude Opus 4.6

This model is a Minimax-m2.7 Opus 4.6 style tune distill lora. Its made to give the model more of the personality and reasoning style of Opus 4.6.

As with any distill NO THIS DOES NOT GIVE YOU A LOCAL OPUS, LoRA TUNED MODELS CAN'T DO THAT THIS IS A STYLE TUNE.

Trained off of 6144 cleaned traces from the Gryphe/Opus-4.6-Reasoning-24k dataset filtered to remove summarized COT, incomplete responses and the Gemini samples that were present in the dataset.

MiniMax M2.7 — Attention Tail LoRA (rank 64)

Fine-tune of MiniMax M2.7 (230B-parameter MoE, ~10B active) trained directly against the AWQ 4-bit checkpoint (compressed-tensors, symmetric int4, group size 32) — no bf16 copy of the full model was ever loaded, rented, or required. All training was done locally on consumer-grade hardware (2× NVIDIA DGX Spark, 128 GB unified memory each).

Method: frozen-prefix activation caching + resident-tail LoRA

Full-model LoRA on a 230B MoE is out of reach for local hardware: the bf16 weights alone are ~457 GB, QLoRA through this quant format has no training path, and full layer-streaming training benchmarks out at roughly 14 hours per epoch. The workaround is to split the network at a boundary layer and treat the two halves differently:

1. One-time prefix pass. Layers 0–49 are run over the entire dataset exactly once as a streaming forward pass — only one dequantized layer's weights are resident at a time, so the ~457 GB bf16 trunk never materializes. The hidden states at the layer-50 boundary are cached to NVMe in multi-GB chunks. Between the streamed layer, the activation buffers being accumulated for each chunk write, and the batch tensors, the pipeline saturates the node: peak observed usage was ~130 GB unified memory plus ~10.6 GB swap on a single 128 GB DGX Spark. The point isn't a small footprint — it's that the entire pass fits one node at all. Since these layers stay frozen, their outputs never change, so this cost is paid per dataset, not per training step.
2. Resident tail training. Layers 50–61 are dequantized to bf16 and held resident (~88 GB), with LoRA adapters on the attention projections (q_proj, k_proj, v_proj, o_proj). Training consumes the cached boundary activations as inputs, so each step only pays for 12 layers of compute. Throughput is memory-bandwidth-bound at a few seconds per sample. Because LoRA's B matrices are zero-initialized, the step-0 model is exactly the base model, giving a free pre-training baseline on held-out validation.
3. Deployment. For local serving, the adapter was merged and requantized directly into the base checkpoint (dequantize the targeted weights, add the low-rank update, requantize to int4 g32 in place). This repository publishes the GGUF adapter that you can load in llama.cpp, not the merged model — re-uploading ~120 GB of mostly-unchanged shards to carry a small low-rank delta is wasteful. See Using this adapter below. FFN/expert weights are untouched at every stage. Attention-only adaptation on the tail leaves the model's routing, expert knowledge, and output head frozen, which preserves special-token behavior (tool calling, structured output) by construction rather than by regularization.

Adapter configuration

Rank (r)	64
Alpha	64 (scaling = 1.0)
Dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj
Layers	50–61 (final 12 of 62)
Trainable parameters	~20M
Base checkpoint	cyankiwi/MiniMax-M2.7-AWQ-4bit (compressed-tensors int4 g32)
Training precision	bf16 (tail resident), frozen int4 prefix

A control set monitored during training serves as a catastrophic-forgetting alarm; the frozen prefix, frozen experts, and frozen lm_head bound how far the model can drift from base behavior.

Intended use & limitations

This is a style/behavior adaptation of the final attention layers, not new-knowledge injection — the method deliberately cannot alter expert FFNs or the first 50 layers. Anything the base model cannot do, this model cannot do. Inherits all limitations and licensing of the base MiniMax M2.7 checkpoint.

Using this adapter

This repo ships the adapter in GGUF format for runtime use with llama.cpp — no merging required:

 --lora-scaled <this-adapter>.gguf:1.0

Works with llama.cpp-based frontends that expose LoRA loading (llama-server, llama-cli, and downstream UIs that pass --lora through).

Quantization-mismatch expectations, stated honestly: this adapter was trained against weights and activations derived from the int4 AWQ checkpoint linked above. Applying it over a GGUF quant means the base features differ slightly from the training features. The learned behavior transfers, but expect it somewhat muted or shifted relative to the AWQ-merged original — the higher the GGUF bit-rate (Q6_K / Q8_0), the closer it should track. This is inherent to cross-quant adapter serving, not a defect of any particular GGUF.

The adapter contains weights only for layers 50–61 (attention projections); all other layers are untouched by construction.

• 🧬 Datasets:

  • Gryphe/Opus-4.6-Reasoning-24k

• 🏗 Base Model:

  • cyankiwi/MiniMax-M2.7-AWQ-4bit

• ⚡ Use cases:

  • Coding
  • Creative Writing
  • Visual Understanding
  • General Purpose

Citations and Contributions

• @Gryphe - Providing the Opus 4.6 traces (see the original dataset here)
• @Minimax - Providing a good base model
• @cyankiwi - Providing the quantized checkpoint that made the training possible on consumer hardware

Usage

If you need help setting up and configuring this model please follow the Minimax team's instructions in the original model's README