Technical clarification: which Flint innovations are portable to other MoE architectures?
Title: Technical clarification: which Flint innovations are portable to other MoE architectures?
Congratulations on the Flint release. We are studying how its architectural and inference innovations might inform a reproducible redesign of IFM's K2-Horizon-MoVA-36B-A4B.
We have read Recursive's Flint announcement and the published model documentation. We understand that Flint combines a highly sparse MoE design with hybrid attention, a shared expert, specialized inference kernels, and a quantization-and-distillation pipeline.
We would appreciate clarification on several questions:
- Which architectural decisions produced the largest measured improvements in the trade-off between active parameters, coding capability, and inference speed?
- Which choices are generalizable to an existing MoE architecture, and which require a model to be trained from scratch or substantially retrained?
- What evidence supports the selected expert configuration and routing policy? Are router losses, load-balancing methods, expert dimensions, or relevant ablation results documented?
- Which Eureka Machine discoveries are available as reproducible training recipes, technical reports, code, or experiment artifacts, and which remain internal?
- For the low-bit inference path, what are the essential implementation details behind the output-head quantization and offline self-distillation procedure?
- What would you recommend to researchers who want to reproduce the efficiency improvements without conflating architectural changes with inference-kernel optimizations?
Our aim is not to assume that Flint can be transplanted unchanged into another model, but to identify which innovations can be tested independently and evaluated rigorously.
Any documentation, reference implementations, or recommendations would be greatly appreciated.
Follow-up: How was the 35B/3B → 39.8B/2.5B architecture decision made?
Please enlighten us with the reasoning behind the starting point and stopping point:
- Why was the initial baseline specifically 35B total / 3B active? Was it an established model configuration, a hardware-budget choice, or an initial point selected by the search system?
- What was the objective used to rank candidate architectures? Was it a weighted score, a Pareto frontier across coding quality and tokens per second, or a constrained optimization problem?
- Were alternative configurations fully trained, partially trained, or evaluated using proxy estimates? The announcement's word “projected” makes this distinction particularly interesting.
- Did the search explore alternatives near 39.8B total / 2.5B active, such as a similar total parameter count with fewer active parameters, or a smaller total model with the same active budget?
- Which specific changes drove the improvement: more experts, narrower experts, a different number of selected experts, or a combination?
- Were the selected dimensions optimal for a particular hardware and memory budget, or did they generalize across the target devices?
For research reproducibility, even a summary of the candidate configurations, ranking criteria, and measured or estimated quality-speed trade-offs would be extremely useful.
It is understood that the full internal search history may not be public.
To be able distinguish the general design principle from the particular architecture that performed best in your tested search space would help us better to understand the Eureka Machine.