Title: InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Fusion

LLM
Abstract: Recent advances in large language models (LLMs) have intensified efforts to fuse heterogeneous open-source models into a unified system that inherits their complementary strengths.  Existing logit-based fusion methods maintain inference efficiency but treat vocabulary dimensions independently, overlooking semantic dependencies encoded by cross-dimension interactions. These dependencies reflect how token types interact under a model's internal reasoning and are essential for aligning models with diverse generation behaviors. To explicitly model these dependencies, we propose \textbf{InfiGFusion}, the first structure-aware fusion framework with a novel \textit{Graph-on-Logits Distillation} (GLD) loss. Specifically, we retain the top-$k$ logits per output and aggregate their outer products across sequence positions to form a global co-activation graph, where nodes represent vocabulary channels and edges quantify their joint activations. To ensure scalability and efficiency, we design a sorting-based closed-form approximation that reduces the original $O(n^4)$ cost of Gromov-Wasserstein distance to $O(n \log n)$, with provable approximation guarantees. Experiments across multiple fusion settings show that GLD consistently improves fusion quality and stability. InfiGFusion outperforms SOTA models and fusion baselines across 11 benchmarks spanning reasoning, coding, and mathematics. It shows particular strength in complex reasoning tasks, with +35.6 improvement on Multistep Arithmetic and +37.06 on Causal Judgement over SFT, demonstrating superior multi-step and relational inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces InfiGFusion, a novel framework for fusing heterogeneous large language models (LLMs) that overcomes the limitations of prior logit-based methods. The authors argue that existing techniques treat vocabulary dimensions independently, failing to capture the crucial semantic dependencies and co-activation patterns that underlie a model's reasoning process. To address this, InfiGFusion proposes a structure-aware approach centered on a new Graph-on-Logits Distillation (GLD) loss. This method constructs feature-level graphs from the logits, where nodes represent vocabulary dimensions and edges quantify their joint activations, thereby capturing the model's internal reasoning structure. To make this approach computationally feasible, the authors develop a novel, sorting-based closed-form approximation of the Gromov-Wasserstein (GW) distance, reducing its complexity from a prohibitive \n$O(n^4)$ to a highly efficient $O(n\\log n)$ with provable approximation guarantees. Experiments across 11 benchmarks demonstrate that InfiGFusion significantly outperforms existing fusion methods and baselines, showing particular strength in complex reasoning, mathematics, and coding tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Existing methods that align output logits often treat vocabulary dimensions independently, ignoring the rich semantic relationships and dependencies that underpin a model's reasoning process. InfiGFusion is the first framework to propose a structure-aware approach, representing logit co-activation patterns as graphs and aligning these structures. This is a fundamentally new and promising direction for model fusion.\n\n- The method shows dramatic improvements on tasks that require multi-step and relational inference, such as a +35.6 point gain on Multistep Arithmetic and a +37.06 point gain on Causal Judgement over the SFT baseline. This provides strong evidence that capturing semantic structure is crucial for fusing complex reasoning abilities.\n\n-  A major challenge of using the Gromov-Wasserstein (GW) distance is its prohibitive computational complexity ($O(n^4)$). A key contribution of this work is a novel sorting-based approximation that reduces the complexity to a practical  $O(n\\ log n)$. Crucially, the authors provide a  provable error bound for this approximation, ensuring its fidelity to the original metric.\n\n## Weaknesses\n\n- The authors explicitly state in the NeurIPS checklist that they did not report error bars or conduct statistical significance tests, citing computational cost. While many of the reported gains are substantial, this is a notable omission for an empirical paper. Without statistical validation, it is difficult to be certain that smaller performance improvements are meaningful and not simply the result of experimental noise.\n\n- The paper acknowledges that the method focuses on aligning relational consistency and lacks an explicit mechanism for resolving factual conflicts between source models. This is a significant limitation in practice, as fusing models with contradictory knowledge could lead to unpredictable or factually incorrect, though structurally coherent, outputs.\n\n- The authors note that the benefits of GLD are marginal for tasks dominated by factual recall, where simple token matching is sufficient. While this is a reasonable trade-off, a more in-depth discussion of when structural alignment might be unnecessary or even detrimental would strengthen the paper.\n\nQUESTIONS:\nCould the authors please provide error bars or significance tests for a critical subset of the results? I would suggest focusing on:\n\n- The main average scores in Table 2, comparing InfiGFusion to the Pivot-SFT and the next-best fusion baseline.\n\n- The most dramatic gains on complex reasoning tasks reported in Table 3, specifically \"Multistep Arithmetic\" and \"Causal Judgement,\" which are central to the paper's narrative.\n\nLIMITATIONS:\nplease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new model fusion method via knowledge distillation (KD) where the inner relationship between tokens are taken into consideration by adding a distance on the token probability space as additional KD loss. Using an expensive Gromov-Wasserstein distance to measure the internal structure similarity on the token probability space, the authors further propose an approximation to significantly reduce the computation complexity. The experiments show that the methods successfully fuse different capacity from different source models and outperformed than baseline fusion methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea to use the relation between the tokens to further align the models is very impressive. The authors keenly discovered the structural relationship implicit in the output token probabilities and used this relationship to model the properties of a certain task. I believe the motivation is well-explained.\n2. The approximation seems useful. And the key insight \"the structure of each similarity matrix can be summarized using scalar node-level features\" is crucial to make the expensive distance computable.\n3. The experiments successfully proof the effectiveness of the methods. The results support the authors' claim well. The datasets contain many different fields.\n4. The paper is well-organized and reader friendly.\n\nWeaknesses:\nI do not find major weaknesses but I have some minor suggestions about the paper.\n1. Although the related work is well explained. I believe the paper [34] (or maybe other paper which first achieve cross tokenizer fusion)  should have been introduced earlier in the text to allow readers to better appreciate its innovations and avoid potential misunderstandings during reading.\n2. Besides the absolute error bound, the approximation analysis should also include a relative error analysis. Specifically, when the error bound is about O(1/N), what is generally the scale of the distance? The relative error is the key to determining whether the approximation is suitable. And I believe it is better to show some empirical results on the approximation error itself. At the same time, I understand the original O(N^4) is very expensive but I really wonder what is the performance without the approximation. I wonder whether an ablation study here is possible, (maybe just for explain the upper bound of the performance?).\n3. The current case study is based on the output sequence, which could be influenced by so many factors. I believe a more straight-forward case study (and also I are really curious about) is the actual learned tokens relation for different models and tasks. For example, can the distance really help models to learn the period of digit 1-9? Or whether some real-world concept like \"cities\" can be modeled by your methods? I really wonder whether there is some results about the interpretability of the models.\n\nQUESTIONS:\nSee above.\n\n1. The relative error analysis and empirical error analysis.\n2. Is it the case that without the GW distance approximation the methods can learn better?\n3. More case study based on the learned token relation instead of the output token sequence.\n\nIf at least two questions have been answered with an explicit answer, I will raise my score to 6.\n\nLIMITATIONS:\nI do not find the discussion about the limitation. I suggest the authors should add a limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper proposes to incorporate a new term into the LLM distillation objective. The new term is based on Gromov-Wasserstein distance that aims to preserve relations between top-k logits for each token in a sequence. Empirical results demonstrate improvement across several benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
- I found the idea of incorporating GW into distillation interesting. Proposed GW approximation also makes it practical.
- Empirical results overall seem positive and the ablation studies are fairly detailed.

Weaknesses:
- I'd appreciate a more detailed analysis of the GW approximation. Perhaps with a simple simulation study comparing it to other (less efficient) approaches to computing GW.
- Ablation study in Table 4 suggests that gains due to fusing more than 2 models are fairly marginal. Main results (Table 2) also demonstrate that the fused model outperforms all of the source models in only about half of the tasks. While I acknowledge that empirical results are good overall, the (ideal) goal of a fusion method is not only to improve on average, but to also consistently outperform source models and benefit from additional source models.
- Why did you choose k=30 for ablation studies if the optimal k=10? Does it mean that in Table 4 results for singleton source models are with k=30 and InfiGFusion results are with k=10?
- The paper repeatedly states that the proposed GW term captures "internal reasoning". I don't think this claim is supported in the paper. While the GW term seems useful, how is it related to "internal reasoning"?

QUESTIONS:
See weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces INFOBLEND, a caching system for accelerating inference in Multimodal Large Language Models (MLLMs) by enabling position-independent reuse of key-value (KV) caches. Unlike traditional prefix-based context caching (CC) systems that can only reuse KV caches when the prefix is an exact match, INFOBLEND supports partial and flexible reuse of multimodal caches, even when text and image inputs are interleaved or reordered. The system tackles both system-level and algorithm-level challenges by storing large KV caches on disk and loading them in parallel, introducing a selective attention mechanism that selectively recomputes critical tokens to avoid accuracy degradation, and achieving significant improvements in response time (TTFT) and throughput with minimal quality loss. The approach is validated through experiments on two MLLMs (LLaVA 1.6 Vicuna-7B and Mistral-7B) across MMDU and SparklesEval datasets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n1. It proposes a hybrid reuse-recompute strategy based on attention sparsity and token impact, which is insightful and well-motivated.\n\n2. It demonstrates a 54% reduction in TTFT and 2× throughput over prefix caching with only minor quality loss (~13.6%).\n\n3. The paper thoughtfully combines systems design (e.g., parallel KV cache loading) and algorithmic insight (e.g., selective attention) for real-world deployment.\n\n### Weaknesses:\n\n1. The experiments are conducted only on 7B-parameter MLLMs. While understandable given hardware constraints (single H800 GPU), it limits the generalizability to large-scale production settings.\n\n2. There is no formal analysis of the quality degradation due to partial reuse or how selective attention scales with number of modalities and sequence length.\n\n3. The token selection process (based on K-distance and attention) may add computational complexity or engineering burden in practice.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}