Title: Chain-of-Model Learning for Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Foundation Model

Next-Gen Language Models

Chain-of-Model

Dynamic Activation

Efficiency
Abstract: In this paper, we propose a novel learning paradigm, termed *Chain-of-Model* (CoM), which incorporates the causal relationship into the hidden states of each layer as a chain style. thereby introducing great scaling efficiency in model training and inference flexibility in deployment.We introduce the concept of *Chain-of-Representation* (CoR), which formulates the hidden states at each layer as a combination of multiple sub-representations (i.e., chains). In each layer, each chain from the output representations can only view all of its preceding chains in the input representations. Consequently, the model built upon CoM framework can progressively scale up the model size by increasing the chains based on the previous models (i.e., chains), and offer multiple sub-models at varying sizes for elastic inference by using different chain numbers. Based on this principle, we devise *Chain-of-Language-Model* (CoLM), which incorporates the idea of CoM into each layer of Transformer architecture. Based on CoLM, we further introduce CoLM-Air by introducing a *KV sharing* mechanism, that computes all keys and values within the first chain and then shares across all chains. This design demonstrates additional extensibility, such as enabling seamless LM switching, prefilling acceleration and so on. Experimental results demonstrate our CoLM family can achieve comparable performance to the standard Transformer, while simultaneously enabling greater flexiblity, such as progressive scaling to improve training efficiency and offer multiple varying model sizes for elastic inference, paving a a new way toward building language models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **Chain-of-Model (CoM)**, a progressive learning paradigm inspired by Progressive Neural Networks, where newly added neurons are causally dependent on the preceding ones. CoM is applied to progressively expand the **width** of Transformer models. The authors provide detailed implementations for each module of the Transformer architecture (e.g., FFN, multi-head attention, normalization, and embedding). Experimental results demonstrate the effectiveness of CoM in model expansion, elastic inference, and continual learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The proposed method is intuitive and well-motivated for scaling the width of Transformers. The design is supported by experimental evidence.\n- The paper provides a clear and detailed description of the method, including pseudocode and implementation details.\n- The experiments cover a variety of scenarios, such as model expansion, tuning, and inference.\n- The paper is well-written and easy to follow.\n\n\n**Weaknesses**\n\nWhile the method is novel and well-implemented, its **superiority over existing approaches remains unclear**. Although the experiments cover multiple settings, **comparisons with strong baselines are missing**, making it difficult to assess the practical advantages of CoM.\n\n1. **Comparison in 'Chain Expansion' (Section 4.2)**:\n    The authors should compare CoM with other model expansion techniques, such as vanilla width expansion.\n\n   > For example, when expanding the representation space from NNN to N+MN+MN+M, CoM adds a matrix of shape (N+M,M)(N+M, M)(N+M,M). How would this compare to simply adding a matrix of shape (M,M)(M, M)(M,M) without causal constraints?\n   >  This ablation could help validate the core contribution of CoM—namely, the necessity of causal dependencies across chains.\n\n2. **Comparison in 'Chain Tuning' (Section 4.5)**:\n    The paper should include comparisons with parameter-efficient fine-tuning methods such as **LoRA**, to show how CoM performs in terms of efficiency and performance in limited-resource settings.\n\n3. **Lack of comparison with MoE**:\n    The authors claim that CoM is compatible with Mixture-of-Experts (MoE) models. However, a natural question arises:\n\n   > Is it more effective to use CoM to increase the capacity of each expert, or to increase the number of experts directly?\n   >  Both approaches aim to scale model capacity—some comparison or analysis would be valuable.\n\n4. **Concerns about evaluation tasks**:\n    All evaluated benchmarks appear to be **multiple-choice tasks**. It would be beneficial to include results on **generative tasks**, which may better reflect the model's performance in real-world use cases.\n\n\n\n\n**Overall**\n\nThis work addresses an important problem: how to progressively scale Transformer models without retraining from scratch. The method is well-structured and the engineering implementation is solid. However, to fully validate its contribution, the paper should include **comparisons with relevant baseline methods** and consider **broader evaluation settings**, including generative tasks.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel learning paradigm, termed \"Chain-of-Model\" (CoM), which aims to address the core challenges in the model scaling and deployment of current Large Language Models (LLMs). Its core idea originates from \"Chain-of-Representation\" (CoR), which treats the representation vectors in the model's hidden layers as a concatenation of multiple \"sub-chains\". The authors  apply this paradigm to the Transformer architecture, designing the \"Chain-of-Language-Model\" (CoLM). Experimental results show that this method performs slightly worse than the vanilla Transformer in terms of effectiveness, but has advantages in incremental learning, elastic inference, and efficient deployment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The \"Chain-of-Representation\" idea proposed in this paper is highly innovative. The resulting model is inherently capable of multi-scale adaptation, which can solve many problems in incremental learning, elastic inference, and efficient deployment.\n\n- The paper provides a thorough discussion on various aspects of the Chain-of-Representation concept, including model training  inference, etc.\n\n- The experimental results demonstrate that the proposed method achieves good performance.\n\n**Weaknesses:**\n\n- Potential Impact of KV Sharing on Generation Quality: The paper mentions that the KV sharing mechanism \"slightly affects the performance\" , a claim verified by the accuracy on commonsense reasoning tasks. However, the specific impact of this on open-ended text generation tasks (e.g., in terms of coherence, logic, and creativity) remains unclear. Since higher-order chains rely on the semantic information (K and V) generated by lower-order chains, this could lead to an information bottleneck. Providing some qualitative analysis or results on generation-focused benchmarks would allow for a better assessment of the trade-offs of this design.\n    \n- Compatibility Issues with Tensor Parallelism (TP): The authors acknowledge in the appendix that the naive implementation of the Chain-of-Linear layer is not well-suited for Tensor Parallelism (TP), which is a very practical obstacle for training very large-scale models. Although the authors mention optimization via a custom-developed kernel, the paper does not provide a clear answer as to whether the additional communication overhead brought by TP can be fully resolved. This could be a significant barrier to the widespread adoption of this architecture in the industry.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, the authors propose a novel learning paradigm called \"Chain-of-Model\" (CoM). Within this paradigm, the hidden representations are divided into multiple sub-representations, each of which represents a different scale. With such division of hidden representations, the authors propose the concept of \"Chain-of-Layer\" (CoL), where the scales are learned causally within a layer. That is, the scale $i$ of the layer output only depends on the scales $\\leq i$ of the layer input. By stacking layers that satisfy the CoL property, the entire model becomes CoM. Based on this concept, the authors redesign the common layers for language models to satisfy the CoL property and propose Chain-of-Language-Model (CoLM). The proposed CoLM enjoys several properties: 1) The existing LLMs can be treated as chain=1 and easily extended by adding another chain without training from scratch. 2) A model trained with CoLM can be deployed with dynamic inference capacity. 3) CoLM models with KV sharing achieve better pre-filling latency. These properties are evaluated on 1B scale model pertaining with several common-sense reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper studys an important problem as scaling LLM models is extremely resource-consuming. The proposed CoLM could extend an existing model with more chains (more capacity) without training from scratch, which provides a valuable solution for the community\n2. Though there are several weaknesses, the proposed method is well-presented in the paper and looks quite novel to me\n\nWeakness:\n1. The proposed CoLM is mainly validated on models with 1B parameters, which seems not enough to guarantee that it will remain the same performance for large scales such as 3B, 8B or even larger. Though it is understandable that pre-training 3B/8B model is resource-restricted, it would be nice to see experiments where the proposed CoLM is used to extend some larger models such as Llama-3.2-3B and Llama-3.1-8B. \n2. In Table 2, although the CoLM expansion introduces additional 0.8B parameters, the improvement of LLaMA-3.2-1B looks trival to me given that it is trained with 8B tokens. \n3. Since the proposed CoLM makes changes to the layer structure. It is unclear whether the proposed method is compatible to the existing large-scale training techniques (e.g., Different parallelism).\n4. As the author mentioned, each scale should have its own loss function and that potentially limits the training efficiency when the number of chains is huge \n5. With the KV sharing mechanism, the key/value heads only contain the information of the first scale. When the model is expanded with multiple rounds, the model capacity will be limited by the key/value heads.\n\nQUESTIONS:\n1. The proposed CoLM is mainly tested on Llama variants. Is the proposed CoLM applicable to other model series such as Qwen and DeepSeek MLA? \n\n2. For the chain expansion experiment, each model is only expanded with 1 chain. It would be good to see the performance of extending more chains (e.g., $c_2=4, c_3=4$) and discuss the pros/cons.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Chain-of-Model (CoM) learning: split every hidden vector into ordered “chains,” force each output chain i to depend only on the first i input chains (Chain-of-Layer, CoL) and stack such layers to obtain a network that contains several nested sub-models. The instantiated Chain-of-Language-Model (CoLM) keeps the first chain identical to a standard Transformer and adds extra chains that can be trained later (chain expansion) or dropped at inference time (elastic inference) without recomputing KV caches thanks to a KV-sharing trick (CoLM-Air) .\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1- The paper is well-written and it is easy to understand. The application is technically sound and potentially useful as it shows an architectural innovation with tangible speed.\n\n2- The method is innovative and tries to tackle an interesting problem.\n\n3- Demonstrates that a single checkpoint can serve multiple width targets with negligible quality loss.\n\nWeaknesses:\n\n1- Some low level details are missing. For example in section 3.2.3 the details of normalisation scheme has not explained.\n\n2- Performance does decline as more chains are added, and even the two-chain CoLM-Air lags the dense baseline by ~0.3 points. Although that margin is still modest (well under one point), it indicates that the KV-sharing trick may exact a small but measurable accuracy cost, which could widen at larger model scales or on harder benchmarks. Demonstrating results at ≥7 B parameters (or providing significance tests to show whether 0.3 is within noise) would help establish how serious the penalty really is.\n\n3- Based on the speed-up curves in Figure 5, latency gains emerge only when the pre-fill prompt exceeds roughly 4 K tokens; below that, CoLM is nearly as slow as the dense baseline. This implies the architecture is chiefly advantageous in long-context settings. Yet all reported generalization scores rely on short-context LM-Harness benchmarks or GLUE tasks. To demonstrate practical value, the authors should also measure long-context accuracy (e.g., on Needle-In-a-Haystack or LongBench) to verify that KV-sharing retains quality.\n\nQUESTIONS:\n1- Which exact normalization scheme do you use (RMSNorm vs. LayerNorm)? Please include the details of how did you adapt the layer norm in your method.\n\n2- Does CoLM (with and without KV sharing) maintain accuracy in long sequence scenarios?\n\n3- How does the method behave at ≥7 B parameters? Do the small accuracy gap of CoLM-Air persist?\n\nLIMITATIONS:\nThe paper has no separate limitation section. Including a short “Limitations & Societal Impact” subsection covering the points such as Scalability, Long-context quality etc. would make the submission more transparent and align with NeurIPS best practices.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}