Title: InsNeXt: Training Scalable Insertion-based Language Models from Scratch
Venue: NeurIPS 2025
Accepted: no
Keywords: language model

controllable generation

insertion-based model
Abstract: Insertion-based language models like Insertion Transformer and InsNet have shown promises as strong alternatives to autoregressive models with better inference-time efficiency and controllablility. However, their training-time scalability has been limited by computational inefficiency and obsolete model designs. We aim to tackle this problem with \textbf{InsNeXt}, an insertion-based language model architecture integrating recent advancements of language model systems to achieve improved scalability. We scale InsNeXt from 154M up to as large as 0.6B parameters with context window of 4096 by combining sentence-level training and document-level training to better encode the context and bring out the benefits of insertion-based models to encode bi-directional contexts. In addition, we propose a novel context encoding mechanism specialized for insertion-based decoding. The inference-time mechanism sparsely introduces bidirectional re-encoding of context, thus effectively leverages the models' bidirectional context reception while preserving the same level of computational efficiency as conventional autoregressive decoding. We evaluate the pretrained InsNeXt models from the perspective of representation learning, commonsense reasoning and controllable generation. InsNeXt models achieve similar or  better performance in comparison to the state-of-the-art similar-sized autoregressive models, making them a class of solid representation learners and powerful controllable insertion-based generators.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents InsNeXt, a new scalable insertion-based language model that integrates recent innovations from large language model training (e.g., efficient attention, improved normalization, and flexible position encodings). The work addresses key inefficiencies in previous insertion-based models such as InsNet, proposing novel techniques like efficient bidirectional re-contextualization and soft-capped position logits. The authors train models from 154M to 573M parameters and evaluate them on natural language understanding (GLUE), commonsense reasoning (HellaSwag, WinoGrande, OBQA), and controllable generation tasks (CommonGen, Yelp). The proposed model demonstrates strong controllability and competitive performance compared to autoregressive baselines.\n\nSTRENGTHS AND WEAKNESSES:\nHonestly, I am not an expert in this field, but I would like to give some comments based on my understanding and background knowledge:\n\nStrengths:\n- Presents a modern and scalable insertion-based LM, which is timely given growing interest in controllable and efficient generation.\n- Proposes methodologically novel components, such as the bidirectional re-contextualization strategy and soft-capped position logits, addressing known limitations in insertion-based models.\n- Offers extensive empirical evaluation across NLU, reasoning, and generation tasks, with comparisons to autoregressive and encoder-only models. Provides clear architectural ablations and implementation details that contribute to reproducibility.\n\nWeaknesses:\n- While the model shows improvements over prior insertion models, its representation quality still lags behind SOTA encoder-based models like RoBERTa, limiting its applicability in high-performance NLU tasks.\n- The re-contextualization mechanism, though efficient in expectation, introduces computational spikes and its cost is not fully quantified.\n\nQUESTIONS:\n1. The proposed bidirectional re-contextualization mechanism aims to mitigate exposure bias while maintaining efficiency. Can the authors provide quantitative measurements of how often re-encoding is triggered during inference and its relative impact on decoding latency and throughput?\n2. InsNeXt still underperforms RoBERTa and BERT-large on NLU tasks like MNLI. What are the authors’ hypotheses on this gap? Is it due to the insertion paradigm itself, or could further scaling or architectural tuning (e.g., attention heads or deeper layers) close this gap?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed InsNeXt that integrates the recent advancement of LM to achieve improved scalability. The InsNeXt is designed to overcome the training inefficiencies and architectural limitations of earlier insertion-based approaches (e.g., InsNet). The authors build upon and improve prior work by integrating modern LLM advances such as efficient attention mechanisms (e.g., FlexAttention), improved normalization (e.g., pre-LN, DeepNorm), and optimized training pipelines. The InsNeXt is scaled from 154M to 0.6B parameters. Furthermore, a novel context encoding mechanism is proposed for insertion-based decoding. During the inference time, a sparse re-encoding mechanism is proposed. The evaluation on tasks such as representation learning, commonsense reasoning, and controllable generation show that the InsNeXt models are competitive as the state-of-the-art autoregressive models with similar size.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- I would acknowledge the effort of thorough ablation studies conducted in this work. The paper details an exhaustive ablation of model components to reveal good practices to the research community.\n- One novelty brought by the paper is the efficient bidirectional re-contextualization, that addresses exposure bias and better handling the user-input prompt.\n- The model is tested across representation, reasoning, and generation, offering a well-rounded assessment of its strengths. The InsNeXt-base model outperforms its baseline InsNext. \n### Weaknesses\n- Performance itself is still a concern to me. While it is competitive, InsNeXt does not match SOTA discriminative models like RoBERTa-base (when compared with InsNeXt-base) and RoBERTa-large (when compared with InsNeXt-advanced) in representation learning, suggesting some architectural constraints in my mind. But I did acknowledge the model performance on other task categories.\n- While position encoding is a core design point (InsNet-style sinusoidal vs. insertion-aligned ALiBi), the paper lacks direct side-by-side comparisons across tasks and scales. It's unclear how each encoding affects models e.g. generalization etc.\n\nQUESTIONS:\n- Any statistics on the training/inference efficiency?\n- Do you think if there are any challenges (algorithm aspect or training aspect) when the model size is further scaled up?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an insertion based model called InsNeXt, which is improved based on InsNet. For training, it combines sentence-level pretraining (on Wikipedia and books dataset) and document-level pretraining (on SlimPajama dataset). For inference, it uses a proposed bidirectional context re-encoding mechanism, which keeps a bidirectional encoded part and only use unidirectional updates until the generated tokens surpass the bidirectional part.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. This paper integrates components and methods from autoregressive LLMs to insertion-based models, so it is more scalable than InsNet.\n2. They have done study on various alternative designs for different components of insertion-based models to select the best solutions.\n3. The experimental and model details are clearly stated in the paper.\n\n\nWeakness\n1. Most important works, such as overcoming the volatile position problem, has been done by InsNet. The main innovative contribution is Efficient Bidirectional Re-contextualization in 2.4, but it is simple and lacks proof. Please add more explanation in 2.4.\n2. In Section 2, revisiting of the existing InsNet, ablation study on alternative design details, and the novel proposed idea are all included. Please make them separate sections.\n3. It is claimed that this model has better computational efficiency, but no theoretical proof or experimental numbers is shown.\n4. Insertion-based models can potentially be more efficient than autoregressive models in specific tasks, such as code editing, but its special feature is not well utilized to design a novel efficient method.\n\nQUESTIONS:\nPlease prove that the expected computational overhead of a decoding process with the proposed re-contextualization is at the same scale of the vanilla uni-directional one.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces InsNeXt, a new insertion-based language model designed to address the scalability limitations of previous models like Insertion Transformer and InsNet during training. Insertion-based models offer advantages over autoregressive models due to their inference-time efficiency and controllability, allowing tokens to be generated at arbitrary positions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper directly tackles the training-time inscalability issue of previous insertion-based models. InsNeXt's ability to scale up to 0.6B parameters and handle larger context windows (4096 tokens) is a substantial improvement, making these models more practical for real-world applications.\n2. The adoption and refinement of the offset matrix for positional encoding (building on InsNet) is a strong point. This approach correctly addresses the \"volatile position problem\" inherent in insertion-based models, ensuring stable and efficient training by preserving previously computed position encodings.\n\nWeakness\n1. The paper describes a mechanism designed for efficiency, it seems that the authors do not provide enough quantitative data in the experimental sections to prove that this mechanism doesn't introduce significant, practical inference latency trade-offs, especially compared to models that don't employ such re-encoding or to \"conventional autoregressive decoding\" as claimed.\n2. The experiment is not sufficient and scalable.\n    - The involved model size for pretraining is only 0.6B, which cannot verify the scalability of the proposed method.\n    - The paper highlights a \"two-stage training\" process (sentence-level and document-level) as crucial for better context encoding and scalability. However, the paper lacks detailed ablation studies to demonstrate the individual contribution and necessity of each training stage.\n\nQUESTIONS:\n1. How does InsNeXt perform when faced with extremely long sequences requiring many insertions, or when a high density of insertions needs to be made in a very short span? \n2. Are there specific types of text generation where the insertion mechanism might introduce artifacts or degrade coherence?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-11"}