Pretrained Hybrids with MAD Skills

Authors: Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, Frederic Sala
Primary area: foundation or frontier models, including LLMs
Keywords: hybrid architectures, large language models, transformers, state space models, model merging, neural architecture search, mechanistic search

Abstract:
While Transformers underpin modern large language models (LMs), a growing list of alternative architectures with new capabilities, promises, and tradeoffs is emerging. This makes choosing the right LM architecture challenging. Recently proposed *hybrid architectures* seek a best-of-all-worlds approach that reaps the benefits of all architectures. Hybrid design is difficult for two reasons: it requires manual expert-driven search, and new hybrids must be trained from scratch. We propose **Manticore**, a framework that addresses these challenges by *automating the design of hybrid architectures* while reusing pretrained models to create *pretrained* hybrids. Our approach augments ideas from differentiable Neural Architecture Search (NAS) by incorporating simple projectors that translate features between pretrained blocks from different architectures. We then fine-tune hybrids that combine pretrained models from different architecture families---such as the GPT series and Mamba---end-to-end. With Manticore, we enable LM selection without training multiple models, the construction of pretrained hybrids from existing pretrained models, and the ability to *program* pretrained hybrids to have certain capabilities. Manticore hybrids match existing manually-designed hybrids, achieve strong performance on the Long Range Arena benchmark, and improve on pretrained transformers and state space models on various natural language tasks.

Decision: reject
Reviewer scores: 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730652664682

thread_type:
official_review

<thread_object>
creator: reviewer_dze5

<content>
summary:
Manticore is a framework for automating the creation of hybrid language model architectures by reusing pre-trained models from different architectures. By leveraging NAS and projectors, Manticore enables pre trained hybrid models that combine the strengths of multiple architectures, allowing for flexible and high-performing LLMs without extensive manual design or retraining.

strengths:
This paper focuses on an important and relevant problem of using architectural components from different state-of-the-art model architectures to construct a hybrid model that provides the best of all worlds without incurring expensive pre-training and search space exploration overheads.

It introduces a novel idea of projectors that enable different architectures to interact in each other feature space by projecting an intermediate shared feature space that acts as a translator for them.

weaknesses:
Although the idea of projectors is novel but using gating to combine the contributions of different architectures has been explored in Mixture of experts [1], weighted ensemble averaging and finds a direct use in this paper.

The evaluation compares the combined hybrid that has 2x the number of parameters and >2x FLOPs due to projectors and gating against individual models of half the size.

In Table 1, Mamba is already better than Pythia in all of the tasks, in Table 2, Mambaformer is also better in all tasks but one, In Table 4, GPT-Neo is better than all tasks, So creating a hybrid is achieving almost the same score as the better model.

[1] https://arxiv.org/abs/1701.06538

questions:
A fair comparison would either be a model of the same architecture with 2x size or the hybrid being discretized and having the same amount of FLOPs/parameters of the one of the individual models. This would address weakness 2.

 A stronger test would be to consider models that perform better in some tasks than others and then validate if the hybrid does achieve the best of both worlds. This would address the weakness 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679599814

thread_type:
official_review

<thread_object>
creator: reviewer_sexn

<content>
summary:
The paper introduces Manticore, a framework for creating hybrid architectures by combining pretrained components from different language models (LMs). Manticore aims to automate hybrid model design by reusing pretrained model components from distinct architectures, such as Transformers and state-space models, using projectors to align feature representations between these models. The framework's goal is to merge model architectures in a way that preserves their respective strengths, ideally achieving better performance than the individual components. Through experiments on the Long Range Arena (LRA) and MAD tasks, Manticore demonstrates comparable or improved performance over some individual models and existing hybrid models.

strengths:
+ Manticore’s approach to combining pretrained models from different architectures using projectors and mixture weights is innovative and extends beyond typical model merging methods.
+ Manticore’s design, which allows for fine-tuning and programming pretrained hybrids, offers a degree of flexibility, making it potentially beneficial for practitioners looking to leverage diverse model architectures.
+ Testing across LRA and MAD tasks provides an initial sense of the framework's potential, although the evaluation depth limits the conclusions drawn.

weaknesses:
- The main claim, "Pretrained hybrids can outperform their component models on fine-tuning tasks," is not well-supported. A fair comparison would entail fine-tuning Manticore and its component models under the same budget to evaluate relative gains. Without this, it’s unclear whether the hybrid approach provides substantial benefits beyond those of individually optimized models.
- Manticore requires a dedicated training process for projector layers and mixture weights, potentially adding overhead and limiting applicability in constrained environments.
- The absence of publicly available code restricts reproducibility and verification of the results, limiting the community's ability to assess the framework’s impact fully.

questions:
1. How would Manticore perform relative to its component models if they were fine-tuned under the same computational budget? Would this comparison validate Manticore’s primary claim of outperforming its component models?
2. Could the authors elaborate on the memory and computational efficiency of Manticore hybrids, especially compared to traditional single-model architectures? Does the use of projectors introduce significant overhead?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709875724

thread_type:
official_review

<thread_object>
creator: reviewer_nca4

<content>
summary:
The paper builds by proposing a new framework for automating the design of hybrid language models by re-using existing pre-trained models using ideas from Neural Architecture Search. The authors show that their approach allows for merged language models to be competitive with their component models and also outperform them on fine-tuning tasks.

strengths:
1. The paper is well-written and the the motivation is clear and convincing.
2. The figures are well-designed and helpful.
3. Interesting results that prove the effectiveness of their framework on specific models.

weaknesses:
1. The experiments done are mostly on smaller models and it's not clear if MAD remains effective with larger models. The authors could do additional experiments with other open-source models to validate this.

questions:
1. Have you done experiments to validate whether the performance continues to hold at scale?
2. What is the overhead of the projectors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732411786646

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
Thank you all for the helpful comments, insights, and questions! Your responses included many excellent questions that have allowed us to highlight the advantages of Manticore.

Since submission, we have added a variety of new content to the paper. This includes new baselines, insightful analyses, clarifying discussions, and an anonymous link to our code repository. We have highlighted these changes to the text in red. 

**Most importantly, in Appendix D, we have added a thorough discussion of the tradeoffs and challenges associated with selecting baselines under different definitions of a “budget.”** To summarize, we cover three different definitions:

**Parameter count**
- Comparison along the axis of parameter count is challenging because larger models are typically pretrained for longer and pretrained models of specific sizes aren’t guaranteed to always exist. For example, if we were to combine Mamba-2.8B and GPT-Neo-2.7B, we would not be able to obtain 5.5B parameter Mamba and GPT-Neo/Pythia baselines for comparison to the Manticore hybrid. 

**Training FLOPs**
- The fact that we compare Manticore to the better of two component models means that both component models need to be fine-tuned in order to compare Manticore to the best component model–this is the baseline that we compare against in the main text. We conduct a training FLOPs analysis and show that the total training FLOPs contribution is dominated by pretraining, and note that the (already relatively small) projector pretraining cost is amortized over potentially many fine-tuning runs. 

**Inference FLOPs**
- It is true that our component model baselines naturally incur only about half of the inference FLOPs required by Manticore. We show that the inference FLOPs contribution of the projectors is minimal, and our analysis motivates a new baseline which we have added: ensembles of component models. **Manticore outperforms this baseline while roughly matching the inference FLOPs and total training FLOPs.** 

We are excited about these new additions to the paper, and we would love to dive into more detail on these new results should you have any questions about them.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel framework for building hybrid language models from pretrained components. While reviewers praised the idea and clarity, they raised concerns on fairness of comparisons, scalability, and demonstration of true hybrid benefits over simpler baselines. The authors provided additional analyses and new baselines, but concerns remained. Although the approach is promising, the paper currently falls short of acceptance. Future work should include more rigorous baseline comparisons, larger-scale experiments, and stronger evidence that hybrids outperform simpler ensembles. Unfortuantely, at this time, rejection is recommended.