More Experts Than Galaxies: Conditionally-Overlapping Experts with Biologically-Inspired Fixed Routing

Authors: Sagi Shaier, Francisco Pereira, Katharina von der Wense, Lawrence Hunter, Matt Jones
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Deep learning, Mixture of Experts, Modularity, Sparsity, Conditional Computation

Abstract:
The evolution of biological neural systems has led to both modularity and sparse coding, which enables energy efficiency and robustness across the diversity of tasks in the lifespan. In contrast, standard neural networks rely on dense, non-specialized architectures, where all model parameters are simultaneously updated to learn multiple tasks, leading to interference. Current sparse neural network approaches aim to alleviate this issue but are hindered by limitations such as 1) trainable gating functions that cause representation collapse, 2) disjoint experts that result in redundant computation and slow learning, and 3) reliance on explicit input or task IDs that limit flexibility and scalability.
In this paper we propose Conditionally Overlapping Mixture of ExperTs (COMET), a general deep learning method that addresses these challenges by inducing a modular, sparse architecture with an exponential number of overlapping experts. COMET replaces the trainable gating function used in Sparse Mixture of Experts with a fixed, biologically inspired random projection applied to individual input representations. This design causes the degree of expert overlap to depend on input similarity, so that similar inputs tend to share more parameters. This results in faster learning per update step and improved out-of-sample generalization. 
We demonstrate the effectiveness of COMET on a range of tasks, including image classification, language modeling, and regression, using several popular deep learning architectures.

Decision: accept (poster)
Reviewer scores: 6, 8, 3

Reviews and discussion:
<thread>
creation date:
1730143532424

thread_type:
official_review

<thread_object>
creator: reviewer_rveg

<content>
summary:
The paper, titled Conditionally Overlapping Mixture of Experts (COMET), proposes a method aimed at overcoming limitations in existing sparse neural network architectures. COMET introduces a modular, sparse structure with a biologically inspired fixed routing approach that eliminates the need for task IDs and trainable gating functions, commonly associated with representation collapse and redundancy. Instead, the authors implement a k-winner-take-all cap operation, enabling experts to overlap based on input similarity. This approach aims to improve generalization and facilitate faster learning, validated across various tasks and architectures, including MLPs, Vision Transformers, and GPT-based models.

strengths:
1. COMET presents a novel routing method that replaces trainable gating functions with fixed, biologically inspired routing, which is rare in modular neural network approaches.

2. The proposed method is tested across diverse architectures and tasks, such as image classification, language modeling, and regression, suggesting versatility.

weaknesses:
1. Section 3 lacks crucial methodological details necessary for a complete understanding of the proposed COMET approach. For instance, it is unclear which specific design elements in COMET were directly influenced by the concept of biological random projections. 

2. The paper’s writing lacks clarity, making it difficult to fully understand the design of COMET. I recommend including a preliminary section that outlines the foundational Mixture of Experts (MoE) framework, followed by a clear discussion on how COMET’s design diverges from and improves upon existing methods.

3. While COMET is designed to improve modularity and interpretability, the authors do not demonstrate how the model’s interpretability has improved. More extensive interpretability metrics or qualitative evaluations would support the claimed benefits of COMET.

4. The absence of code and essential implementation details significantly hampers reproducibility and raises concerns about the robustness of the results.

questions:
1. Which components of COMET are inspired by the concept of biological random projection?

2. How should the hyperparameter $k$ in Equation (3) be determined?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580732181

thread_type:
official_review

<thread_object>
creator: reviewer_etwf

<content>
summary:
This paper introduces COMET (Conditionally Overlapping Mixture of ExperTs), a new method for creating sparse neural networks. The authors show using fixed, biologically-inspired routing can create more efficient and effective neural networks, particularly for larger models, while avoiding common problems in sparse architectures like representation collapse and poor knowledge transfer. The key insight is that COMET creates input-dependent sparsity without needing to learn the routing mechanism. COMET uses a fixed, biologically-inspired random projection combined with a k-winner-take-all operation to route inputs through the network, rather than using trainable gating functions.

strengths:
Unlike other methods, the proposed COMET method has no trainable gating functions (unlike standard Mixture of Experts) and avoids representation collapse.

Does not require explicit input/task IDs or pre-defined expert specialization.

Works across multiple architectures (MLPs, ViTs, GPT, MLP-Mixers).

The work is particularly similar to \cite{cheung2019superposition}, especially with the use of a random projection matrix V to handle the decision to mask. The justifications for using random projections in \cite{cheung2019superposition} seem to align well with the described capacity benefits of the COMET method in larger networks as compared to smaller networks. In particular, with a larger number of neurons, the probability of interference between masks rapidly decreases.

@article{cheung2019superposition,
  title={Superposition of many models into one},
  author={Cheung, Brian and Terekhov, Alexander and Chen, Yubei and Agrawal, Pulkit and Olshausen, Bruno},
  journal={Advances in neural information processing systems},
  volume={32},
  year={2019}
}

weaknesses:
The arguments for the number of experts is based on the possible permutations of masks that can be created which gives an unrealistically large number of possible experts. But this does not account for interference issues and establishing a bound more grounded in reality would be very helpful. The theory work in \cite{cheung2019superposition} should help better define these bounds.

There's a claim of "improved generalization through enhanced forward transfer", but it's unclear what experiments in this paper demonstrates better transfer learning.

questions:
Is there any reason to believe this phenomenon does not already occur in large networks? \cite{elhage2022toy} describe a situation where neural networks encode the phenomenon observed in \cite{cheung2019superposition} during the course of training. Are there advantages of explicitly creating the superposition?

Figure 4 shows a strange result where the smaller_model performs consistently as well as the standard_model even for fairly low p_k values. It's unclear to me why there would be a benefit for COMET if at the neurons=3000, the smaller_network at pk=.1 will perform as well as the standard_model.  What is being gained here?

@article{elhage2022toy,
  title={Toy models of superposition},
  author={Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and Hatfield-Dodds, Zac and Lasenby, Robert and Drain, Dawn and Chen, Carol and others},
  journal={arXiv preprint arXiv:2209.10652},
  year={2022}
}
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676986048

thread_type:
official_review

<thread_object>
creator: reviewer_fx4n

<content>
summary:
The paper introduces Conditionally Overlapping Mixture of ExperTs (COMET).
COMET uses biologically inspired, fixed random projections to generate binary masks that define subnetworks know as 'experts'.
The mask generation process is input-dependent, causing similar inputs to activate overlapping sets of experts.
The authors test the models on a range of benchmark tasks, finding that COMET performs well, particularly for large model sizes.

strengths:
The paper is well written and the core idea is explained clearly. 

The authors demonstrate key properties of the COMET model, notably showing that similar inputs tend to activate overlapping experts, facilitated by the fixed gating mechanism.

The model is tested on a wide selection of benchmark tasks including computer vision, language modelling, and regression. 

The authors demonstrate the benefit of using COMET, particularly at large model sizes.

The use of COMET requires no additional trainable parameters which is quite advantageous.

weaknesses:
In other works these gating functions can help alleviate catastrophic forgetting for tasks that are presented sequentially, but this is something that has not been tested in this paper. 

Given that previous work has similarly employed networks to determine gating, I am not entirely convinced that the novelty here is sufficient. However, I acknowledge that unlike prior approaches, which relied on trainable gates, this method uses fixed random projections.

There will be additional computational costs to using COMET but there is not an in-depth analysis of these costs in the paper. An analysis of training/inference times and GPU memory usage between COMET and the standard models would strengthen the submission.

questions:
How does this model perform on a continual learning benchmark, such as permuted MNIST or split-CIFAR-100? 

What are the additional costs to using COMET in terms of training time and memory usage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732161746434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal

comment:
We would like to thank the reviewers for their thoughtful feedback, and were pleased to see agreement on the following positive points:

**Importance & Novelty:**

COMET is a novel method for creating sparse neural networks. It achieves input-dependent sparsity using a fixed, biologically inspired routing mechanism in place of the trainable gates used by prior methods. In more detail: 

* fx4N: “unlike prior approaches, which relied on trainable gates, this method uses fixed random projections.”
* ETwf: “a new method for creating sparse neural networks (...) “COMET creates input-dependent sparsity without needing to learn the routing mechanism (...) Unlike other methods, the proposed COMET method has no trainable gating functions”
* rVeg: “COMET presents a novel routing method that replaces trainable gating functions with fixed, biologically inspired routing, which is rare in modular neural network approaches”.

**Key Properties of COMET:**

Similar inputs tend to activate overlapping experts, without the need for task IDs. In more detail:

* fx4N: “similar inputs tend to activate overlapping experts, facilitated by the fixed gating mechanism”.
* ETwf: “Does not require explicit input/task IDs or pre-defined expert specialization”
* rVeg:  “biologically inspired fixed routing approach that eliminates the need for task IDs and trainable gating functions

**Efficiency and Effectiveness:**

COMET shows improved performance relative to baseline approaches, especially for larger models, while avoiding representation collapse. In more detail:

* fx4N: “COMET requires no additional trainable parameters which is quite advantageous. (...) “The authors demonstrate the benefit of using COMET, particularly at large model sizes”
* ETwf: “The authors show using fixed, biologically-inspired routing can create more efficient and effective neural networks, particularly for larger models, while avoiding (...) representation collapse and poor knowledge transfer”
* rVeg: “eliminates the need for task IDs and trainable gating functions, commonly associated with representation collapse and redundancy”

**Versatility and generalization:**

COMET works well across diverse architectures and tasks. In more detail:

* fx4N: “The model is tested on a wide selection of benchmark tasks including computer vision, language modelling, and regression”.
* ETwf: “Works across multiple architectures (MLPs, ViTs, GPT, MLP-Mixers)”.
* rVeg: “The proposed method is tested across diverse architectures and tasks, such as image classification, language modeling, and regression”.

There were also concerns raised by more than one reviewer in two different areas: experimental evaluation and distinction from existing methods. We group these below, but given that each concern touches on a different point, we address these through individual reviewer responses.

**Experimental Evaluation:**

* fx4N: “these gating functions can help alleviate catastrophic forgetting for tasks that are presented sequentially, but this is something that has not been tested in this paper”.
* ETwf: “There's a claim of "improved generalization through enhanced forward transfer", but it's unclear what experiments in this paper demonstrates better transfer learning”.
* rVeg:  “While COMET is designed to improve modularity and interpretability, the authors do not demonstrate how the model’s interpretability has improved”.

**Distinction from Existing Methods:**

* fx4N: “Given that previous work has similarly employed networks to determine gating, I am not entirely convinced that the novelty here is sufficient”
* rVeg: “I recommend including a preliminary section that outlines the foundational Mixture of Experts (MoE) framework, followed by a clear discussion on how COMET’s design diverges from and improves upon existing methods.”
</content>
</thread_object>

</thread>
<thread>
creation date:
1732696595536

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Posted

comment:
We have just posted our revised paper. All new text is temporarily marked in orange. Please see the following major additions which respond to all the main points in the reviews. We hope you’ll agree the paper is now significantly strengthened.
* Role of biological inspiration (p. 2: 83-86)
    * Random projections, $k$-winner-take-all capping, and sparse representations with overlap that depends on input similarity
    * Correspondence between these mechanisms and elements of our formalism (p. 4: 202-203)
* Novelty and relation to previous work
    * Key ideas COMET brings together from several research areas (p. 2: 93-98)
    * Critical differences from other methods (p. 5: 265-269)
* Delineate present contributions from future work (p. 2: 98-102)
    * We clarify that transfer learning, continual learning, and catastrophic forgetting are promising directions but not studied here (except see sec A.9)
    * We have removed mention of interpretability which was never a goal
* Link to code repo (placeholder) (p. 2: 106)
* Relationship to model superposition of Cheung et al. (2019) (p. 3: 157-161)
* Preliminary section detailing MoE and input-dependent masking (p. 4: sec 3)
    * COMET and our proposal of overlapping experts synthesize these two literatures
* Considerations for number of experts (p. 4: 246-251)
    * Exponentially many experts are possible even with bounded interference (Elhage et al., 2022)
    * More importantly, ‘interference’ is desirable with our similarity-dependent routing
* Explained the significance of COMET beating the Standard Model when the Standard Model cannot beat the Smaller Model (p. 8: 424-426)
* Possibility that superposition emerges naturally in large networks (p. 10: 533-538)
    * If so, COMET still adds an advantage by explicitly imposing this structure
* Analysis of running times and memory usage (pp. 23-25, sec A.8)
* Preliminary test of transfer learning (pp. 26-27, sec A.9)
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper introduces Conditionally Overlapping Mixture of Experts (COMET), a novel method for enhancing sparse neural networks by using biologically inspired, fixed random projections to generate input-dependent binary masks. These masks define subnetworks called 'experts' whose activation overlaps based on input similarity, using a k-winner-take-all mechanism for routing inputs. This approach avoids the need for task IDs and trainable gating functions, which are common sources of issues like representation collapse and redundancy in sparse architectures. COMET is shown to perform well, especially in larger models, and improves efficiency, generalization, and speed of learning across various benchmarks and neural network architectures, including MLPs, Vision Transformers, and GPT-based models.



**Strengths**


* COMET introduces a novel routing method that replaces trainable gating functions with fixed, biologically inspired routing, a unique approach in modular neural network designs.
* The proposed method has been tested on diverse architectures and tasks, including image classification, language modeling, and regression, demonstrating its merits.
* COMET does not require explicit input/task IDs or pre-defined expert specialization, moreover, it does not require additional trainable parameters. 

**Weaknesses**

* In light of the prior work in the literature, the novelty of this paper is marginal.
* The argument for the number of experts, based on the potential permutations of masks creating an unrealistically large number, fails to consider interference issues.
* The paper lacks an analysis of the computational costs of the proposed algorithm.
* The paper suffers from overclaiming. For example, while the authors assert that COMET enhances modularity and interpretability, they fail to demonstrate any actual improvement in the model's interpretability.
* The lack of experiments in continual learning seems like a missed opportunity.

**Conclusion** 

The paper elicited polarized reviews. Reviewers fx4N and ETwf viewed it positively, though they noted its limited novelty. Reviewer rVeg assigned a low rating of 3 with a confidence of 4, but their review did not substantiate this negative assessment. The authors responded with a substantial rebuttal and engaged positively with the reviewers. After thorough consideration of the paper, the reviews, and the rebuttal, I find the work to be marginally above the acceptance threshold and therefore vote to accept the paper.