More Experts Than Galaxies: Conditionally-Overlapping Experts with Biologically-Inspired Fixed Routing

## Abstract

Abstract The evolution of biological neural systems has led to both modularity
and sparse coding, which
enables efficiency in energy usage, and robustness across the diversity of tasks in the lifespan. In contrast, standard neural networks rely on dense, non-specialized architectures, where all model parameters are simultaneously updated to learn multiple tasks, leading to representation interference. Current sparse neural network approaches aim to alleviate this issue, but are often hindered by limitations such as 1) trainable gating functions that cause representation collapse; 2) non-overlapping experts that result in redundant computation and slow learning; and 3) reliance on explicit input or task IDs that impose significant constraints on flexibility and scalability.
In this paper we propose Conditionally Overlapping Mixture of ExperTs (COMET), a general deep learning method that addresses these challenges by inducing a modular, sparse architecture with an exponential number of overlapping experts. COMET replaces the trainable gating function used in Sparse Mixture of Experts with a fixed, biologically inspired random projection applied to individual input representations. This design causes the degree of expert overlap to depend on input similarity, so that similar inputs tend to share more parameters. This facilitates positive knowledge transfer, resulting in faster learning and improved generalization.
We demonstrate the effectiveness of COMET on a range of tasks, including image classification, language modeling, and regression, using several popular deep learning architectures.

## 1 INTRODUCTION

In recent years, there has been a trend towards developing increasingly larger models (OpenAI, ; ; Fedus et al., ; Shuster et al., ; Chowdhery et al., ), driven by the understanding that a neural network’s learning capacity depends on its number of parameters (Shazeer et al., ). This approach has yielded impressive results in various fields, including computer vision (Dosovitskiy et al., ; Kirillov et al., ) and language modeling (OpenAI, ; Chowdhery et al., ). However, with such large size come difficulties, including increased training costs,
decreased interpretability,
and growing requirements for large amounts of memory and storage.

One approach to mitigating some of these challenges is sparsity, where a subset of the model’s parameters is selectively utilized in the computational graph. This concept of sparsity has been widely explored in machine learning (Jacobs et al., ; Jordan & Jacobs, ; LeCun et al., ; Zhou et al., ; Hoefler et al., ; Shazeer et al., ; Bair et al., ). Researchers have observed significant benefits of sparsity, including reduced inference costs (Han et al., ; Shazeer et al., ), improved generalization capabilities (LeCun et al., ; Jacobs & Burkholz, ; Frankle & Carbin, ; Paul et al., ), enhanced learning efficiency (LeCun et al., ), increased interpretability (Hossain et al., ), accelerated learning speed (LeCun et al., ; Mittal et al., ), less interference and forgetting (Raia, ), forward knowledge transfer (Andle et al., ; Raia, ; Yildirim et al., ), and compositionality
(Pfeiffer et al., ).

Early work on sparsity in neural networks focused on simple methods such as Dropout (Srivastava et al., ) and L1 regularization (Tibshirani, ; Ng, ). Recent research has explored more sophisticated approaches to selectively utilize neural network parameters (Shazeer et al., ; Bengio, ). These methods generally utilize a routing or gating function (Rosenbaum et al., ; ; Shazeer et al., ; Pfeiffer et al., ), which decides which parameters or sub-networks of the model to activate based on the input. The resulting input-dependent sub-networks have been referred to as “experts” in the context of Mixture of Experts (MoE) architectures (Jacobs et al., ; Jordan & Jacobs, ; Shazeer et al., ), which typically consist of multiple individual parameters grouped together (e.g., a whole MLP or a transformer). Many works also explore sparsity at a much finer level of granularity, including individual neurons (Xu et al., ), CNN filters (Chen et al., ), or even single parameters (Mallya et al., ; Mallya & Lazebnik, ).

However, existing sparse methods have limitations, which we would summarize as five key concerns: Firstly, most approaches rely on trainable gating functions (Shazeer et al., ; Mostafa & Wang, ; Rasmussen & Ghahramani, ; Li et al., ; Rahaman et al., ; Chen et al., ; Shazeer et al., ; Zhou et al., ; Gururangan et al., ; Lin et al., ; Ba & Frey, ; Bengio et al., ; Fedus et al., ; Mallya et al., ; Mallya & Lazebnik, ; Fernando et al., ; Keshari et al., ). This design choice is problematic for several reasons, including forgetfulness in continual learning (Pfeiffer et al., ; Raia, ), representation collapse (i.e., degenerate experts; Chen et al., ; Pfeiffer et al., ), complex training procedures (Rosenbaum et al., ), and other issues (Rosenbaum et al., ; Pfeiffer et al., ; Rosenbaum et al., ). Moreover, using non-trainable routing functions can be more effective (Mittal et al., ; Muqeeth et al., ). Secondly, many state-of-the-art systems employ architectures based on non-overlapping experts, where the experts do not share parameters (Shazeer et al., ; Pfeiffer et al., ). This design choice limits the transfer of previous knowledge to future tasks, also known as forward transfer, and can lead to redundancies; overlap can also be beneficial (French, ; Maini et al., ). Thirdly, even when experts overlap, it is unclear whether models can effectively learn to map similar inputs to the same experts, potentially resulting in redundancies (Chen et al., ) or interference (Pfeiffer et al., ). Furthermore, the routing mechanism often lacks interpretability, making it difficult to understand why specific inputs are routed to specific experts (Pfeiffer et al., ).
Fourthly, many existing methods require input or task IDs to determine which mask to apply (Mallya et al., ; Yang et al., ; Masse et al., ; Maini et al., ; Pes et al., ; Mittal et al., ; Muqeeth et al., ; Kang et al., ; Wortsman et al., ), which can be restrictive, as meta information about inputs is rarely available in real-world applications (Aljundi et al., ; Ye & Bors, ; Wang et al., ). Lastly, the number of experts in current systems is limited, often ranging from a few to a couple of thousand (Shazeer et al., ; Jiang et al., ), which may not be sufficient for complex tasks (Rasmussen & Ghahramani, ).

In this paper we introduce Conditionally Overlapping Mixture of ExperTs (COMET), a general deep learning method that induces a modular, sparse architecture in neural networks, with a number of important properties. First, COMET uses a non-trainable gating function, eliminating the need for iterative pruning or continuous sparsification. Instead, we employ a $k$-winner-take-all cap operation, inspired by the brain’s efficient use of a limited number of active cells via lateral inhibition. This design choice is both biologically motivated and automatically determined through fixed random projections. Because of it, COMET does not require fixed specialization of each network module, or advanced knowledge of the combination of modules required for each task, enabling more flexibility and adaptability. Second, the number of possible experts in COMET is exponential in the model size, exceeding the limit of a few thousand in recent work, to effectively tackle more complex tasks. Third, these experts overlap based on unsupervised information from input similarities. This, in turn, enables positive knowledge transfer between items, resulting in faster learning and improved generalization. It does this without increasing the number of trainable parameters, or requiring input or task IDs to determine which mask to apply.

We validate our approach through experiments on seven diverse tasks, including image classification, language modeling, and regression, demonstrating that our method is applicable to many popular model architectures such as vision transformers, MLP-mixers, GPTs, and standard MLPs, and consistently provides improved performance.

## 2 RELATED WORK

Sparsity
Sparsity in deep learning refers to systems where not all parameters are active or participating in the computational graph. Fixed sparsity, such as L1 regularization, intends to minimize the number of active parameters through optimizing a continuous loss function. Variable sparsity includes various methods, such as randomness as in Dropout (Srivastava et al., ), and input-dependent, such as in MoE (Shazeer et al., ). Notably, continuously optimizing deep networks to be sparse
is an NP-hard problem (Jacobs & Burkholz, ), and pre-defined sparse architectures, such as MoE, can be restrictive. To circumvent that, we propose sparsification using a biologically motivated approach of random projection followed by a cap operation, which activates the strongest cells in the network, similar to the sensory system of fruit flies (Bruhin & Davies, ).

Modularity
Modularity is related to sparsity, where information is conditionally routed to a subset of the network’s parameters (Pfeiffer et al., ; Rosenbaum et al., ). Modular methods can be split into two. Those with trainable routing functions, including the standard MoE (Shazeer et al., ), adaptively determine the active parameters during training. Although this approach is widely used, it can lead to representation collapse, forgetfulness, and redundant computation (Pfeiffer et al., ). Secondly, fixed routing function methods, where the decision on active parameters is fixed throughout training, have been shown to be more effective (Mittal et al., ; Muqeeth et al., ). However,
these methods
typically require prior knowledge of module specialization
(Pfeiffer et al., ; Muqeeth et al., ; Mittal et al., ), which is often not available in practice.
In contrast, our work demonstrates that module specialization can be achieved in a fully unsupervised manner using fixed random projections. Unlike previous studies that employed fixed random projections as a routing function, either with non-overlapping experts (Roller et al., ; Chen et al., ) or with reduced performance (Bruhin & Davies, ), our approach focuses on the more challenging setting of overlapping experts and achieves large performance gains across diverse tasks. Additionally, our expert overlap correlates with input similarity, making it more likely that parameters are shared between similar inputs. This, in turn, enables positive knowledge transfer between items, resulting in faster learning and improved generalization. Moreover, as the experts overlap, our approach benefits from input-dependent sparsity without an explicitly modular architecture (like in MoE).

Conditional Computation
Conditional computation integrates sparsity and modularity, in that parameters are dropped in a learned and optimized manner, rather than randomly and independently (Bengio, ). A notable example is the standard MoE framework (Shazeer et al., ). Our approach diverges from previous work by uniquely integrating several desirable properties, including fixed routing, overlapping expert assignments based on input similarities, mask determination without requiring meta information, and scalability to an exponentially large number of experts.

Dynamic Neural Networks
Dynamic neural networks (Han et al., )
share similarities with conditional computation. Unlike traditional MoE approaches, dynamic neural networks do not have explicitly defined experts. Instead, they dynamically select units, layers, or components from the main model for each input (Han et al., ). Our approach has parallels with some dynamic neural network methods, such as Piggyback (Mallya et al., ) and PackNet (Mallya & Lazebnik, ), which also do not define explicit experts. However, our method differs in two key ways. Firstly, it does not require meta information to determine the expert mask. Secondly, it ensures that any given input always maps to the same expert throughout both training and inference. This contrasts with architectures that use trainable gating functions, where the same input can activate different experts or representations at different stages of learning, which can be seen as a “moving target”.

## 3 CONDITIONALLY OVERLAPPING MIXTURE OF EXPERTS (COMET)

Figure: Figure 1: Illustration of a 2-layer MLP with embedded COMET layers. Note that COMET layers do not contain predefined experts, but instead dynamically selects a subset of the backbone MLP’s parameters to activate, effectively creating implicit experts. The sparsity level determines the proportion of parameters to activate. Real value in teal, zeros in white, ones in grey.
Refer to caption: x1.png

Our proposed COMET method applies to any backbone NN, augmenting it with a second NN called a routing network that computes input-dependent masks for all layers of the backbone network.

We first describe the COMET architecture for the case where the backbone network is an MLP.
Let the backbone MLP have $L$ layers, with layer $\ell$ having $N_{\ell}$ neurons and learnable parameters comprising a weight matrix ${\bm{W}}_{\ell}\in\mathbb{R}^{N_{\ell}\times N_{\ell-1}}$ and bias ${\bm{b}}_{\ell}\in\mathbb{R}^{N_{\ell}}$.
Then the forward pass of the unmodified MLP is defined by

$$ $\displaystyle{\bm{a}}_{\ell}$ $\displaystyle={\bm{W}}_{\ell}{\bm{x}}_{\ell-1}+{\bm{b}}_{\ell}$ (1) $\displaystyle{\bm{x}}_{\ell}$ $\displaystyle=f({\bm{a}}_{\ell})$ (2) $$

for $1\leq\ell\leq L$, where ${\bm{a}}_{\ell}$ is the pre-activation at layer $\ell$, $f$ is the elementwise activation function, ${\bm{x}}_{0}$ is the input to the network, and ${\bm{a}}_{L}$ is its output.

COMET’s routing network is a second MLP with the same shape, defined by random weight matrices ${\bm{V}}_{\ell}$ (for simplicity we omit bias parameters). We sample ${\bm{V}}_{\ell}$ from the same distribution used for initializing ${\bm{W}}_{\ell}$ ($U(-N_{\ell-1}^{-1/2},N_{\ell-1}^{-1/2})$ in our experiments).
We denote this network’s pre-activations and activations as ${\bm{c}}_{\ell}$ and ${\bm{z}}_{\ell}$ (analogous to ${\bm{a}}_{\ell}$ and ${\bm{x}}_{\ell}$ in the backbone network), with input ${\bm{z}}_{0}={\bm{x}}_{0}$.
The computation of the routing network is similar to that of the backbone MLP, except that at each layer it computes a binary vector ${\bm{m}}_{\ell}$ that is then used to mask the activations in both networks.
The mask is computed using a capping function $C_{k}$ that identifies the top $k$ elements of a vector:

$$ $\displaystyle[C_{k}({\bm{v}})]_{i}=\left\{\begin{array}[]{ll}1&|\{j:v_{j}\geq v _{i}\}|\leq k\\ 0&\text{otherwise}\end{array}\right.$ (5) $$

We allow a fixed proportion $p_{\rm k}$ of neurons at each layer to survive the mask, so that $k_{\ell}=p_{\rm k}N_{\ell}$.
Then the forward pass of the routing network is defined by

$$ $\displaystyle{\bm{c}}_{\ell}$ $\displaystyle={\bm{V}}_{\ell}{\bm{z}}_{\ell-1}$ (6) $\displaystyle{\bm{m}}_{\ell}$ $\displaystyle=C_{k_{\ell}}({\bm{c}}_{\ell})$ (7) $\displaystyle{\bm{z}}_{\ell}$ $\displaystyle={\bm{m}}_{\ell}\circ g({\bm{c}}_{\ell})$ (8) $$

where $\circ$ indicates elementwise multiplication and $g$ is the routing network’s activation function (we use the identity $g(c)=c$ in the present experiments).
The layerwise masks ${\bm{m}}_{\ell}$ computed by the routing network are then applied to the backbone network, so that eqs. 1 and 2 are replaced by

$$ $\displaystyle{\bm{a}}_{\ell}$ $\displaystyle={\bm{W}}_{\ell}{\bm{x}}_{\ell-1}$ (9) $\displaystyle{\bm{x}}_{\ell}$ $\displaystyle={\bm{m}}_{\ell}\circ f({\bm{a}}_{\ell})$ (10) $$

Note that the network’s output ${\bm{a}}_{L}$ is computed before ${\bm{m}}_{L}$ would be applied, avoiding undesirable masking of the model’s prediction.

COMET layers differ from standard MoE (Shazeer et al., )
layers
in two major ways:

- 1.
Architecturally:
Whereas a layer in a standard layered MoE architecture consists of $e$ experts and a gating network, a COMET layer contains a random, non-trainable matrix and a $k$-winner-take-all cap operation. Note that a COMET layer does not contain pre-defined experts. Instead, it dynamically modifies the computation of the MLP to activate only a subset of its parameters; this subset can be seen as an implicit expert. This input-dependent masking at the neuron level of each layer results in a maximum number of experts that is exponential in the model size, specifically ${N\choose k}$. For instance, a small layer with 1000 neurons and
a moderate sparsity level of $10\%$ yields an upper bound of ${1000\choose 100}\approx 6.38e^{139}$ experts. Therefore in practical settings every input will have its own expert.
- 2.
In the way the information is passed: Whereas in MoE the input to the gating network is the same as the input to each of the MLP layers, in a COMET architecture
the inputs are distinct (except for the first layer of the network). That is, the routing network operates independently of the backbone network. This ensures that the same inputs always map to the same implicit expert throughout both training and inference.

## 4 EXPERIMENTS

### 4.1 SYNTHETIC DATA EXPERIMENTS

In this section, we describe experiments to verify key properties of a COMET network. First, we verify that the combination of the fixed routing function and cap operator maps similar inputs to similar masks and show how this sharpens the model’s generalization. Second, we verify that the network makes an effective use of the available neurons.

#### 4.1.1 FIXED INPUT-DEPENDENT ROUTING NETWORK

Figure: (a)
Refer to caption: x2.png

Our goal is to develop a fixed routing function
that maps similar inputs to similar (i.e., overlapping) experts, thereby facilitating knowledge transfer between items and leading to faster learning and improved generalization.
One way to approximate generalization between individual training and test items is with the neural tangent kernel (NTK; Jacot et al., ).
Let ${\bm{\theta}}=({\bm{W}}_{1},{\bm{b}}_{1},\dots,{\bm{W}}_{L},{\bm{b}}_{L})$ denote the flattened concatenation of all model parameters,
let ${\bm{x}}^{\rm train}$ and ${\bm{x}}^{\rm test}$ be arbitrary training and testing items, and let ${\bm{a}}_{L}^{\rm train}$ and ${\bm{a}}_{L}^{\rm test}$ be the corresponding model predictions for some fixed setting of ${\bm{\theta}}$.
Generalization from ${\bm{x}}^{\rm train}$ to ${\bm{x}}^{\rm test}$ can be defined as the change in prediction ${\bm{a}}_{L}^{\rm test}$ from including ${\bm{x}}^{\rm train}$ in the training set.
Formally, under a vanilla GD optimizer on loss $\mathcal{L}$, and in the limit of a small learning rate $\alpha$, the contribution of ${\bm{x}}^{\rm train}$ to change in ${\bm{a}}_{L}^{\rm test}$ is

$$ $\displaystyle\frac{1}{\alpha}\Delta{\bm{a}}_{L}^{\rm test}\underset{\alpha\to 0 }{\longrightarrow}K({\bm{x}}^{\rm train},{\bm{x}}^{\rm test})\nabla_{{\bm{a}}_ {L}^{\rm train}}\mathcal{L}^{\rm train}$ (11) $$

where $K({\bm{x}}^{\rm train},{\bm{x}}^{\rm test})$ is the $N_{L}\times N_{L}$ matrix-valued NTK

$$ $\displaystyle K({\bm{x}}^{\rm train},{\bm{x}}^{\rm test})=\frac{\partial{\bm{a }}_{L}^{\rm test}}{\partial{\bm{\theta}}}\left(\frac{\partial{\bm{a}}_{L}^{\rm train }}{\partial{\bm{\theta}}}\right)^{\top}$ (12) $$

The RHS of eq. 12 sums over elements of ${\bm{\theta}}$, and the contribution from ${\bm{W}}_{\ell}$ is

$$ $\displaystyle\sum_{ij}\frac{\partial{\bm{a}}_{L}^{\rm test}}{\partial W_{\ell, ij}}\left(\frac{\partial{\bm{a}}_{L}^{\rm train}}{\partial W_{\ell,ij}}\right) ^{\top}=\sum_{j}a_{\ell-1,j}^{\rm test}a_{\ell-1,j}^{\rm train}\sum_{i}\frac{ \partial{\bm{a}}_{L}^{\rm test}}{\partial a_{\ell,i}^{\rm test}}\left(\frac{ \partial{\bm{a}}_{L}^{\rm train}}{\partial a_{\ell,i}^{\rm train}}\right)^{\top}$ (13) $$

Thus the contribution of ${\bm{W}}_{\ell}$ to generalization from ${\bm{x}}^{\rm train}$ to ${\bm{x}}^{\rm test}$ is proportional to the inner product $\langle{\bm{a}}^{\rm train}_{\ell-1},{\bm{a}}^{\rm test}_{\ell-1}\rangle$.
This inner product will be positively related to input similarity $\langle{\bm{x}}^{\rm train},{\bm{x}}^{\rm test}\rangle$ even in an unmodified MLP, but our question is how the relationship changes under COMET.

To answer this question, we conducted an experiment using 500 random pairs of input vectors. Each input had length 100 with components sampled iid from $\mathcal{N}(\mu,25)$, with $\mu$ a random integer between 0 and 100. We calculated the cosine similarity (i.e., normalized inner product) between each pair of inputs. We then randomly initialized 10 COMET networks for every pair, each comprising a backbone network and a routing network which were both MLPs containing 10 hidden layers ($L=11$) with 512 neurons per hidden layer. We passed both inputs through each of the 10 COMET networks, using eqs. 6, 7, 8, 9 and 10 with varying degrees of sparsity $p_{\rm k}$.

To analyze the behavior of our fixed gating function, we performed two complementary analyses. First, we computed the cosine similarity between the masks obtained for the two inputs in each pair, concatenated across layers as $({\bm{m}}_{1},\dots,{\bm{m}}_{L-1})$. Note that cosine similarity between binary vectors equals their degree of overlap, i.e. the proportion of active neurons for one input that are also active for the other. Second, we measured the cosine similarity between the two inputs’ representations in the backbone network after applying the gating function as in eq. 10, again concatenating across layers as $({\bm{x}}_{1},\dots,{\bm{x}}_{L-1})$. To obtain a more robust estimate, we averaged the cosine similarities across the 10 COMET networks for each input pair, yielding the results in fig. 2.

This experiment reveals that when input distributions are more similar
the overlap between their binary masks increases (fig. 2a). This in turn strengthens the relationship between input similarity and activation similarity in the backbone network relative to the baseline MLP with $p_{\rm k}=1$ (fig. 2b). Drawing on the NTK analysis above, we conclude that COMET’s routing function leads the model to generalize using a narrower effective kernel. A narrower kernel should not be expected to yield universal improvement, but it should be beneficial when the base model has excess capacity for the task. The experiments in the next subsections support this prediction, in that we see an advantage for COMET particularly with larger models.

#### 4.1.2 EXPERT UTILIZATION

Figure: Figure 3: Illustration of neuron activity across COMET layers in a 4-layer MLP. We visualize the utilization of neurons in two randomly initialized networks with varying sparsity levels, using the CIFAR10 dataset. The plots show that our network effectively utilizes all its parameters, with no “dead neurons” and no signs of representation collapse, even at very high sparsity levels.
Refer to caption: x4.png

One of the challenges in training sparse architectures is representation collapse, where a small subset of experts or neurons becomes dominant, leading to under-utilization of others. This issue is particularly concerning when training gating networks, as it can result in “dead” experts or neurons that are never activated. Given that our gating network is fixed, it is essential to investigate whether this behavior occurs, as it would be permanent.

To address this, we conducted an experiment where we generated 1000 randomly initialized 4-layer COMET MLPs with varying neuron counts per layer $N_{\ell}\sim U(100,1000)$, and varying sparsity level $1-p_{\rm k}\sim U(0.05,1)$, and passed the CIFAR10 dataset through each one. We then analyzed the utilization of each neuron, measured by how often it is activated with the given masks, and found that only $7.6e^{-4}\%$ of neurons across the ensemble had 0% utilization, and fewer than $2\%$ of the models had any such neurons, mostly models with a high sparsity level. Figure 3 presents utilization plots for two representative networks, illustrating that our fixed gating network avoids representation collapse and “dead neurons.” Moreover, when we passed a different dataset (such as CIFAR100) through these models, we discovered that many previously inactive neurons were now being utilized. Thus, neurons that are infrequently utilized appear to be reserved for unseen data, highlighting the network’s adaptability and capacity for generalization.

The finding is consistent with our previous analysis in Section 4.1.1, which showed that the input-dependent gating design inherently activates similar parameters for similar inputs, facilitating forward knowledge transfer.
Our results suggest that our approach effectively mitigates the risk of representation collapse and promotes healthy utilization of neurons in the network, all without relying on supplementary mechanisms, such as specialized loss terms (Shazeer et al., ).

### 4.2 IMAGE CLASSIFICATION

We extend our investigation by integrating the COMET method into a diverse range of popular architectures, including Vision Transformers (ViTs), MLP-Mixers, and standard MLPs.

#### 4.2.1 STANDARD MLP - CIFAR10

We apply the COMET method to a standard MLP with 4 layers, varying the number of neurons in each layer and the sparsity levels.
To evaluate its performance, we compare it to 10 related methods:

Standard Model: A standard MLP model with the same number of neurons and no sparsity.

Smaller Model: A smaller model with a reduced number of neurons, specifically $p_{\rm k}N_{\ell}$ where $N_{\ell}$ is the width of the standard model.

Dropout Model: A standard model with a dropout rate equal to $1-p_{\rm k}$.

Topk Model: An MLP with a trainable routing function. The cap operation is applied directly to the backbone network by replacing eq. 7 with ${\bm{m}}_{\ell}=C_{k_{\ell}}({\bm{a}}_{\ell})$, so that the routing function selects the highest $k$ values and masks the remaining ones.

MoE Trainable: A MoE model with a trainable routing function, where the number of experts is equal to $\lfloor 1/p_{\rm k}\rfloor$, and each expert has $p_{\rm k}N_{\ell}$ neurons in each layer. The routing network is a trainable MLP with one hidden layer, whose output is a sparse $\lfloor 1/p_{\rm k}\rfloor$-dimensional vector.

MoE Non-trainable: Same as MoE Trainable, with a fixed routing function.

Layer-wise Routing: An MLP where each backbone hidden layer representation is projected using a fixed random matrix, which is then used to develop the binary mask for the next layer. This is done by replacing eq. 6 with ${\bm{c}}_{\ell}={\bm{V}}_{\ell}{\bm{x}}_{\ell-1}$.

Bernoulli Masking: An MLP where each example in the training data is associated with a fixed binary mask drawn from a Bernoulli distribution, with probability equal to $p_{\rm k}$. Thus the relationship between inputs and their masks is arbitrary, rather than being mediated by the routing network in COMET.

Example-tied Dropout: Example-tied dropout (Maini et al., ), where each example in the training data is associated with a fixed binary mask drawn from a Bernoulli distribution, with probability equal to $p_{\rm k}$, and a fixed number of “generalization neurons” are active for all examples (reduces to Bernoulli Masking when there are zero generalization neurons).

Standard model L1: A standard MLP model, but using L1 regularization to induce sparsity.

We evaluate these models on the CIFAR10 dataset Krizhevsky (), with results shown in Figure 4.
Overall, the optimal model architecture depends on the capacity of the network. When the number of neurons is limited and the network has a low capacity to learn the task (i.e., low $p_{\rm k}$), the standard model that utilizes all neurons outperforms most models. However, as network capacity increases with more neurons, the COMET model emerges as the top performer. This suggests that the benefits of selective neuron activation become more pronounced as capacity increases.

Figure: Figure 4: Illustration of 4-layer MLP networks trained on CIFAR10, showcasing the impact of varying network capacity and sparsity levels. As we systematically increase the number of neurons and decrease sparsity (moving from top left to bottom right), we observe a shift in the best-performing model. Initially, the standard model outperforms the COMET model when network capacity is low. However, as network capacity grows, the COMET model emerges as the top performer.
Refer to caption: x5.png

#### 4.2.2 CONTEMPORARY ARCHITECTURES

We further extend the COMET method to contemporary architectures in the Vision domain, including ViT Dosovitskiy et al. () and MLP-Mixer Tolstikhin et al. (). To do this, we apply the COMET random projection followed by the cap operation in the MLP layers of each with $p_{\rm k}=0.5$. We evaluate the performance of these models on four widely-used image classification datasets: SVHN Netzer et al. (), CIFAR10 Krizhevsky (), CIFAR100 Krizhevsky (), and Tiny ImageNet Le & Yang ().
Our results can be seen in Figures 6, 6, 8, 8.

A similar trend emerges in these architectures: as network capacity increases, the optimal model architecture shifts. In smaller networks, where the number of neurons in the MLP layer is limited, the standard model performs roughly similarly to the COMET model. However, even in these networks, incorporating COMET layers yields notable performance improvements. As we scale up the network by adding more neurons, COMET displays superior performance across all five model architectures and four datasets. It achieves faster convergence and significantly higher accuracy, with gains of up to $9\%$ in ViT Large on CIFAR100. Moreover, we observe that the performance gap between the COMET-based models and their standard counterparts widens as the model size increases, with larger models exhibiting both better performance and faster learning rates. This reinforces our finding that selective neuron activation becomes increasingly beneficial as network capacity grows.

Figure: Figure 5: ViTs and MLP-Mixers on SVHN. [Highest accuracy] (# trainable param.)
Refer to caption: x6.png

Figure: Figure 7: ViTs and MLP-Mixers on CIFAR100. [Highest accuracy] (# trainable param.)
Refer to caption: x8.png

Figure: Figure 9: GPTs trained on WikiText. [Lowest perplexity] (# trainable param.)
Refer to caption: x10.png

### 4.3 LANGUAGE MODELING

We apply COMET to language modeling on Wikitext Merity et al. () and CodeParrot Tunstall et al. () with varying GPT model sizes, with results in Figures 10 and 10.
We again observe the same pattern: as network capacity increases, the COMET model outperforms the standard model, with larger models exhibiting not only a more significant performance difference but also faster learning rates, highlighting the benefits of selective neuron activation in language modeling tasks.

### 4.4 REGRESSION

To further validate our results, we also evaluated COMET on the SARCOS regression dataset. Our findings in Appendix A.5 show that our conclusions generalize to this setting as well.

## 5 CONCLUSIONS

In this work, we propose a sparse neural network method, COMET, that induces a modular, sparse architecture with an exponential number of overlapping experts and alleviates key limitations of existing modular approaches. Specifically, we tackle the issues of trainable gating functions that often lead to representation collapse, non-overlapping experts that hinder knowledge transfer, and the need for explicit input IDs that can be restrictive. By leveraging a fixed, biologically-inspired random projection, COMET determines expert overlap based on input similarity in a fully unsupervised manner, enabling faster learning and improved generalization through enhanced forward transfer. Through extensive experiments on various tasks, including image classification, language modeling, and regression, we demonstrate that COMET achieves improved performance, especially for larger models, demonstrating that our method is applicable to many popular model architectures.

## Appendix A EXPERIMENT DETAILS

In this section, we provide a detailed description of the experiments conducted to evaluate the performance of COMET. The following subsections outline the experimental setup, including the datasets used, model architectures, and hyperparameters. We also present additional experimental results and analyses.

### A.1 MAIN RESULTS: STANDARD MLP - CIFAR10

In this subsection, we present the main results of our experiments on the CIFAR10 dataset using a standard 4-layer MLP architecture. We first describe the model configuration and training setup, followed by a description of the hyperparameter settings used to vary model capacity and sparsity levels.

##### Model Configuration and Training

We employ a standard 4-layer MLP architecture, utilizing the SGD optimizer with a learning rate of 1e-4. To ensure robustness, we train each model over 3 random seeds for 100 epochs. We systematically explore the effects of varying model capacity and sparsity levels by modifying the number of neurons in each layer and the sparsity ratio.

##### Model Capacity Variations

We consider four different model capacities by setting the number of neurons in each layer to 100, 500, 1000, 3000.

##### Sparsity Level Variations

For each model capacity, we investigate the impact of three different sparsity levels: 0.1, 0.5, and 0.9.

### A.2 ADDITIONAL RESULTS: STANDARD MLP - CIFAR10

This subsection presents additional experimental results on the CIFAR10 dataset using a standard 4-layer MLP architecture, but with a higher learning rate of 1e-3. We describe the model configuration and hyperparameter settings used, and report the results of varying model capacity and sparsity levels.

##### Model Configuration and Training

To further assess the robustness of the COMET method, we also conduct experiments using a higher learning rate of 1e-3. In these experiments, we again employ a standard 4-layer MLP, SGD optimizer, and systematically vary model capacity and sparsity levels by adjusting the number of neurons in each layer and the sparsity ratio, respectively.

##### Model Capacity Variations

We consider an additional model capacity to a total of five different model capacities by setting the number of neurons in each layer to 100, 500, 1000, 3000, or 9000.

##### Sparsity Level Variations

For each model capacity, we investigate the impact of three different sparsity levels: 0.1, 0.5, and 0.9.

As illustrated in Figure 11, a consistent trend emerges as we systematically vary the number of neurons and sparsity levels. Moving from top left to bottom right, we observe a shift in the optimal model configuration. Initially, when network capacity is limited, the standard model outperforms the COMET model. However, as network capacity increases, the COMET becomes the top performer, surpassing every other model. This trend reinforces our key finding: selective neuron activation becomes increasingly beneficial as network capacity increases, enabling faster learning and improved generalization through enhanced forward transfer.

Figure: Figure 11: Illustration of 4-layer MLP networks trained on CIFAR10 with a higher learning rate (1e-3), showcasing the impact of varying network capacity and sparsity levels. As we systematically increase the number of neurons and decrease sparsity (moving from top left to bottom right), we observe a shift in the best-performing model. Initially, the standard model (without sparsity, in blue) outperforms the COMET when network capacity is low. However, as network capacity grows, the COMET model emerges as the top performer.
Refer to caption: x12.png

### A.3 CONTEMPORARY ARCHITECTURES

This subsection presents our experimental results on contemporary architectures, including ViT and MLP-Mixer models. We describe the hyperparameter settings used for each architecture, and detail the modifications made to analyze the effect of our sparsity method, COMET.

##### Hyperparameter Settings

We built upon the tuned hyperparameters from Yoshioka () and made the following modifications to analyze the effect of our sparsity method, COMET. Specifically, we systematically increased model capacity by adding more neurons to the MLP layers of each model. Additionally, we used the $\tanh$ activation function on the backbone MLP layers.
See Appendix A.6 for further analysis on activation functions.

##### Vision Transformer Hyperparameters

The standard ViT uses a set of hyperparameters, which we modified as follows. The number of classes is set to 10 for CIFAR10 and SVHN, 100 for CIFAR100, and 200 for Tiny Imagenet. The model depth is 6, with 8 attention heads. We increased the MLP dimension from 512 to 3072 for the ViT medium model and to 6144 for the ViT large model. The dropout rate is 0.1, and the patch size is 4. The embedding dropout rate is also 0.1. We trained the models for 200 epochs with a learning rate of 1e-4.

##### MLP-Mixer Hyperparameters

The MLP-Mixer model uses a different set of hyperparameters. The patch size is 4. We increased the dimension from 512 to 3072 for the MLP-Mixer medium model. The model depth is 6, and the number of classes is set to 10 for CIFAR10 and SVHN, 100 for CIFAR100, and 200 for Tiny Imagenet. We trained the models for 500 epochs with a learning rate of 1e-3.

##### Training Setup

All models were trained using the Adam optimizer with a cosine learning rate schedule on a single A100 GPU. To ensure robustness, we train each model over 3 random seeds.

### A.4 LANGUAGE MODELING

We extend our evaluation of COMET to the task of language modeling, examining its performance on various GPT model variants.

#### A.4.1 Main Results

This subsection presents our main results on the language modeling task, detailing the performance of COMET on three different GPT model variants. We describe the hyperparameter settings and training setup used for each model, and report the results of our experiments.

##### GPT Model Variants

We train three variants of the GPT model, each with a different set of parameters. The standard GPT-2 model has 12 layers, 768 hidden units, 12 attention heads, and 117M parameters. The GPT-2-Medium model has 24 layers, 1024 hidden units, 16 attention heads, and 345M parameters. The GPT-2-Large model has 36 layers, 1280 hidden units, 20 attention heads, and 774M parameters.

##### Hyperparameter Settings

We built upon the tuned hyperparameters from . Our optimizer of choice was AdamW, with a learning rate of 5e-4, weight decay of 0.1, and 1,000 warmup steps. We also used gradient accumulation with 8 steps, which resulted in an effective batch size of 256, calculated by multiplying the per-device train batch size (32) by the gradient accumulation steps (8). We used tanh activation function on the backbone MLP layers and a cosine learning rate schedule with warmup. We also enabled mixed precision training to accelerate computations.

##### Training Settings

Each model was trained from scratch on a single A100 GPU. Due to computational constraints and time limitations, we restricted training to either 3 epochs or a maximum of 24 hours. To ensure robustness, we train each model over 3 random seeds.

#### A.4.2 Additional Results

We conducted additional experiments to investigate how the choice of hyperparameters influences the performance of our method, COMET, when applied to the MLP layers of GPT models. This analysis aims to provide a deeper understanding of the robustness and adaptability of COMET under various hyperparameter settings.

The following Figures mark COMET models with spec_true, due to the fact that COMET’s input-dependent gating mechanism leads to the formation of experts that are selectively specialized for specific inputs. The standard models are marked with spec_false.

##### Tokenizer Effect

We note that, in general, the GPT models we evaluated tend to learn faster when using the tokenizer provided by . However, due to its widespread adoption, we opt to use the standard GPT-2 tokenizer for the remainder of our experiments:

To validate our main findings, we first assess the performance of the different GPT-2 model sizes on WikiText and CodeParrot using the standard GPT-2 tokenizer and $50\%$ sparsity level. Our results are presented in Figures 12, 13, 14, 15, 16, and 17.

Consistent with our main results, we find that the COMET-based model learns faster, even with a smaller model size, when using the standard GPT-2 tokenizer. This suggests that the observed pattern is robust and not specific to the tokenizer used.

Figure: Figure 12: Validation perplexity of GPT-2 on Wikitext dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_wikitext_gpt2-tokenizer.png

Figure: Figure 13: Validation perplexity of GPT-2 Medium on Wikitext dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_med_wikitext_gpt2-tokenizer.png

Figure: Figure 14: Validation perplexity of GPT-2 Large on Wikitext dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_large_wikitext_gpt2-tokenizer.png

Figure: Figure 15: Validation perplexity of GPT-2 on CodeParrot dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_codeparrot_gpt2-tokenizer.png

Figure: Figure 16: Validation perplexity of GPT-2 Medium on CodeParrot dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_med_codeparrot_gpt2-tokenizer.png

Figure: Figure 17: Validation perplexity of GPT-2 Large on CodeParrot dataset using the standard GPT-2 tokenizer.
Refer to caption: extracted/5938409/images/gpt2_large_codeparrot_gpt2-tokenizer.png

##### Learning Rate Effect

To investigate the impact of learning rate on the COMET method, we scale the learning rate by a factor of 10, from 1e-4 to 1e-3. The results of this experiment are presented in Figures 18, 19, 20, 21, 22, and 23.

Our findings show that, across all experiments, the COMET models consistently learn faster than their standard counterparts. However, at the $50\%$ sparsity level, we observe that the smaller models often perform slightly worse than the standard models. This result is consistent with our previous findings, which suggest that adding sparsity to models with limited capacity can negatively impact performance.

Furthermore, we identify an interesting trend as the model size increases, as seen in Figures 19, 20, 22, and 23. Specifically, when using a larger learning rate and larger model sizes, the standard models tend to overfit or experience exploding gradients, resulting in a significant increase in validation perplexity. In contrast, adding sparsity using COMET not only enables faster learning but also mitigates overfitting and gradient explosion, allowing for stable training with a larger learning rate.

Figure: Figure 18: Validation perplexity of GPT-2 on Wikitext dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_wikitext_gpt2-tokenizer_lr_1e-3.png

Figure: Figure 19: Validation perplexity of GPT-2 Medium on Wikitext dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_med_wikitext_gpt2-tokenizer_lr_1e-3.png

Figure: Figure 20: Validation perplexity of GPT-2 Large on Wikitext dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_large_wikitext_gpt2-tokenizer_lr_1e-3.png

Figure: Figure 21: Validation perplexity of GPT-2 on CodeParrot dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_codeparrot_gpt2-tokenizer_lr_1e-3.png

Figure: Figure 22: Validation perplexity of GPT-2 Medium on CodeParrot dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_med_codeparrot_gpt2-tokenizer_lr_1e-3.png

Figure: Figure 23: Validation perplexity of GPT-2 Large on CodeParrot dataset using the standard GPT-2 tokenizer and a learning rate of 1e-3.
Refer to caption: extracted/5938409/images/gpt2_large_codeparrot_gpt2-tokenizer_lr_1e-3.png

##### Batch Size Effect

To further investigate the robustness of the COMET method, we examine the effect of reducing the batch size by a factor of 4, achieved by decreasing the gradient accumulation step from 8 to 2. Our results are presented in Figures 24, 25, 26, 27, 28, and 29.

Our findings demonstrate that COMET remains effective even with a reduced batch size, making it a viable option for users with limited computational resources. We observe that on the Wikitext dataset, the smaller GPT model with COMET learns faster and achieves comparable performance to the standard model at the end of training. In contrast, on the CodeParrot dataset, the smaller model with COMET outperforms the standard model. Moreover, we note that COMET’s benefits extend to smaller batch sizes, where standard models may struggle with overfitting or exploding gradients. By incorporating sparsity, COMET enables more stable training and better performance, even in resource-constrained environments.

Figure: Figure 24: Validation perplexity of GPT-2 on Wikitext dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_wikitext_gpt2-tokenizer_grad_acc_2.png

Figure: Figure 25: Validation perplexity of GPT-2 Medium on Wikitext dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_med_wikitext_gpt2-tokenizer_grad_acc_2.png

Figure: Figure 26: Validation perplexity of GPT-2 Large on Wikitext dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_large_wikitext_gpt2-tokenizer_grad_acc_2.png

Figure: Figure 27: Validation perplexity of GPT-2 on CodeParrot dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_codeparrot_gpt2-tokenizer_grad_acc_2.png

Figure: Figure 28: Validation perplexity of GPT-2 Medium on CodeParrot dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_med_codeparrot_gpt2-tokenizer_grad_acc_2.png

Figure: Figure 29: Validation perplexity of GPT-2 Large on CodeParrot dataset using the standard GPT-2 tokenizer and a reduced batch size of 64 ($\frac{1}{4}$ of original).
Refer to caption: extracted/5938409/images/gpt2_large_codeparrot_gpt2-tokenizer_grad_acc_2.png

##### Mixed Precision Effect

We further investigate the robustness of the COMET method by evaluating its performance under mixed precision training. Specifically, we assess the impact of switching from FP16 to FP32 precision on the COMET-based models. Our results are presented in Figures 30, 32, and 31.

In summary, our experiments demonstrate that the COMET method consistently enables faster learning across various model sizes, datasets, and training settings. By incorporating COMET, we observe improved performance and robustness, even when switching from FP16 to FP32 precision. This is particularly valuable, as FP32 precision is often preferred in certain applications where numerical stability is crucial. Moreover, having models that can effectively learn in both FP16 and FP32 precision regimes provides greater flexibility and adaptability, allowing for more efficient deployment on a wide range of hardware platforms.

Figure: Figure 30: Validation perplexity of GPT-2 on Wikitext dataset using the standard GPT-2 tokenizer and FP32 precision.
Refer to caption: extracted/5938409/images/gpt2_wikitext_gpt2-fp32.png

Figure: Figure 31: Validation perplexity of GPT-2 Medium on Wikitext dataset using the standard GPT-2 tokenizer and FP32 precision.
Refer to caption: extracted/5938409/images/gpt2_med_wikitext_gpt2-fp32.png

Figure: Figure 32: Validation perplexity of GPT-2 on CodeParrot dataset using the standard GPT-2 tokenizer and FP32 precision.
Refer to caption: extracted/5938409/images/gpt2_codeparrot_gpt2-fp32.png

### A.5 REGRESSION

This subsection presents our evaluation of COMET on a regression task using the SARCOS dataset. We describe the experimental setup, including the dataset, model architecture, and hyperparameter settings, and report the results of our experiments.

##### Dataset

To conclude our evaluation, we apply the COMET method to a regression task using the SARCOS dataset. This dataset is derived from an inverse dynamics problem involving a 7-joint anthropomorphic robot arm, where the goal is to predict the 7 joint torques based on a 21-dimensional input space consisting of joint positions, velocities, and accelerations. We focus on a single output dimension, following the approach of Rasmussen & Williams (). The dataset is publicly available at [https://gaussianprocess.org/gpml/data/](https://gaussianprocess.org/gpml/data/). Building on the work of Jones et al. (), we employ a 4-layer MLP network, but experiment with varying the number of neurons in each layer and the level of sparsity.

Figure: Figure 33: Illustration of 4-layers MLP trained on SARCOS.
Refer to caption: x13.png

We use SGD optimizer with a learning rate of 1e-2. To ensure robustness, we train each model over 3 random seeds for 50 epochs. We systematically explore the effects of varying model capacity and sparsity levels by modifying the number of neurons in each layer and the sparsity ratio.

##### Model Capacity Variations

We consider five different model capacities by setting the number of neurons in each layer to 100, 500, 1000.

##### Sparsity Level Variations

For each model capacity, we investigate the impact of three different sparsity levels: 0.1, 0.5, and 0.9.

##### Results

The results, presented in Figure 33, demonstrate the effectiveness of our approach in this domain. Consistent with our previous findings, we again observe that the COMET model outperforms the standard model as network capacity increases, confirming the benefits of selective neuron activation in regression tasks.

### A.6 ACTIVATIONS ANALYSIS

In this section, we analyze the impact of different activation functions on the performance of COMET-based models. We present an experimental evaluation of various activation functions on a 4-layer MLP network trained on the CIFAR10 dataset, and discuss the results.

##### Evaluating the Impact of Activation Functions on COMET

We now investigate the impact of utilizing various activation functions within the backbone network on the performance of a COMET-based model during training.

##### Experimental Setup

We evaluate the effect of different activation functions on a 4-layer MLP network trained on the CIFAR10 dataset with the following settings: SGD + Momentum optimizer, a learning rate of 1e-3, a sparsity level of 0.5, and 3000 neurons in each layer.

##### Results

Our results are presented in Figure 34. We find that the COMET model outperforms the standard model when most activation functions are applied on the backbone network.
However, we observe that COMET does not perform as well as the standard model for certain non-monotonic activation functions, specifically GELU, Mish, and SiLU, under the settings we chose (sparsity level = 0.5 and 3000 neurons in each layer).
Due to time constraints, we do not delve deeper into the reasons behind this phenomenon but suggest that future work could investigate why non-monotonic functions may hinder the effectiveness of COMET.

Figure: Figure 34: Comparison of the performance of COMET-trained MLP networks on the CIFAR10 dataset using different activation functions.
Refer to caption: x14.png

### A.7 COMET: Routing Network Architecture

We have implemented COMET using a routing network having the same architecture as the backbone MLP network, but this is not necessary. COMET only requires the routing network to generate a mask at each layer having the same shape as (or a shape that can be broadcast to) the shape of the corresponding backbone layer. Future work will investigate alternative routing network architectures to determine if they can improve the performance and efficiency of COMET-based models.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: How does this model perform on a continual learning benchmark, such as permuted MNIST or split-CIFAR-100? 

What are the additional costs to using COMET in terms of training time and memory usage?
soundness: 3
strengths: The paper is well written and the core idea is explained clearly. 

The authors demonstrate key properties of the COMET model, notably showing that similar inputs tend to activate overlapping experts, facilitated by the fixed gating mechanism.

The model is tested on a wide selection of benchmark tasks including computer vision, language modelling, and regression. 

The authors demonstrate the benefit of using COMET, particularly at large model sizes.

The use of COMET requires no additional trainable parameters which is quite advantageous.
summary: The paper introduces Conditionally Overlapping Mixture of ExperTs (COMET).
COMET uses biologically inspired, fixed random projections to generate binary masks that define subnetworks know as 'experts'.
The mask generation process is input-dependent, causing similar inputs to activate overlapping sets of experts.
The authors test the models on a range of benchmark tasks, finding that COMET performs well, particularly for large model sizes.
weaknesses: In other works these gating functions can help alleviate catastrophic forgetting for tasks that are presented sequentially, but this is something that has not been tested in this paper. 

Given that previous work has similarly employed networks to determine gating, I am not entirely convinced that the novelty here is sufficient. However, I acknowledge that unlike prior approaches, which relied on trainable gates, this method uses fixed random projections.

There will be additional computational costs to using COMET but there is not an in-depth analysis of these costs in the paper. An analysis of training/inference times and GPU memory usage between COMET and the standard models would strengthen the submission.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Is there any reason to believe this phenomenon does not already occur in large networks? \cite{elhage2022toy} describe a situation where neural networks encode the phenomenon observed in \cite{cheung2019superposition} during the course of training. Are there advantages of explicitly creating the superposition?

Figure 4 shows a strange result where the smaller_model performs consistently as well as the standard_model even for fairly low p_k values. It's unclear to me why there would be a benefit for COMET if at the neurons=3000, the smaller_network at pk=.1 will perform as well as the standard_model.  What is being gained here?

@article{elhage2022toy,
  title={Toy models of superposition},
  author={Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and Hatfield-Dodds, Zac and Lasenby, Robert and Drain, Dawn and Chen, Carol and others},
  journal={arXiv preprint arXiv:2209.10652},
  year={2022}
}
soundness: 3
strengths: Unlike other methods, the proposed COMET method has no trainable gating functions (unlike standard Mixture of Experts) and avoids representation collapse.

Does not require explicit input/task IDs or pre-defined expert specialization.

Works across multiple architectures (MLPs, ViTs, GPT, MLP-Mixers).

The work is particularly similar to \cite{cheung2019superposition}, especially with the use of a random projection matrix V to handle the decision to mask. The justifications for using random projections in \cite{cheung2019superposition} seem to align well with the described capacity benefits of the COMET method in larger networks as compared to smaller networks. In particular, with a larger number of neurons, the probability of interference between masks rapidly decreases.

@article{cheung2019superposition,
  title={Superposition of many models into one},
  author={Cheung, Brian and Terekhov, Alexander and Chen, Yubei and Agrawal, Pulkit and Olshausen, Bruno},
  journal={Advances in neural information processing systems},
  volume={32},
  year={2019}
}
summary: This paper introduces COMET (Conditionally Overlapping Mixture of ExperTs), a new method for creating sparse neural networks. The authors show using fixed, biologically-inspired routing can create more efficient and effective neural networks, particularly for larger models, while avoiding common problems in sparse architectures like representation collapse and poor knowledge transfer. The key insight is that COMET creates input-dependent sparsity without needing to learn the routing mechanism. COMET uses a fixed, biologically-inspired random projection combined with a k-winner-take-all operation to route inputs through the network, rather than using trainable gating functions.
weaknesses: The arguments for the number of experts is based on the possible permutations of masks that can be created which gives an unrealistically large number of possible experts. But this does not account for interference issues and establishing a bound more grounded in reality would be very helpful. The theory work in \cite{cheung2019superposition} should help better define these bounds.

There's a claim of "improved generalization through enhanced forward transfer", but it's unclear what experiments in this paper demonstrates better transfer learning.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Which components of COMET are inspired by the concept of biological random projection?

2. How should the hyperparameter $k$ in Equation (3) be determined?
soundness: 2
strengths: 1. COMET presents a novel routing method that replaces trainable gating functions with fixed, biologically inspired routing, which is rare in modular neural network approaches.

2. The proposed method is tested across diverse architectures and tasks, such as image classification, language modeling, and regression, suggesting versatility.
summary: The paper, titled Conditionally Overlapping Mixture of Experts (COMET), proposes a method aimed at overcoming limitations in existing sparse neural network architectures. COMET introduces a modular, sparse structure with a biologically inspired fixed routing approach that eliminates the need for task IDs and trainable gating functions, commonly associated with representation collapse and redundancy. Instead, the authors implement a k-winner-take-all cap operation, enabling experts to overlap based on input similarity. This approach aims to improve generalization and facilitate faster learning, validated across various tasks and architectures, including MLPs, Vision Transformers, and GPT-based models.
weaknesses: 1. Section 3 lacks crucial methodological details necessary for a complete understanding of the proposed COMET approach. For instance, it is unclear which specific design elements in COMET were directly influenced by the concept of biological random projections. 

2. The paper’s writing lacks clarity, making it difficult to fully understand the design of COMET. I recommend including a preliminary section that outlines the foundational Mixture of Experts (MoE) framework, followed by a clear discussion on how COMET’s design diverges from and improves upon existing methods.

3. While COMET is designed to improve modularity and interpretability, the authors do not demonstrate how the model’s interpretability has improved. More extensive interpretability metrics or qualitative evaluations would support the claimed benefits of COMET.

4. The absence of code and essential implementation details significantly hampers reproducibility and raises concerns about the robustness of the results.

Meta-review: **Summary**

The paper introduces Conditionally Overlapping Mixture of Experts (COMET), a novel method for enhancing sparse neural networks by using biologically inspired, fixed random projections to generate input-dependent binary masks. These masks define subnetworks called 'experts' whose activation overlaps based on input similarity, using a k-winner-take-all mechanism for routing inputs. This approach avoids the need for task IDs and trainable gating functions, which are common sources of issues like representation collapse and redundancy in sparse architectures. COMET is shown to perform well, especially in larger models, and improves efficiency, generalization, and speed of learning across various benchmarks and neural network architectures, including MLPs, Vision Transformers, and GPT-based models.



**Strengths**


* COMET introduces a novel routing method that replaces trainable gating functions with fixed, biologically inspired routing, a unique approach in modular neural network designs.
* The proposed method has been tested on diverse architectures and tasks, including image classification, language modeling, and regression, demonstrating its merits.
* COMET does not require explicit input/task IDs or pre-defined expert specialization, moreover, it does not require additional trainable parameters. 

**Weaknesses**

* In light of the prior work in the literature, the novelty of this paper is marginal.
* The argument for the number of experts, based on the potential permutations of masks creating an unrealistically large number, fails to consider interference issues.
* The paper lacks an analysis of the computational costs of the proposed algorithm.
* The paper suffers from overclaiming. For example, while the authors assert that COMET enhances modularity and interpretability, they fail to demonstrate any actual improvement in the model's interpretability.
* The lack of experiments in continual learning seems like a missed opportunity.

**Conclusion** 

The paper elicited polarized reviews. Reviewers fx4N and ETwf viewed it positively, though they noted its limited novelty. Reviewer rVeg assigned a low rating of 3 with a confidence of 4, but their review did not substantiate this negative assessment. The authors responded with a substantial rebuttal and engaged positively with the reviewers. After thorough consideration of the paper, the reviews, and the rebuttal, I find the work to be marginally above the acceptance threshold and therefore vote to accept the paper.

Decision: Accept (Poster)