MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition
Authors: Nicolas Menet
Michael Hersche
Geethan Karunaratne
Luca Benini
Abu Sebastian
Abbas Rahimi
Venue: NeurIPS 2023 poster (2023)
Keywords: Computation in superposition
Vector-symbolic architectures
Convolutional neural networks
Transformers
TL;DR: We present Multiple-Input-Multiple-Output Neural Networks (MIMONets) that can handle many inputs at once by performing computation in superposition.

Abstract: With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition. To reduce the computational burden per input, we propose Multiple-Input-Multiple-Output Neural Networks (MIMONets) capable of handling many inputs at once. MIMONets augment various deep neural network architectures with variable binding mechanisms to represent an arbitrary number of inputs in a compositional data structure via fixed-width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations to process the data structure holistically, leading to a speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, an unbinding mechanism recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by an instantaneous on-demand switching between a set of accuracy-throughput operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOConv and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves $\approx 2$–$4\times$ speedup at an accuracy delta within [+0.68, -3.18]% compared to WideResNet CNNs on CIFAR10 and CIFAR100.  
Similarly, MIMOFormer can handle $2$–$4$ inputs at once while maintaining a high average accuracy within a [-1.07, -3.43]% delta on the long range arena benchmark. 
Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer. Our code is available at https://github.com/IBM/multiple-input-multiple-output-nets.

MIMONets: Muttipl-Imput-Wlliipll-uuuutt Neural
Networks Exploiting Computation in Superposition
Nicolas Menet 1.2+ menetn tenetn@ethz, ch
Michael Hersche hertzurich. imm. com
Geethan Karunaratne karbzurich ibm. com
Luca Benini
Abu Sebastian ase@zurich ibm com
Abbas Rahimi! abrezurich, ibm, com
lbenini@iin tiisi ee ethz. com
IBM Research Zurich, 2ETH Zurich
Abstract
With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition. To reduce the computational burden per input, we propose Multiple-Input-Muulee Output Neural Networks (MIMONets) capable of handling many inputs once. MIMONets augment various deep neural network architectures with variable bind- ing mechanisms to represent arbitrary number of imputs in compositional data structure via width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations process the data structure holistically, leading to speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by instantaneous on-demand switch- ing between a set of securacy-throughpe operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOCony and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves 2 4x speedup an accuracy delta within [+0.68, 3. 18) % compared to WideResNet CNNs on CI- FARIO and CIFAR100. Similarly, MIMOFormer can handle 2-4 inputs once while maintaining high average accuracy within a [-1.07, -3.43/ % delta the long range arena benchmark. Finally, we provide mathematical bounds on the in- terference between superposition channels in MIMOFormer. Our code is available at htttts::/g htt.iitthb.oooomeeeeeemmmoetetet Cetom/IIMmmleeeeee tiple- wiiiippepepuu....ee-mm multiple- out-ut-nttettttt
unbinding mechanism
-3.
1 Introduction
Driven by the successes of deep learning in image and natural language processing tasks, increasingly large neural network models have been developed to reach state-of-the-an performance 44. These large models, however, increase computational complexity in terms of operation count for every event of input processing. One viable option to reduce the computational cost of processing per input is create a compositional data structure where variable number of input items (i.e., values) can be bound to corresponding protection keys, creating key-value pairs that can coexist and be processed
Research conducted at IBM Research Zurich.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Deep
Convolutional
Dog
Neural Network
Cat
Transformer
"The dog"
"barks"
cat"
Figure MIMONets simultaneously pass multiple inputs through a nonlinear function, e.g., deep convolutional network (on top) or a Transformer (on bottom). Input samples are bound with high-dimensional keys project the samples into quasi-orthogonal subspaces. The results of the individual samples are retrieved at the end of the network by unbinding with corresponding keys.
concurrently. This variable-sized data structure can be represented by fixed-width distributed rep- resentations in vector-symbolic architectures (VSAs) 7 In VSAs, the composition of different items in the data structure is based on functional compositionality (i.c.. key-value binding), which yields a
distributed representation, rather than concatenative composi- tionality. Interestingly, the resulting fixed-width distributed data structure can be transformed by one-time application function, whereby all input items are jointly transformed, leading to holistic transformation or computation in superposition [O]. This concept of computation in superposition can reduce the effective number of operations per imput by factor of the number of input items the data structure, because the function applied to the data structure holistically without decomposing the constituent items individual transformations. However, processing the VSA data structure via computation in superposition has so far been limited to linear maps (8H10..
Motivated by these observations, make the following contributions:
(I) We introduce a principled and transparent approach to Multiple-Inpul Saappl--Input-Mulipleee WwMMMMMMullttpppte Neural Networks (MIMONets) based VSA, enabling computation in superposition for highly nonlinear transformations in neural networks (Section 2). The MIMONets concept can be applied to various architectures, embracing the rich capacity provided by increasingly large models. The resulting network can handle many inputs at once, thus reducing the computational cost per input. We describe and overcome the challenges of computation in superposition, which originates from nonlinear interference of inputs in separate superposition channels.
(2) We propose MIMOConv, realization of MIMONets for deep convolutional neural network (CNN) architectures (Section 3). We provide several strategies for mitigating interference between different superposition channels, including a novel locality-preservinn preserving binding operation (PWHRR) and ssometry-inuccin regularization. Empirical evaluations on CIFARIO and CIFAR 100 show that a inputs in superposition (= 2x speedup) even with a slightly higher accuracy (0. (0.11-0.68% gain), and four inputs ( 4x speedup) with marginal drop (1.24-3.18%) (see Section 5.1)
MIMOCony built WideResNet-22- (11] can process concurrently
(3) We further extend the concept MIMONets Transformer architectures, where the calculation of attention scores poses additional challenges for the computation in superposition paradigm. To this end, we propose MIMOFormer, which relies on a 2D grid binding scheme for computing attention in superposition (Section 4). We derive probabilistic tail bounds on the distortion caused by inter- channel interference and show that our method converges to noise-free attention the limit high dimension. Our method succeeds empirically (> 96.5 66.52% accuracy) at synthetic sequence modelling tasks 12 while previous work [13) fails (< 20.04%). We also provide evaluations on the long range arena (LRA) dataset 14] in Section 5.2 using MIMOFormer that is based on the Performer architecture [15]. Compared to the Performer, MIMOFormer maintains a high average accuracy with marginal drop of 1.07% and 3.43% when handling two and four inputs at once, respectively.
(4) MIMONets allow a dynamic trade-off at inference time between accuracy and speed, i.e., they offer an instantancous on-demand switching between accuracy throuuhyt operating points using

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
single set of fixed parameters. Experimental results in Section 5.] show that a dynamic MIMOConv can seamlessly operate at various modes (== 1-4x speedup) while maintaining high accuracy compared to the best static models (< 1.82% drop).
2 MIMONets enabling Computation in Superposition
The central idea behind MIMONets (see Figure T is to simultancously pass multiple imputs as a superposition through nonlinear function fo parameterized by neural network weights 0. We isolate the individual inputs into separate protected channels by binding them with protection keys resulting in key-value data structure 7
Concept. Assuming two inputs (x(1) and x2²2), which can be generic embeddings either from images or natural language, we define a unique high-dimensional key (a and (² for each protected channel, drawn randomly at initialization. Owing to the Blessing of Dimensionality, randomly drawn high-dimensional vectors are quasi-orthogonal with high probability (see Appendix A). Consequently, binding (o) the inputs with these keys yields quasi- matsioorthogonal key-value pairs (x(1) C ( and (2(2) ④ a (2)),, ), which enables one to superpose the pairs with low interference:
a x + a (2) O xzx (2)
As discussed in Appendix A, s admits a noisy retrieval of ((1) and (2() through unbinding:
( a (1)
x (1) +a
= x11) + noise
(2)
To accelerate computing, inspired by the above-mentioned noisy retrieval, we pass the superposition through nonlinear function fe with parameters 0, such neural network, before retrieval. The quasi-orthogonality of the bound inputs allows processing each a separate protected subspace-al with single function call. To be able to recover the first processed sample fa(x!!) from fo(s), we aim to find an unbinding key (11) for which
a (1)
fo(()
fo (a 2 )x11) + (0)) O (0 o xl
(3)
So ( + )()) fo (2) x(()
The first approximation holds exact for linear fo. As discussed Section 3 a nonlinear fa can still be encouraged to allow such an approximation through appropriate weight regularization techniques and well-suited activation functions. Further, by optimizing over unbinding keys (ā(*), the second estimation (Eq. 44) can be achieved. Consequently, matching binding and unbinding keys (a(+) and a()) that confirm the approximation (Eq. and (4) set up a protected channel through the nonlinear function fe(s). Appendix A lists the design choices of the adopted VSA, which define the operations of key-value binding and unbinding, for all MIMONet variants presented in this work. In the case of image embeddings, we use circular convolution
(6) for binding and Matrix Binding of Additive Terms (MBAT) [16] for unbinding. In the case of sequence tokens, we bind and unbind using the Hadamard product (17). Binding and unbinding keys always data-independent, i.e., they depend only on the index of the protected channel. See [8] for alternative binding and unbinding options.
Dynamic Inference. Setting up N protected channels through neural network So gives almost speedup of Nx due to most computations taking place in superposition. However, as is explored empirically, increasing N adds inter-channel noise leading to a decrease in predictive accuracy. If fixed trade-off is unsatisfactory, one can build dynamic model capable of running superposition of one up N different inputs. By inserting the same input into multiple channels and averaging the output, one effectively forms an in-network ensemble, similar to [19 20). Using all protected channels for different inputs leads to a fast but less accurate model, whereas using all protected channels for the same input yields slower but accurate ensemble model. By partitioning the superposition channels on demand, arbitrary configurations between may be reached. Note that our method can instantaneously adapt to the current computational demand, without loading different model weights from the main memory. To perform across slow and fast configurations, the model should randomly switch between them during training. See Appendix A for more detailed explanation.
3

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Onc Cort
HRR (PQHIR)) Sinding
Unbinding
Deep CNN
Cat
First
Fully Connected
Conv
Superpos
Dog
Layer
Gecko
60 8
0
8
Figure 2: MIMOConv configured with N=3 channels. The input images are passed individually through the first convolutional layer before binding each feature value with a channel-specific high-dimensional key. The key-value pairs are superposed yielding a dimeesionality-preseee composition and passed through the rest of the CNN layers. The output is unbound with corresponding keys, and the unbound representations are classified separately with a shared fully-connected layer.
3 MIMOConv
This section presents how the MIMONets concept, introduced in Section 2 can be applied to construct a mutinne-inpu--multill--oupppppp CNN (MIMOConv) capable of simultaneously processing multiple images superposition. The MIMOConv architecture is shown in Figure [ Multiple input samples (N) are passed through the first convolutional layer, bound with a unique high-dimensional key based on Holographic Reduced Representations (HRR) (6. and superposed by clement-wise addition. After passing the superposed tensors through the network's main CNN layers, we obtain a combined feature vector with information on all imputs. By unbinding with separately learned keys based on MBAT (16), which amounts to matrix multiplication, we extract the individual processed information, which is then passed through final fully-commn culle-co layer to produce logits for classification. In the following. we introduce our three main contributions that lead to a highly accurate MIMOConv.
Augmenting CNNs with Decality-preserviie variable bindings. We embrace a principled and transparent binding mechanism from HRR. Accordingly, binding is performed using circular con- volution with binding key of dimension D, drawn from i.i.d. Gaussian distribution with mean and 1 1/D variance. Instead of convolving the flattened image tensor, we repeatedly apply circular convolution between the binding key and each pixel volume spanning across the feature maps (Dx1 1 1). This binding operation, which we call position-wise HRR (PWHRR), is translation equivariant and maintains locality, an essential property for subsequent layers with limited receptive fields. More concretely, binding and unbinding are performed as
(a (k) 2 (z(()))) wh = =a(k) x:
(k) w.
(5)
(a 4 h) = (kk) hi,w.h
(6)
with image tensors (+)) E DxWx hidden representation h E DD'' binding key (8) E RD and unbinding key (kk) e D''D', Here, D, D', W, and H denote the hidden dimension at binding, the hidden dimension at unbinding (generally differs from D). image width, and image height, respectively. is the circular convolution, the matrix multiplication, and i indexes the superposition channel. Unbinding is applied after the global (average) pooling to reduce computational costs. The binding keys can be either learned fixed during training (see ablation study in Appendix E).
Embracing high dimensional embeddings. According to the Blessing of Dimensionality (see Appendix A), random vectors quickly become quasi- orthogonal as their dimension increases. To reduce interference between protected channels, we increase the number of feature maps by adopting Wide Residual Networks [TT]. the most commonly used CNN architecture to achieve state-of-the -ftte-or-theart accuracy on CIFAR 100 [21). The input tensors are passed individually through the first convolutional layer before being superposed in suitably high dimensional space. We set the number of feature maps after the first convolutional layer to D=64. This 4x more than the standard Wide-ResNNt-22 [11 which results in improved training stability at marginally higher compute cost (see Section 5.1

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Encouraging isometric layers. We aim to preserve the quasi-orthogonalin of our protected channels the superposition passed through many network layers. To that end, residual connections are used and each subfunction go of type (strided) spatial convolution or activation function is made approximately inner- mnerrprrductt preserving, i.e.,
(go(z), go (ol = (ar.))
we deploy regularization to the CNN weights and use parametric ReLU [23 activation function, a learnable affine combination between identity and ReLU. Those adjustments lead to near- -eammisooetric network. The regularization for the individual CNN layers is determined by
Inspired by [22]
I(W)- Conn W) 500 1|
where
S.c..-1c.xG 3-11"11
(8)
L(W") = 2 |Coon(W"" 21 2
.W") - 80:11²
where
8 1i -lax 1 1-
(9)
with y as hyperparameter. We use L(W) if C > Co. else L(W"). See Appendix B for more details.
MIMOFormer
Unbinding
This section presents MIMOFormer, which ap- plies the principles of computation in superposi- tion to dot-product self-attention (24). Figure
MLP
shows a MIMOFormer layer with four protected channels, consisting of single-head" attention block, a concatenation, a linear layer, an MI.P, and skip connection.
Concatenation linear layer
Merely superposing protected attention keys and queries does not yield the desired result. As discussed in Appendix F, with scalar atten- tion scores between pairs of tokens, vanilla dot- product attention irreversibly combines attention scores of separate protected channels, effectively blurring the attention weights. By building in- stead on linear Transformers (15) 25 attention
FAVOR+S
A. 0lg)
Binding+ +
Binding+ calculation
Binding superpos
superpoo
Linear
Linear
Linear
scores are not collapsed to scalars, thus enabling computation in superposition.
Despite being compatible with other linear trans- formers (such as DPFP (25)), for concreteness we discuss changes to the Performer's 's FAVOR+
Figure MIMOFormer layer applying compu- tation in superposition to single-head FAVOR+S attention and to the ML.. This example passes four channels LN-M=2-2-2 to the FAVOR+S attention (see Eq. 19 The individual outputs are retrieved by unbinding after the MLP. The skip connection superposes the individual inputs for alignment, us- ing the same protection keys as in unbinding.
attention block [15]. Enabling computation in superposition, we label the block as FAVOR+S.
Given attention keys (kz))=1- queries (n:)aa1 and values (uz)f=1 FAVOR+ estimates dot- product attention at sequence index through
exp (15.9))/VD)
Σ ()))()))T) x s(g:),
0
Uj
(10)
j=1 21=1 exp (k.gg)/VD)
B,
where Φi P - R approximates the softmax kernel exp (ki, gi) /VD) as an explicit inner product. Since computing & has a computational complexity of O(D)), the construction of Dx takes OLLDR). Equally, multiplying A ((g:) Vi takes OLDRR). Thus, FAVOR+ breaks the quadratic dependence on sequence length L of attention. The denominator B, is discussed in Appendix C.
2The application multi-head head attention straightforward; empirical results are shown in Section 5.2
3 Proteccton keys are denoted by the letter and attention keys by the letter to distinguish between them.
5

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Our extension, FAVOR+S, computes attention in superposition, yielding square-root speedup in the number of protected channels. encodes them in an M x N grid, distributing the computational burden among the setup of the value-key matrix A and its product with ((g:). Importantly, in the limit of high dimensional projections D our mechanism converges to exact self-attention, completely separating the protected channels from one another. In the following, we assume the token values y keys ky. and queries 93 to be in protected subspaces:
_/mm.)) := J|
c a (m..),
k
(m)n)
Kj m)
C (m,.))
(m..) := m) 3 a (m,,), (11)
where a (m,, are i.i.d bipolar vectors of Rademachers [7 and (m, n) denotes a channel.
MIMOFormer benefits from the low time complexity (O(D) of the Hadamard product, especially since binding and unbinding are performed in every MIMOFormer layer. Our derivations rely on two estimates:
M
(())"((a)) & exp exp(((x (1k
(u,
H 2
(R)
famm))
(12)
t=1
intended signal
The approximation P. which improves with increasing R = dim(()(.)). is due to FAVOR+ and is quantified in (15). On the other hand, the approximation H follows from:
Inter-channel distortion. The probability that inter-channel attention distorts the intended signal of the dot-pruduct by a factor outside [1 ar 1 + a] has various upper bounds, most notably decaying exponentially w.r.. Do² cos?
(u.m) (L q. ( - 1)?. See Appendix D foror the full theorem.
4.1 FAVOR+S: Computing self-attention in superposition
We first discuss separately how to use a one-dimensional grid to carry out either the multiplication (x) or the construction of A in superposition. Finally, the integration into 2D grid will be shown.
Placing multiple queries superposition. We set up channels I,...,M M by simultaneously generating a superposition in the construction of A, and of the queries to be applied. To avoid inter-channel attention we superpose value-key tensor products, i.e., we do not construct tensor products between superposed values and superposed keys:
4 M
S. = ΣΣ
(n) S j=
Φ
ΣΣ f=lu=1
(u)
(13)
M
M
M
(u)
exp (14)
Σ
d H/VD)/
H
(a)
exp (165") vi
/
(14)
j=1u=1
1=1
j=1u= u=1
M
(n) exp (14)0) di \/D).
(15)
unnormalized o; channel
We obtain a superposition of bound output values; hence, the cost of computing A X ((g;) for all i amortized across channels. However, the construction complexity of A, is increased M-fold to OLLDR. M), hence the complexity per protected channel remains at (LLDR).
Constructing value-key tensor products in superposition. Next, we demonstrate a value-key tensor product (A,) shared across all channels, but with a mere O(LD(R + N)) setup complexity.
In O(LND) D) we compute Em1 By
(g) and MN kf"") for all j. These are then (re)used to build A..
1
s(m)
Σ
Σ
((
(g)
T Tola (16)
j=1
5=1Ø=1 g=1

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
L
N
N (1Σ w=1
L N
exp
kffm) ai //VD)
(n)
exp (1h5"), //DD
(17)
j=1
q=1
j=
(n) exp (1k)) o (n)) //DD)
( exp (1k) af /VD).
(18)
unnormalized or of charmel
separate protected subspace
The output contains the th channel together with noise. The operation A, (((g)), which takes (LDR)) must be repeated N times to produce outputs for all N channels, causing a bottleneck.
Simultaneous superposition of queries and value-key tensor products using 2D grid. Finally, we combine the two previously described approaches to encode the superposition channels in a 2D grid of size N x M. We multiply a constant matrix (A,) with features derived from superposition
of queries to get the superposition vector s!!))
L
M
N
N K
M
s(mi
ΣΣ
Σ
(x.9) 5
(uxe)) yr
(t.n) q
(19)
construct A, O (LMM(RR+))
construct Vi, in O(LND(R+M)
Computing the multiplication Vi, n takes O(LNDR). we M=N, using only O + LN2D) D) instead of the usual O(LDR N2). Thus, one may achieve a
evaluate 5/()) Vi,
speedup of Ominn( N2 R)) compared to FAVOR+. Since R is normally in the hundreds [15], we
can assume improvements of O N2) for reasonably large N = M MN.. Eq.
(19) simplifies to:
C (u.ar)
Σ q
(t.m))
s(m)
(.))
> ) w
(u,w) T 6
q: (())))))
P Σ
((aa) exp
(20)
VD
J.0
exp
(K/ (K(amm)
o
/DD
(21)
q=1
(K
funm)y
exp
Li
(a.g)
exp
qi
(22)
VD
VD
unnormalized or of channel (u,
noise in separate protected subspace
4.2 Integrating FAVOR+S into MIMOFormer
As is apparent in Eq. 19). the query superposition is along a different axis (M) than the key and value superpositions (N). The output of attention, however, is only superposed along a single axis (M). To be able to set up superpositions along both axes (M and N) at the next layer, we require all channels (i.e., keys, queries, and values) in separation, i.e., not superposed, at the interface between MIMOFormer layers. We present two variants of MIMOFormer with different specdups.
The first computes superposition exclusively during the attention mechanism. The individual tokens of the channel (n, m) are directly retrieved from (!") by unbinding with the key ((n.m) = (n.m) and the remaining computational steps within F performed separately.
The second (and faster) MIMOFormer instance additionally performs the concatenation, the lincar layer, as well as the MLP in superposition (shown in Figure 3). Unlike in the first variant, the skip connection around the attention block must account for the introduced superposition. To allow a potential embedding dimension mismatch, we instantiate two different sets of randomly drawn bipolar keys: one for the skip connection and post-MLP unbinding, and one for FAVOR+S binding. All keys frozen during training: it is up the trainable weights in the linear layer after concatenation to find the relationship between the binding and unbinding keys.
The function Φ in the self-attention block consists of an R=256 dimensional projection and a ReLU activation (15). Appendix provides theoretical justification for using ReLU ø and its benefits for MIMOFormer. Empirically, ReLU shows better numerical stability than unbiased softmax FAVOR+.
7

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Empirical Results
This section evaluates the proposed MIMONets various model architectures and benchmarks. Appendix and Appendix F describe the experimental setup for MIMOConv and MIMONets, respectively. All experiments are repeated five times with a different random seed. We report the mean and standard deviations of accuracy to account for variability in training.
5.1 MIMOCony
CIFARIO and CIFARI00. Our main baseline, which we adapt to MIMOConv, is a reproduced WideResNee-22-10 (. i.e., 28-layer CNN with the number of feature maps of each convolutional layer enlarged by 10x. Moreover, as stronger baseline, we include the isometry regularization in the training of WideResNNet226 10, and call the resulting network WidelsoNet-258 10. Both baselines demand 5,251 GMACs (Giga multiply-accumulate operations) per sample. MIMOConv's 's inference complexity per sample 5.335 GMACs for N=1, 2.667 GMACs for N=2, and 1.334 GMACs for N=4; hence, we get N speedup despite not accelerating the first and last layer. See Appendix E.
Table shows the accuracy of the static MIMOConv, which is exclusively trained to support either 1, 2, or 4 channels. The MIMOConv with one channel (N=1) outperforms both baselines, which may be attributed to regularizing effects of the key-value binding. MIMOConv with N=2 channels still outperforms WideRessNet228-10 while reducing the inference complexity by 2x. The complexity can be further reduced by increasing the number of superpositions to N=4 at slight accuracy drop of $3.18% 18%, compared to WideResNet-28-. 10.
Next, we evaluate the dynamic partitioning of the superposition channels select speed-accuracy operating point instantaneously, which is main feature of our approach and sets it apart from other static approaches that opt for fixed performance point like model downsizing, quantization aware training, and pruning (see Appendix A for a discussion). We set up a model with four channels, but evaluate its performance in different configurations: fast inputs/pass), normal inputs/pass), and a slow mode (1 input/pass). The fast mode maps each input to one channel; the medium mode distributes two inputs over pairs of channels; and the slow mode uses all channels for the same input. The models are trained on 80% of the batches in fast mode and on 20% of the batches in slow mode. Appendix E provides more details on the trade-off between fast and slow mode training. As Table shows, single dynamic model can seamlessly switch between operation points while maintaining a high accuracy compared to the static models (.1.82% drop).
Table 1: Average accuracy (%) of WideResNet-28-10 10 variants and our MIMOConv. Static models are trained to process N inputs in one pass, speeding up inference by Nx. Dynamic models are trained with a variable number of inputs (N=1-4), and can process a variable number of inputs per pass. We report the average accuracy + the standard deviation over five runs with different seeds.
CIFARIO
CIFAR100
inputs/pass
WidcResNet-28-10 WideIsoNet-28-100
06002
n.a.
n.a.
$1.62200.07
n.a.
n.u.
97. 90.11
n.a.
82.3840.20
n.a.
n.a.
MIMOConV static (N=1)
07.49
n.a.
n.a.
83. 19.117
n.a.
n.a.
MIMOConV static (N=2)
n.a.
96.93
n.a.
n.a.
n.a.
MIMOConV static (N=4)
n.a.
n.a.
95.58
n.a.
n.a.
78.44==0.50
MIMOConv dynamic (N=1-4)
97. 97.13+0.11
96.41
95.43+0.07
82.52+0.09
80. 40..8
+0010
The detailed ablation study in Appendix E makes the following findings: (1) isometry regularization improves accuracy for any number of channels; (2) training MIMOConv for more epochs closes the performance gap to the single-input baseline; (3) an appropriate number of feature maps (32 or 64) in the first layer stabilizes training; and (4) the binding keys can frozen during training without loss (whereas unbinding keys are never frozen).
8

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
MNIST and SVHN. Figure @egure4comee compares MIMO-
Conv with DataMux 13 on the MNIST dataset. Even with trivial downsizing for fair compar- ison from 28-layer very-wide (10x) ResNet
100
to 10-layer narrow (1x) network, MIMOCony scales much better to high superposition chan- nels (N) than DataMUX does. Indeed, our model shows an accuracy of 80.4% against their 52.9% in case of N=16 superposition channels (high- est number of channels reported by DataMUX for vision tasks), despite being computationally cheaper (0.47 MMAC/s vs. 0.65 MMAC/s). Also, DataMux's 's binding overhead results in mere 1.35x reduction in MACs compared to our 10.9x as N goes from to 16. Ergo, our method scales Figure 4: Classification accuracy (%) on MNIST better in accuracy and throughput as N increases. for downsized model.
Accuracy (%)
70
60
MIMOConv-10-1
DataMUX: CNN Nonlinear (8x)
16
Number of superposition channels (N)
Finally, tested MIMOConv on the SVHN dataset. Despite limited hyperparameter tuming, MIMOConv achieves a high accuracy of 97.17% (N=1), and can maintain its performance with larger superpositions (97.05% and 96.84% for N=2 and N=4, respectively).
5.2 MIMOFormer
LRA. We evaluate MIMOFormer on five tasks from LRA [14,, and compare against the vanilla Transformer (24) and the Performer (15) using FAVOR+ attention with ReLU projection. Moreover, we also consider wide Transformer variants (26), consisting of only one layer but many heads as their deep counterparts. saek-sp architectures and training hyperparameters are kept the same for the Performer and the MIMOFormer (see Appendix F).
Table 2: Test accuracy (%) the long range arena (LRA). MIMOFormer
equal number of query superpositions (M) and value-key superpositions (N), i.e., N=M. Computation superposi- tion performed either in attention only (att.) in both attention and MLP (att. is the number of layers, // the number of heads, and indicates curriculum learning.
Lis1Ops
Text
Retrieval
Image
Pathfinder
Avg.
Deep models
L=6, H=8
L=6, H=8
L=4, H=l
L=3, H=l
L=4, H=$
Transformer
24
36.37
64.27
37.46
42.44
71.40
53.39
Performer
18.01
65.40
53.82
42.77
77.05
51.41
Performer reproduced)
38.94
65.70 65.7040.31
81.5840.18
40.14
73.8240.78
60.04
MIMOFormer (N=2, att.)
38.0840.21
65.0010.29
19.3740.81
38.21
72.36
58.61 10.49
MIMOFormer (N=2, att.+MPP
37.65+0.55
61.39t0.22
76.02
33.85
67. 67.98
55 65.98:00.37
MIMOFormer (N=4, alt.)
37. 47.22
64.5940.11
60.9949.06
28. 28.16+20.08
5-.50
49 49.29
MIMOFormer (N=4, att.)"
37.64
64.46+0.15
74.38
30.52
67. 67.10)+0.45
54 54.82+0.33
MIMOFormer (N=4, MMI.+
17.7440..0
60.71 15.14
L0.20
24.01 10.47
50.33 50.33 10.16
45.00 11.34
Wide models
L=1, H=
L=1, HF488
L=1, H=16
L=1, H=12
L=1, H==2
Performer (reproduced)
30.4
65, 65,73 82
83,6740.25
41.67700:4
74.11
60.93++0.37
MIMOFormer (N=2, all.)
38.9040.53
65.3940.14
81.2740.29
40.25
73.51 4.0.23
59 59.86
MIMOFormer (N=2, att. +MP)
37.59
64.6410.25
78.3020.52
36.69
68. 68.22
57 07.09
MIMOFormer (N=4, att.)
37.71#0.24
64.22*0.14
44.9
35.43+0060
69 00.32
56.37
MIMOFormer (N=4, att.)"
37.68t0.36
64.56t00.25
76.37
35.53
73.37+0.22
57. t..56
MIMOFormer (N=4, alI.+MPP)
18.5244.99
63.5341.11
44.30
26.54 10.29
56.33
47 47.84 10.36
Owing to an improved training setup, our replicated deep and wide Performer baselines substantially outperform the results reported in [14] (see Table 2). Moreover, MIMOFormer enables accurate computation in superposition for both deep and wide attention models. The performance drop is less pronounced in wide models (only 1.07% drop compared to Performer with N=2, att.). which may be
attributed to the larger number of heads, increasing the effective dimension (Dr.t = D Dnead).
When computing both attention and the MI.P in superposition (atL++MPP), we observe better scaling (in N) for wide models. Also, MIMOFormer reduces the gap to the baseline as the number of epochs increases (sce Appendix F).

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
To stabilize training in the case of N=4, we implemented a curriculum training procedure where the number of superpositions reduced N2=N/2 during a warmup phase (1/6th of the training steps), improving the average accuracy of MIMOFormer in both wide and deep models.
Comparing against reproduced DataMUX (38.08% vs. 30.54% accuracy) when using models of similar size and ==2, see Appendix F.
[
MIMOFormer (att.) outperforms it ListOps
Synthetic sequence modeling. Table 3 reports the accuracy on two synthetic sequence modeling tasks, which Trans- former alternatives such as S4 (27) have difficulties solving [12). On these more nuanced NLP tasks, the accuracy of Data- MUX 13 drops to 20.04% and 6.06% for 2 despite significant efforts in training, while MIMOFormer, at a score of 96.52% and 99.40% respectively, succeeds. We attribute this difference in performance attention score blurring in DataMux, discussed in Appendix F. Contrastingly, our method converges to exact attention without blurring. It is versatile and can be adjusted other linear Transformers such as DPFP 25
Table 3: Accuracy (%) on synthetic sequence modelling.
Associative recull [12]
Induction head [2]
Architecture
Attention
Transformer Performer
Softmax FAVOR+
98.4841 87
100
96.32
31.58 +33.67
MIMOFormer (N=2, att.)
FAVOR+
96.52
99. 09.10
MIMOFormer (N=2, att.)
DPFP 25
93.64
98.56 98.56+90.6
DutaMUX (N=2) 13 Softmax
20.04+1.72
6.06
achieving a score of 93.64% and 98. 58.56%
Related Work
So far, superposition principles have been applied in order to store the weights of multiple models in a single neural network 30), to circumvent catastrophic forgetting in continual learning (2]
and to render symbolic reasoning tractable (33). To address privacy concerns when running remote inference, single inputs were bound with random channels to implement
(34).
Recently in [35]. HRR was used to define unconventional version of self-attention, whose attention scores are processed to a diagonal matrix. The value vectors are scaled according to their importance in the sequence instead of being combined in weighted sum. In contrast us, none of these works superpose multiple inputs into a data structure to speed up computation.
In 19[20), an ensemble of CNN models was fit into one network. However, by only broadcasting a single input over the channels and by averaging all the outputs, this approach collapses to (SISO) network. On the contrary, explore using protected channels for different inputs at inference, resulting in an actual multiple-input-mullee multiple-output (MIMO) network.
There has also been a line of work to accelerate Transformers using inputs in superposition [3] [36].
DataMux (13) claims to retain high performance for language understanding tasks, even when using up to 40 inputs in superposition. However, none of the reported tasks require attention layers at all (37]. In Section 5.2 we show failure of their method when actual attention required (see also Appendix F). MUX-PLMs (36) improves on DataMux with contextual binding and replaces token prefixes with unbinding keys, but does not address the blurry attention mechanism. In contrast to DataMUX and MUX-PLM PLMs, our work approximates true attention and our theoretical derivations show convergence to actual dot-product attention as the dimension of attention projections increases, giving us even stronger case for applicability to large language models.
Conclusion
We present MIMONets that simultancously process multiple inputs by performing computation in superposition. MIMONets bind arbitrary inputs with high- nigh-dimensional keys, which projects them orthogonal subspaces that, together with near-isometric subfunctions, guarantee low interference through all nonlinear layers. Unbinding with (learned) keys can safely retrieve information on individual channels. We provide two MIMONets instances, MIMOCony and MIMOFormer, that show the effectiveness of computation in superposition through two dominant operations in neural network architectures: convolution and attention. Further investigations could explore the MIMO- capability of architectures that contain additional nonlinearities (e.g.. max-pooling) and use different input modalities. MIMONets could be suitable candidates accelerate dynamically and on-demand the inference of foundation models (38).
10

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
Acknowledgement
This work supported by the Swiss National Science foundation (SNF), grant 200800. We thank Dario Bolli for conducting initial experiments and Aleksandar Terzić for helping with ablation studies.
References
[1] Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint P/Xi::8/00008005 2018.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P.Syamm G. Sastry, A. Askell et al., "Language models are few-shot learners," " Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877-1901, 2020.
[3] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 10012-10022.
[4] Z. Liu, Mao, C.-Y. Wu, C. Feichtenhofer, Darrell, and S. Xie, "A convnet for the 2020s," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. pp. 11976-11999 986.
[5] R. W. Gayler, "Vector symbolic architectures answer Jackendoff's challenges for cognitive neuroscience," in Joint International Conference on Cognitive Science (ICCS/ASCS), 2003.
[6] T. A. Plate, "Holographic reduced representations," IEEE Transactions on Neural Networks, vol. 6, no. 3, pp. 623-641, 1995.
[7] P. Kanerva, "Hyperdimensional computing: An introduction to computing in distributed rep- resentation with high-dimensional random vectors," Cognitive Computation, vol. no. 2, pp. 139-159, 2009.
[8] Neumann, "Learning holistic transformation of HRR from examples," International Conference on Knowledge-Boud Based Intelligent Engineering Systems and Allied Technologies (KES), 2000, pp. 557-560.
[9] "Learning the systematic transformation of holographic reduced representations," Cogni- live Systems Research, vol. 3, no. 2, pp. 227-235, 2002.
[10] D. Kleyko, M. Davies, E. P. Frady, P. Kanerva, S. J. Kent, B. A. Olshausen, E. Osipov, J. M. Rabaey, D. A. Rachkovskij, A. Rahimi, and F. Sommer, "Vector symbolic architectures computing framework for emerging hardware," Proceedings of the IEEE, vol. 110, no. 10, pp. 2022.
[11] S. Zagoruyko and N. Komodakis, "Wide residual networks." " in Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, September 2016, pp. 87.1-87.12.
[12] D. Y. Fu, Dao, Towards language modeling with state space models," in The Eleventh International Conference on Learning Representations (ICLR), 2022.
Saab, W. Thomas, Rudra, and C. Re, "Hungry hungry hippos:
[13] V. Murahari, C. Jimenez, R. Yang, and K. Narasimhan, "DataMUX: Data multiplexing for neural networks." Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 17 17555-17521 527, 2022.
[14] Y. Tay, M. Dehghnni, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, "Long range arena: A benchmark for efficient transformers," in International Conference Learning Representations (ICLR), 2021.
[15] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser et al., "Rethinking attention with performers." International Conference on Learning Representations (ICLR). 2020.
[16] S. I. Gallant and T. W. Okaywe, "Representing objects, relations, and sequences," Neural Computation, vol. 25, no. 8, pp. 2038-2077 2013.
[17] R. W. Gayler, "Multiplicative binding, representation operators & analogy." in Advances in Analogy Research: Integration of Theory and Data from the Cognitive, Computational, and Neural Sciences, 1998.
11

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
[18] D. Kleyko, D. Rachkovskij. E. Osipov, and A. Rahimi, "A survey on hyperdimensional comput- ing aka vector vector symbolic architectures, part I: Models and data transformations." " ACM Computing Surveys, vol. 55, no. 6, 2022.
[19] M. Havasi, R. Jenatton, S. Fort, J. Z. Liu, J. Snoek, B. Lakshminarayanan, A. M. Dai, and D. Tran, "Training independent subnetworks for robust prediction," International Conference on Learning Representations (ICLR), 2021.
[20] A. Ramé, R. Sun, and M. Cord, "Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks," " in Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2021, pp. 823-833.
[21] A. Krizhevsky, "Learning multiple layers of features from tiny images," " University of Toronto. 2009.
[22] H. Qi, C. You, X. Wang, Y. Ma, and J. Malik, "Deep isometric learning for visual recognition," " in International Conference on Machine Learning (ICML). PMLR, 2020, pp. 7824-7835.
[23] K. He, X. Zhang. S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026-1034.
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017.
[25] I. Schlag, K. Irie, and J. Schmidhuber, "Linear transformers secretly fast weight pro- grammers," in International Conference on Machine Learning (ICML). PMLR, 2021, pp. 9355-9366.
[26] J. R. Brown, Y. Zhao, I. Shumailov, and R. D. Mullins, "Wide attention is the way forward for transformers?" in NeurlPS '22 Workshop on All Things Attention: Bridging Different Perspectives on Attention, 2022.
[27] A. Gu, K. Goel, and C. Re, "Efficiently modeling long sequences with structured state spaces." in International Conference on Learning Representations (ICLR), 2022.
[28] B. Cheung, A. Terekhov, Y. Chen, P. Agrawal, and B. Olshausen, "Superposition of many models into one," Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
[29] M. Hersche, P. Rupp, L. Benini, and A. Rahimi, "Compressing subject-specific brain-computer interface models into one model by superposition in hyperdimensional space," " in 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2020, pp. 246-251.
[30] M. Zeman, E. Osipov, and Z. Bosnié, "Compressed superposition of neural networks for deep learning in edge computing," in 2021 International Joint Conference on Neural Networks (ICCN). IEEE, 2021, pp. 1-8.
[31] M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi. M. Rastegari, J. Yosinski, and A. Farhadi, "Supermasks in superposition." Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 15 173-15 184, 2020.
[32] M. Zeman, J. F. Pucer, I. Kononenko, and Z. Bosnić, "Superformer: Continual learning superposition method for text classification," Neural Networks, vol. 161, pp. 418-436, 2023.
[33] M. Hersche, M. Zeqiri, L. Benini, A. Sebastian, and A, Rahimi, "A neuro-vector-symboli architecture for solving Raven's 's progressive matrices," " Nature Machine Intelligence, vol. 5,
no. 4. pp. 363-375, 2023.
[34] M. M. Alam, E. Raff, T. Oates, and J. Holt, "Deploying convolutional networks on untrusted platforms using 2D holographic reduced representations," in International Conference on Machine Learning (ICML). PMLR, 2022, pp. 367-393.
[35]
"Recasting self-attention with holographic reduced representations." in Proceedings of 8TH SIGKDD International Workshop on Mining and Learning from Time Series - Deep Forecasting; Models, Interpretability and Applications (MiLeTS 2022), 2022.
[36] V. Murahari, A. Deshpande, C. Jimenez, I. Shafran, M. Wang, Y. Cao, and K. Narasimhan, "Mux- plms: Data multiplexing for high-throughput language models," in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 4540-4554.
12

---

37th Conference on Neural Information Processing Systems (NeurIPS 2023)
[37] M. Hassid, H. Peng, D. Rotem, J. Kasai, I. Montero, N. A. Smith, and R. Schwartz, "How much does attention actually attend? Questioning the importance of attention in pretrained transformers." in Findings of the Association for Computational Linguisties: EMNLP 2022,
2022, pp. 1403-1416.
[38] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., "On the opportunities and risks of foundation models," " arXiv preprint arXiv:2108. 07258,, 2021.
13


Reviews:
[{"review_id": "Qs5NZ4wzx5", "reviewer": "Reviewer_b2eJ", "summary": "UPDATE: scores updated based on rebuttal.\n\nThis paper proposes a method (MIMONets) for multiplexing multiple independent samples in superposition in such a way that one can train neural networks to simultaneously process those samples in training and inference. The method is adapted both for CNNs and Transformers and there is empirical evaluation for both that show the viability of the method. ", "questions": "See above", "limitations": "Comparison for DataMUX missing, some apples-to-oranges comparisons in the experimental section.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easily understandable, and the method is relatively well grounded in foundations of previous works and new analysis. ", "weaknesses": "The authors claim that this is the first time this has been done, but they fail to mention some related work such as Murahari, Vishvak, et al. \"DataMUX: Data multiplexing for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 17515-17527. Please review that work and discuss similarities and differences, and if possible conduct experimental comparison. The DataMUX paper claims an impressive 40 samples multiplexing for the transformers, while MIMONet uses just 4. At least, the CNN part of the MIMONet work seems to give better empirical results, though.\n\nOne of the strongest motivators for the MIMONets is the ability to “dynamically” scale computation vs. accuracy at inference run-time using the same trained weights by changing the way input and output is processed, and by creating in-network ensembles if more resources are available. The authors should center the motivation more in this area unless they can have a strong argument to the “static” case.\n\nRegarding the static case, there is less convincing results. For example, since there is a drop in accuracy with multiplexing 2-4 samples compared to 1, the comparison should be done with a smaller baseline with the same accuracy, and the authors should include full FLOPS figures for the models, including the bind/unbind operation.\n\nAlso, for the static case, some comparison methods include pruning, quantization aware training, compression. It would be needed to compare these in terms of intOPS/FLOPS and power consumption estimates using models of the same accuracy.\n\nMIMOConv\n\nFor the MIMOConv model, the authors should include the mathematical description of bind/unbind including all dimensions and indexes.\n\nTable 1 does not clearly state the FLOPS / sample for each of the rows. \n\nTable 1 does not discuss what would be the FLOPS size of a baseline model that reaches the same accuracy as e.g., N=4 MIMOConv models.\n\nMIMOFormer\n\nThe authors should add some more information on the Transformer model, for example, in the Figure 3 it seems every layer does bind/unbind. This seems different from the CNN models, so that authors should describe this more in detail and give more background to these choices. What is the complexity addition in FLOPS from these bind/unbind operations? \n\nTable 2 (MIMOFormer results) is not fully clear what is the computational complexity (per sample) for each of the rows. For example, do the MIMOFormer models have higher complexity in FLOPS because of the bind/unbind operations? I assume that MIMOFormer N=4 is almost 4x more efficient pers sample than N=1, but it would be great to report full model FLOPS/sample for each of the rows.\n\nTable 2 seems to hint that the +MLP version performs worse (especially for N=4). But do I understand correctly that it also has less FLOPS since the -MLP version repeats the same MLP for each of the superimposed samples? This should be described. Would the +MLP version perform better if MLP was bigger?\n\nTable 2 is missing comparison to a non-MIMO Performer baseline that would have the same FLOPS complexity per sample.\n\nIn Table 2 there seems to be some tasks, such as retrieval, where N=4 seems to make the performance drop dramatically. Could the authors discuss these? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">“[the authors] fail to mention some related work […] Please review that work and discuss”\n\nThank you for pointing out the DataMUX paper. Given its importance, we discuss key differences and compare against it empirically on additional benchmarks (see pdf) in the global response. In short, DataMUX does not design a dedicated binding method for CNNs, and their Transformers not only introduce additional coarse-grained “sentence summary” tokens but also compute fuzzy attention by sharing attention scores over sentences. Empirically, this is visible in their failure on the synthetic language benchmarks “associative recall” and “induction head”, which require faithful attention (scoring 20.04% and 6.06%) while our method succeeds (scoring 96.52% and 99.40%). Moreover, our Transformer with N=2 outperforms DataMux with N=2 on ListOps, 38.08% vs. 30.54%. Also, on high superpositions (N=16) our CNN method outperforms theirs (80.4% vs. 52.9% on MNIST) while being computationally cheaper. We will include these findings in a revision.\n\n>“[the strongest motivator for the MIMONets is the ability to “dynamically” scale computation vs. accuracy at inference run-time […] the authors should center the motivation more in this area”\n\nThanks for the feedback. We tried to highlight dynamic inference as our strongest advantage and most significant innovation, but we will improve on the wording and put further emphasis on it.\n\n>“Regarding the static case, there is less convincing results”\n\nWe agree that dynamic inference is more convincing than static, as we add the capability of high-speed inference without sacrificing accuracy at normal speed. The rationale for including static results is to display the very slim drop in performance of each operating point against the static models. We will make that clearer. Even so, we outperform state of the art alternatives as mentioned above.\n\n>“[the static case should be compared to pruning, quantization aware training, and compression] in terms of intOPS/FLOPS…”\n\nWe do not claim to outperform other throughput-increasing approaches like model downsizing, quantization, and pruning. However, in our opinion, we do not have to due to their lack of ability for dynamic accuracy-throughput selection. Furthermore, our approach is orthogonal to pruning, quantization, compression, and low-rank matrix decompositions. Each of them could be applied on top of our MIMO approach. However, for reasons of transparency, we include more detailed MACs in Table R2, R3, and Figure R1.\n\n>“For the MIMOConv model, the authors should include the mathematical description of bind/unbind including all dimensions and indexes.”\n\nThank you for the suggestion; we will add the following:\n\nPWHRR is given by (with image tensors $x^{(k)} \\in \\mathbb{R}^{D \\times W \\times H}$ and binding key $a^{(k)} \\in \\mathbb{R}^D$)\n\n$$(a^{(k)} \\odot x^{(k)}) {\\scriptsize {:,w,h}} = a^{(k)} * x^{(k)}_{:,w,h}$$\n\nwhere $*$ is the usual circular convolution and $k$ indexes the superposition channel.\n\nUnbinding is performed by a linear layer (with hidden tensor $h \\in \\mathbb{R}^{D \\times W \\times H}$ and unbinding key $\\tilde{a}^{(k)} \\in \\mathbb{R}^{d \\times d}$)\n\n$$ (\\tilde{a}^{(k)} \\oslash h) {\\scriptsize :,w,h} = \\tilde{a}^{(k)} \\cdot h_{:,w,h}$$\n\nwhere $\\cdot$ is the usual matrix multiplication. Unbinding is applied after the global pooling to reduce MACs.\n\n>“In the Figure 3 it seems every layer does bind/unbind. This seems different from the CNN models, so that authors should describe this more in detail and give more background to these choices” \n\nYes, this is indeed different to MIMOConv. In our proposed attention mechanism, we lay out the superposition channels along a 2D grid. We construct two separate superpositions along different axes, which is the only configuration s.t. attention scores are not shared between channels, i.e., remain accurate. Since the output of our attention layer is a single superposition along one of the two axes, we need to dismantle it for the next layer. Section 4.1 and 4.2 discussed these choices in greater detail.\n\n>“What is the complexity addition in FLOPS from these bind/unbind operations?”\n\nThe relative complexity overhead in MACs is very low (0.06% to 0.14% depending on the configuration, see Table R3 in pdf). \n\n>“Table 2 is not fully clear what is the computational complexity (per sample) for each of the rows.”\n\nThank you for the feedback. We added these results in Table R3.\n\n>“I assume that MIMOFormer N=4 is almost 4x more efficient per sample than N=1, but it would be great to report full model FLOPS/sample for each of the rows”\n\nThis is the case. When examining Table R3 and neglecting the cost of K/Q/V-projections, N=4 att.+MLP is exactly 3.98 times faster than Performer. Not superposing before the K/Q/V-projection is a clear oversight. As is apparent from equation (18) in the paper, one could instead superpose before projecting without incurring losses. Future work could address this.\n\n>“[does +MLP have less FLOPs than -MLP, given it has lower accuracy?]”\n\nYes, the -MLP version repeats the same MLP for each superposed sample. The additional data in Table R3 should clarify.\n\n>“Would the +MLP version perform better if MLP was bigger?”\n\nNice point. The performance drop between +MLP and -MLP is significantly bigger in tasks with a low embedding/hidden dimension (e.g., Retrieval, Image, Pathfinder). This is also supported by our theoretical analysis, which indicates vanishing interference as the network increases in size. It also motivates extrapolation and experimental validation on large language models in the future.\n\n>“In Table 2 there seems to be some tasks, such as retrieval, where N=4 seems to make the performance drop dramatically. Could the authors discuss these?”\n\nThose are issues with training stability, as is apparent in the large standard deviation reported in this configuration. We could opt to discard outliers or report the median instead."}, {"review_id": "MxHyb5KhUy", "reviewer": "Reviewer_A964", "summary": "This paper proposes a novel method named multiple-input-multiple-output (MIMO) neural networks, which aims to achieve simultaneous inference for several inputs together by mixing them into one input. To this end, the authors invent one method to first encode the inputs, whose output can be decoded to give separate outputs. The proposed method is evaluated on both convolutional and attention models on CIFAR datasets and LRA framework.", "questions": "Could the author provide more explanation on how to apply the proposed method in practice?", "limitations": "The authors did not provide limitations. I think the practical application of the proposed method can be limited.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method to mix several inputs is novel and interesting. Also, the authors conducted simple analysis and several experiments.", "weaknesses": "The method is only verified on some small datasets like CIFAR. Also, the results in Table 1 seem to give a significant accuracy drop for multiple input cases, compared to using a single input. Also, it would be better if the author could provide more results on other tasks, like object detection or segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">“The method is only verified on some small datasets like CIFAR […] it would be better if the author could provide more results on other tasks […]”\n\nWe appreciate the reviewer's suggestion to conduct additional experiments on other tasks. We addressed this by adding results on MNIST (see Figure R1 of the pdf) and two synthetic language tasks (“associative recall” and “induction head” reported on in Table R1) in addition to the benchmarks present in the paper (CIFAR10/CIFAR100 and LRA). The mentioned synthetic language tasks have been found to be challenging for language models without attention, such as S4 (see Y. Fu, Dao, et al. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models”), but are easily solved by our method (in superposition) with an accuracy of 96.52% and 99.40% respectively. This is a strong indicator of the fidelity of our attention in superposition. \n\nIt is also important to note that this paper is partly of theoretical nature with many insights on the qualitative behavior of error-terms together with quantitative bounds, see the extensive Appendix. The mathematical bounds are especially powerful for large scale models and show vanishing interference in these cases. Given the successful results on smaller benchmarks we plan to extrapolate and experimentally validate on large language models in future.\n\n>“the results in Table 1 seem to give a significant accuracy drop for multiple input cases, compared to using a single input”\n\nAlthough a drop in accuracy is discernible, our method substantially improves on the state of the art Murahari, Vishvak, et al. “DataMUX: Data multiplexing for neural networks”; see our comparison on MNIST, two synthetic language tasks, and a subtask of LRA in Figure R1 and Table R1. In fact, in terms of accuracy, for high superpositions (N=16) our CNN method outperforms theirs 80.4% to 52.9% on MNIST (the only vision dataset they report on) while being computationally cheaper; our Transformer (N=2) outperforms theirs on ListOps 38.08% to 30.54%, and our method does not fail on the challenging synthetic language benchmarks which require faithful attention (scoring 96.52% and 99.40%) while theirs does (scoring 20.04% and 6.06%). This is because DataMUX blurs attention scores by sharing them between superposed sentences. Finally, as is quantified for dynamic inference in Table 1 of the paper submission, our method enables **a single model** with fixed parameters to be run at different accuracy-throughput operating points. In particular, at normal speed (N=1) the model performs as accurate as the baseline. This is a unique case where one obtains essentially free lunch, high accuracy and high throughput are guaranteed and can be balanced at will.\n\n>“Could the author provide more explanation on how to apply the proposed method in practice?”\n\nExcellent request, due to space limitations we did not explore the practical motivation to great extent. However, the following will be added to the next revision:\n\nThink of large language models which cost enormous amounts of money to train and run, require real-time response, and whose usage fluctuates heavily over time. By training such a model for dynamic inference a provider could ensure to service all its costumers no matter the demand, albeit at the cost of a (hopefully) unnoticeable drop in performance. Alternatively, an autonomous system on a tight memory and power-budget might need a higher accuracy in critical situations. Owing to memory-constraints, having multiple models (with different energy consumption) in memory might not be feasible and incurs additional data-transfer costs due to switching. MIMONet allows a framework where these otherwise incompatible demands can be fulfilled.\n\n>“The authors did not provide limitations. I think the practical application of the proposed method can be limited.”\n\nWe agree and will add a more dedicated limitations section. It will read as follows:\n\n- MIMONets make use of the Blessing of Dimensionality, that with high probability exponentially many (in dimension D) vectors are almost orthogonal. Although the components of MIMONet are made near isometric through regularization, a certain number of (hidden) dimensions is still necessary. This naturally limits MIMONets to large (oftentimes over-parametrized) models or models employing low-rank decompositions.\n\n- The number of inputs that can be superposed without incurring heavy losses in accuracy is limited given a fixed neural network due to increasingly strong interference between the superposition channels. \n\n- The proposed superposition capable attention mechanism converges to faithful attention (without interference between channels) as the embedding dimension increases, but at the price of only a speedup of $N$ when using $N^2$ superposition channels. Being built on linearized attention such as FAVOR+, it further inherits all their benefits (linear scaling) and drawbacks (limited parallelization and increased memory accesses for autoregressive training (see Section 3.1 in Hua, Dai, Liu, et al. “Transformer Quality in Linear Time”). On the other hand, trivial superposition would yield a speedup of $N^2$ instead, but at the cost of blurring the attention scores with each token-token score summarizing attention in all superposition channels at once. Such models employing blurry attention are limited to application where imprecise “summarizing” information suffices.\n\nRegarding the last point we demonstrate in Table R1 of the pdf that our proposed method of superposition for attention is indeed not restricted to FAVOR+ from the Performer, but instead widely applicable to other linear transformers such as DPFP (Deterministic Parameter-Free Projection) from I. Schlag et al. “Linear transformers are secretly fast weight programmers\"."}, {"review_id": "1x4yD5Bkaa", "reviewer": "Reviewer_zbz3", "summary": "The main content of the article is about MIMONets, which are multiple-input-multiple-output neural networks that exploit computation superposition. By using fixed-width distributed representations in vector-symbolic architectures, MIMONets can represent a variable number of inputs in a data structure and process them holistically with nonlinear neural transformations. This leads to a significant reduction in computational burden per input and offers a dynamic trade-off between accuracy and throughput. The article presents two instances of MIMONets (MIMOConv and MIMOFormer), which apply the concept of computation in superposition to convolutional neural network (CNN) and Transformer architectures, respectively. Empirical evaluations show that MIMONets achieve significant speedups while maintaining high accuracy.", "questions": "1. In lines 86 and 87, why is it stated that n and $x^{(1)}$ are orthogonal?\n2. I don't fully understand how Dynamic Inference is performed.\n3. Are unbinding keys shared among all data or does each sample have a corresponding unbinding key?", "limitations": "I believe the limitations of the method mainly manifest in two aspects:\n1. Performance significantly decreases when a large number of samples are stacked.\n2. It remains uncertain whether similar performance can be maintained when applying the method to a wider range of structures.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. The method cleverly combines multiple inputs into a single sample for inference, reducing the computational cost. This is highly meaningful for practical applications.\n2. The method achieves promising results in both CNN-based and Transformer-based structures, indicating its versatility and applicability.", "weaknesses": "Weaknesses:\n1. The experimental datasets and network applications in the study are relatively limited. It would be beneficial to apply the method to more datasets, such as ImageNet, and explore its effectiveness on a wider range of networks.\n2. When the number of stacked samples is 4, a noticeable performance drop is observed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">“The experimental datasets and network applications in the study are relatively limited. It would be beneficial to apply the method to more datasets...”\n\nWe value the reviewer's proposal to conduct additional experiments on other tasks. We addressed this by adding results on MNIST (see Figure R1 of the pdf) and two synthetic language tasks (“associative recall” and “induction head” reported on in Table R1). These tasks have been found to be challenging for language models without attention, such as S4 (Fu, et al. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models”), but are easily solved by our method with accuracy 96.52% and 99.40% respectively. This is a strong indicator of the fidelity of our attention in superposition. \n\nUnfortunately, due to licensing issues, we were not able to use some datasets such as ImageNet. We also want to stress that this paper is partly of theoretical nature with many insights on the qualitative behavior of error-terms together with quantitative bounds (see Appendix). The mathematical bounds are especially powerful for large scale models and show vanishing interference in these cases. Given the successful results on smaller benchmarks we plan to extrapolate and experimentally validate on large language models in future.\n\n>“When the number of stacked samples is 4, a noticeable performance drop is observed.”\n\nAlthough a drop in accuracy is discernible, our method clearly improves on the state of the art DataMUX; see our comparison on MNIST, two synthetic language tasks, and a subtask of LRA (in Figure R1 and Table R1). Indeed, for high superpositions (N=16) our CNN outperforms theirs 80.4% to 52.9% on MNIST (their only vision dataset) while being computationally cheaper, our Transformer (N=2) outperforms theirs on ListOps 38.08% to 30.54%, and our method does not fail on the challenging synthetic language benchmarks, which require faithful attention (scoring 96.52% and 99.40%) while theirs does (scoring 20.04% and 6.06%). This is because DataMUX blurs attention scores by sharing them between superpositions. Finally, as is quantified in Table 1, for dynamic inference, our method enables **a single model** with fixed parameters to be run at different accuracy-throughput configurations: at normal speed (N=1) the model performs as accurate as the baseline.\n\n>“In lines 86 and 87, why is it stated that n and x(1) are orthogonal?”\n\nIn the case of Hadamard binding $a^{(1)} \\oslash a^{(1)} \\odot x^{(1)} = x^{(1)}$ and hence the random noise vector $n = a^{(1)} \\oslash a^{(2)} \\odot x^{(2)}$ is a random vector unaffected by $x^{(1)}$. As a high-dimensional random vector it is almost orthogonal to a fixed vector $x^{(1)}$ with high probability. This “Blessing of Dimensionality” is explored in more depth for vectors of Rademachers in Appendix A.2.\n\n>“I don't fully understand how Dynamic Inference is performed.”\n\nThanks for the feedback. With it at the center of our motivation we will add further explanation as follows: \n\nTo illustrate the idea of dynamic inference, suppose only two superposition channels are used with binding keys $a^{(1)}, a^{(2)}$ and unbinding keys $\\tilde{a}^{(1)}, \\tilde{a}^{(2)}$. We already know how the model performs standard computation in superposition (see (1) - (4) in paper). Let us thus examine how a network with the same parameters can instead be used as an ensemble-method with higher accuracy, but lower throughput. A superposition is established of twice the same input $x^{(1)}$:\n\n$$s = a^{(1)} \\odot x^{(1)}  +  a^{(2)}  \\odot x^{(1)} $$\n\nAfter applying the deep neural network $f_\\theta$ to the superposition, we may unbind as\n\n$$ \\tilde{a}^{(1)} \\oslash f_\\theta(s) \\approx \\tilde{a}^{(1)} \\oslash f_\\theta\\left(a^{(1)} \\odot x^{(1)}\\right) +  \\tilde{a}^{(1)} \\oslash f_\\theta\\left(a^{(2)} \\odot x^{(1)}\\right) \\approx f_\\theta\\left(x^{(1)}\\right) + \\tilde{a}^{(1)} \\oslash f_\\theta\\left(a^{(2)} \\odot x^{(1)}\\right). $$\n\nand\n\n$$\\tilde{a}^{(2)} \\oslash f_\\theta(s) \\approx \\tilde{a}^{(2)} \\oslash f_\\theta\\left(a^{(1)} \\odot x^{(1)}\\right) +  \\tilde{a}^{(2)} \\oslash f_\\theta\\left(a^{(2)} \\odot x^{(1)}\\right) \\approx \\tilde{a}^{(2)} \\oslash f_\\theta\\left(a^{(1)} \\odot x^{(1)}\\right) + f_\\theta\\left(x^{(1)}\\right). $$\n\nAfter averaging the two expressions, we get\n\n$$ \\frac{1}{2}\\left( \\tilde{a}^{(1)} \\oslash f_\\theta(s)  + \\tilde{a}^{(2)} \\oslash f_\\theta(s) \\right) \\approx f_\\theta\\left(x^{(1)}\\right) + n$$\n\nwhere $n$ is a random noise vector and $f_\\theta\\left(x^{(1)}\\right)$ is approximated as an average of two predictions. Owing to the introduction of stochasticity by the binding and unbinding process these predictions are decorrelated, i.e., each superposition channel is processed to some degree differently.\n\n>“Are unbinding keys shared among all data or does each sample have a corresponding unbinding key?”\n\nThe unbinding keys are independent of data and depend only on the index of the superposition channel.\n\n>“[Limitations] Performance significantly decreases when a large number of samples are stacked.”\n\nAs described above, the performance decrease of MIMOConv is significantly less pronounced than that of the SOTA. But it is true that there is a limit to the number of superposition channels that can be employed, mostly depending on the size of the hidden dimension.\n\n>“[Limitations] It remains uncertain whether similar performance can be maintained when applying the method to a wider range of structures.”\n\nCNNs and Transformers are the most used DNN architectures. MLPs are used in superposition within our Transformers. To further address the reviewers concerns, in Table R1 we demonstrate our method for attention to not be restricted to FAVOR+, but instead to be widely applicable to other linear transformers such as DPFP from Schlag et al. “Linear transformers are secretly fast weight programmers\". Finally, the theoretical arguments are even more general and explored in more details in the Appendix (e.g., A.2, A.3.)."}, {"review_id": "arR5ZFXx8S", "reviewer": "Reviewer_3r5d", "summary": "This study introduces Multiple-Input-Multiple-Output Neural Networks (MIMONets) that can process multiple inputs simultaneously, reducing computational cost. Two types of MIMONets, MIMOConv for CNNs and MIMOFormer for Transformers, are presented. MIMOConv can handle multiple image inputs with minimal accuracy loss, while MIMOFormer effectively calculates attention scores for two concurrent inputs. These models offer a dynamic balance between accuracy and processing speed, using a fixed set of parameters.", "questions": "1. Can the MIMONets approach be generalized to other types of neural network architectures beyond CNNs and Transformers?", "limitations": "Not applicable.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "MIMONets significantly improve the processing speed by handling multiple inputs simultaneously, reducing the computational cost per input. hey can be applied to various neural network architectures, including CNNs and Transformers.", "weaknesses": "1. The concept of superposition can lead to interference between inputs, which may affect the model's accuracy.\n2. The integration of variable binding mechanisms and transformations for holistic processing may increase the complexity of the model, potentially making it harder to implement and understand.\n3. While the speed of processing is improved, there is a noted drop in accuracy when handling multiple inputs, which may not be suitable for applications requiring very high precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time and feedback. However, it is unclear how the sparse set of stated weaknesses led to a reject decision. In particular, weakness 2 applies to most innovations that are yet to be established. Weaknesses 1 and 3 coincide, are thoroughly addressed in the paper (for instance, Section 5.1 and Appendix E), and do not apply to the developed central use case of dynamic inference. We have responded to your remarks in more detail below and would appreciate an increased rating or further discussion.\n\n>“[Weakness 1] The concept of superposition can lead to interference between inputs, which may affect the model's accuracy.”\n\nAlthough this is true, it is rather a limitation of compute-in-superposition than a weakness of the paper. We bring up the issue of interference at multiple points and in fact come up with mitigation techniques such as isometry regularization and high hidden dimension. Specifically, the ablation study in Section 5.1 (with additional experiments and insights in Appendix E) explores such mitigation techniques to suppress the emergence of interference. Finally, to avoid a trade-off between accuracy and speedup we developed the concept of dynamic inference. In particular, we showed that one can enable a model to compute more quickly (via superposition) while still retaining the usual accuracy in the slow mode.\n\n>“[Weakness 3] While the speed of processing is improved, there is a noted drop in accuracy when handling multiple inputs, which may not be suitable for applications requiring very high precision.”\n\nAlthough a drop in accuracy is discernible, our method clearly improves on the state of the art Murahari, Vishvak, et al. “DataMUX: Data multiplexing for neural networks”, which was awarded second place in the 2022 Bell Lab Prize. Indeed, our comparison on MNIST, two synthetic language tasks, and a subtask of LRA (see Figure R1 and Table R1) shows the following: for high superpositions (16x) our CNN method outperforms theirs 80.4% to 52.9% on MNIST (the only vision dataset they report on) while being computationally cheaper, our Transformer (2x) outperforms theirs on ListOps 38.08% to 30.54%, and our method does not fail on synthetic language benchmarks which require faithful attention (scoring 96.52% and 99.40%) while DataMUX does (scoring 20.04% and 6.06%). Finally, as is quantified in Table 1 of the paper for dynamic inference, our method enables a single model with fixed parameters to be run at different accuracy-throughput operating points. In particular, at normal speed (N=1), the model performs as accurately as the baseline. This is a unique case where one obtains essentially free lunch, high accuracy and high throughput are guaranteed and can be balanced at will. Since there is no viable alternative to obtain such instantaneous switching between accuracy-throughput operating points within a fixed set of model parameters, and since we outperform state of the art, we do not see the observed drop in accuracy as a dealbreaker.\n\n>“[Weakness 2] The integration of variable binding mechanisms and transformations for holistic processing may increase the complexity of the model, potentially making it harder to implement and understand.”\n\nIn terms of computational complexity, the integration of variable binding mechanisms via binding and unbinding operations is inconsequential, amounting to 0.008%--0.031% of the total MACs for MIMOConv and 0.06%--0.14% for MIMOFormer (see Table R2 and Table R3 in the pdf).\n\nRegarding understandability of variable binding, in fact we chose to go with some of the most well-defined binding/unbinding operations (see Table A1 in the paper) of vector-symbolic architectures [8--10]. Vector-symbolic-architectures propose frameworks for constructing symbolic data structures through key-value binding of distributed representations. The construction of this data structure is transparent thanks to the use of explicit binding/unbinding operations with well-defined properties. These properties allow us to compose neural representations on-the-fly as opposed to learning how to compose them from scratch. While it is less well explored to process the resulting data structure using **nonlinear** neural transformations, we believe the choice of well-established binding/unbinding operations leads to a more transparent architecture by design.\n\n>“[Question] Can the MIMONets approach be generalized to other types of neural network architectures beyond CNNs and Transformers?”\n\nCNNs and Transformers are the most used DNN architectures. Furthermore, in Table 2 of the paper we report both on Transformers with superposition applied to MLPs (att.+MLP) and without MLPs in superposition (att.). Also, CNNs are essentially constrained MLPs with weight-sharing and limited connectivity. Since unconstrained MLPs do not require the locality principles of CNNs, binding mechanisms in much higher-dimensional space can be employed making it easier by decreasing interference via the Blessing of Dimensionality. To further convince the reviewer of the wide applicability of our method, we demonstrate in Table R1 of the pdf that our proposed method of superposition for attention is not restricted to FAVOR+ from the Performer, but instead widely applicable to other linear transformers such as DPFP (Deterministic Parameter-Free Projection) from I. Schlag et al. “Linear transformers are secretly fast weight programmers\". To summarise, we have shown our method to work for CNNs, MLPs, and Transformers. Although we do not provide empirical results on other structures such as RNNs and Graph Neural Networks, the theoretical arguments are general and explored in more details in the Appendix (e.g., A.2, A.3.)."}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. We are encouraged that (b2eJ) found our method well-written and well-grounded in previous works. We are pleased that (b2eJ, A964) appreciated our new theoretical analysis and experimental evaluation. We are glad that (zbz3, 3r5d) agree with the versatility of our approach and that (b2eJ, zbz3) assess our results as promising, particularly for increasing throughput as noted by (zbz3, 3r5d). We appreciate that (b2eJ, zbz3) share our assessment that new applications are enabled by our work. In the following, we provide answers to the reviewer’s comments, which will be reflected in a revised version of the paper.

Regarding applications, we would like to remind the reviewers of the central innovation termed “*dynamic inference*”, where one can select on-the-fly an operating point of a given accuracy & throughput. As such, our method is only inadequately comparable with “*static*” approaches that opt for a fixed performance point like model downsizing, quantization aware training, and pruning. Furthermore, these other methods can be added on top.

To address questions by (b2eJ, A964, zbz3) we will add several clarifying paragraphs to the paper. Also, as (b2eJ) requested, we now include two Tables (R2, R3) which indicate the complexity of sublayers in MIMOConv and MIMOFormer respectively. Fitting the suggestion of (A964, zbz3) we conducted experiments on two additional datasets: MNIST and a synthetic language task. Finally, at the remark of (A964) we now include a dedicated limitations list.

We particularly thank (b2eJ) for pointing out the NeurIPS 2022 paper Murahari, Vishvak, et al. “DataMUX: Data multiplexing for neural networks”, awarded second place in the 2022 Bell Lab Prize. Shortly after submission we also discovered this work in the context of a patentability search. While there are some similarities between their methods and ours (e.g., both use Hadamard binding and unbind before the final readout layer), there are fundamental differences that distinguish them qualitatively and quantitatively. These are discussed in the following two points.

1\) As (b2eJ) puts it: “the CNN part of the MIMONet work seems to give better empirical results”. To strengthen this point, we conducted a direct comparison on the MNIST benchmark for which DataMUX was optimized, and report on the findings in Figure R1 of the pdf. Even with a trivial downsizing for fair comparison from a 28-layer very-wide (10x) MIMOConv to a 10-layer narrow (1x) MIMOConv, we scale much better to high superposition channels (N) than DataMUX does. Indeed, our model shows an accuracy of 80.4% against their 52.9% in case of N=16 superposition channels (highest #channels reported by DataMUX for vision tasks), despite being computationally cheaper (0.47 MMAC/s vs. 0.65 MMAC/s).

We attribute the improved performance to a set of innovations which we reiterate here: MIMOConv applies *position-wise binding*, thus retaining the locality property present in natural images and vital for CNNs, whilst as discussed by Murahari, Vishvak, et al. their primary binding does not. As a workaround they proposed binding via two layers CNNs each outputting 8 feature maps. The resulting (pixel-wise) superposition in a low-dimensional space (8-D) leads to high interference. Additionally to using an expensive binding mechanism, it also makes the first layer of the model 8 times as expensive no matter the number of superpositions. We are able to circumvent this issue by applying the first layer of the CNN *before* the pixel-wise binding, increasing the dimensionality of each pixel in an easy-to-understand manner. Another contribution is the use of *isometric neural networks* to further reduce interference during the processing of superposed images. 

2\) Regarding Transformers, new experiments on LRA show that we outperform the Transformer variant of DataMUX on ListOps (38.08% vs. 30.54% accuracy) using models of similar size. Contrary to us, DataMUX blurs attention scores by sharing them between superposed sentences, and the introduction of additional global “sentence summary” tokens (w/o superposition) limits their approach to instances where imprecise “summarizing” information suffices. Notably, none of the tasks (token-level and sentence-level classification) they chose to report on requires attention layers at all; this is also discussed in M. Hassid et al. “How Much Does Attention Actually Attend”. As our experiments confirm (see Table R1), on more nuanced tasks in NLP like “associative recall” and “induction head”, their method drops to 20.04% and 6.06% for N=2, while ours, at a score of  96.52% and 99.40% respectively, succeeds. Despite investing significant efforts in the training of DataMUX, it cannot perform on these synthetic tasks. This is in line with the findings of Y. Fu, Dao, et al. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models” which identify the lack of attention as the reason that the Structured State Space Sequence (S4) model is able to completely outperform state of the art in LRA, but is not feasible for large language models. In contrast to DataMUX, our work approximates true attention and our theoretical derivations *show convergence to actual dot-product attention* as the hidden dimension increases, giving us an even stronger case for applicability to large language models (for instance, GPT-3 uses embedding dimension 12,888, far exceeding the maximum of 512 we report on). 

Finally, note that Linearized Transformers were able to decrease the complexity of dot-product attention from $O(L^2)$ to $O(L)$. Our MIMO-style is compatible with them and thus reduces the complexity from $O(L)$ further to $O(L/N)$. It achieves this while retaining the property that, in the limit of large embedding size, it converges *precisely* to quadratic attention.

We again thank the reviewers for their comments and look forward to entering into a more detailed discussion.

Meta-review: The paper introduces the concept of Multiple-Input-Multiple-Output Neural Networks (MIMONets) that simultaneously process multiple inputs, resulting in a reduced computational burden. Two variants, MIMOConv (for CNNs) and MIMOFormer (for Transformers), are detailed, demonstrating the utility of the approach in both architectures. Empirical evaluations indicate that MIMONets can achieve notable speedups while preserving acceptable accuracies.

### Strengths:
* The idea of processing multiple inputs concurrently in order to significantly reduce computational cost during batching is interesting.
* The method's applicability to both CNN-based and Transformer-based structures demonstrates its versatility.
* Empirical evaluations provide insights into the trade-offs between computational efficiency and accuracy.

### Weaknesses:
* The paper lacks adequate comparison to related works like DataMUX and lacks proper reference to other foundational works based on MIMO (multiple input multiple output). See also [1] which speeds up ensembles via MIMO.
* Limited experimental datasets have been used such as only up to CIFAR100, narrowing the scope of the validation.
* Notable drop in accuracy is observed when handling multiple samples due to interference in superposition. This particularly happens with the stacking of 4 samples.
* Some issues in presentation clarity, specifically concerning the tables detailing the FLOPS/sample, and more information is desired about bind/unbind operations and their computational complexity.

Given the innovation and potential of MIMONets, juxtaposed against the cited weaknesses, especially around comparisons and detailed evaluations, I recommend accept. However, the authors should consider incorporating the feedback, particularly the comparison with relevant prior work, to strengthen their paper.

[1]: Training independent subnetworks for robust prediction. ICLR 2021. https://arxiv.org/abs/2010.06610
Decision: Accept (poster)