Title: MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computation in superposition

Vector-symbolic architectures

Convolutional neural networks

Transformers
Abstract: With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition. To reduce the computational burden per input, we propose Multiple-Input-Multiple-Output Neural Networks (MIMONets) capable of handling many inputs at once. MIMONets augment various deep neural network architectures with variable binding mechanisms to represent an arbitrary number of inputs in a compositional data structure via fixed-width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations to process the data structure holistically, leading to a speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, an unbinding mechanism recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by an instantaneous on-demand switching between a set of accuracy-throughput operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOConv and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves $\approx 2$–$4\times$ speedup at an accuracy delta within [+0.68, -3.18]% compared to WideResNet CNNs on CIFAR10 and CIFAR100.  
Similarly, MIMOFormer can handle $2$–$4$ inputs at once while maintaining a high average accuracy within a [-1.07, -3.43]% delta on the long range arena benchmark. 
Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer. Our code is available at https://github.com/IBM/multiple-input-multiple-output-nets.

MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition

Abstract

With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition . To reduce the computational burden per input, we propose Multiple-Input-MultipleOutput Neural Networks (MIMONets) capable of handling many inputs at once. MIMONets augment various deep neural network architectures with variable binding mechanisms to represent an arbitrary number of inputs in a compositional data structure via fixed-width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations to process the data structure holistically, leading to a speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, an unbinding mechanism recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by an instantaneous on-demand switching between a set of accuracy-throughput operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOConv and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves ≈ 2 -4 × speedup at an accuracy delta within [+0 . 68 , -3 . 18]% compared to WideResNet CNNs on CIFAR10 and CIFAR100. Similarly, MIMOFormer can handle 2-4 inputs at once while maintaining a high average accuracy within a [ -1 . 07 , -3 . 43]% delta on the long range arena benchmark. Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer. Our code is available at https://github.com/IBM/multiple-input-multiple-output-nets .

1 Introduction

Driven by the successes of deep learning in image and natural language processing tasks, increasingly large neural network models have been developed to reach state-of-the-art performance [1-4]. These large models, however, increase computational complexity in terms of operation count for every event of input processing. One viable option to reduce the computational cost of processing per input is to create a compositional data structure where a variable number of input items (i.e., values) can be bound to corresponding protection keys, creating key-value pairs that can coexist and be processed

concurrently. This variable-sized data structure can be represented by fixed-width distributed representations in vector-symbolic architectures (VSAs) [5-7]. In VSAs, the composition of different items in the data structure is based on functional compositionality (i.e., key-value binding), which yields a dimensionality-preserving distributed representation, rather than concatenative compositionality. Interestingly, the resulting fixed-width distributed data structure can be transformed by a one-time application of a function, whereby all input items are jointly transformed, leading to holistic transformation or computation in superposition [8-10]. This concept of computation in superposition can reduce the effective number of operations per input by a factor of the number of input items in the data structure, because the function is applied to the data structure holistically without decomposing the constituent items for individual transformations. However, processing the VSA data structure via computation in superposition has so far been limited to linear maps [8-10].

Motivated by these observations, we make the following contributions:

We introduce a principled and transparent approach to Multiple-Input-Multiple-Output Neural Networks (MIMONets) based on VSA, enabling computation in superposition for highly nonlinear transformations in neural networks (Section 2). The MIMONets concept can be applied to various architectures, embracing the rich capacity provided by increasingly large models. The resulting network can handle many inputs at once, thus reducing the computational cost per input. We describe and overcome the challenges of computation in superposition, which originates from nonlinear interference of inputs in separate superposition channels.

We propose MIMOConv, a realization of MIMONets for deep convolutional neural network (CNN) architectures (Section 3). We provide several strategies for mitigating interference between different superposition channels, including a novel locality-preserving binding operation (PWHRR) and isometry-inducing regularization. Empirical evaluations on CIFAR10 and CIFAR100 show that a MIMOConv built on a WideResNet-28-10 [11] can process concurrently two inputs in superposition ( ≈ 2 × speedup) even with a slightly higher accuracy ( 0 . 11 -0 . 68% gain), and four inputs ( ≈ 4 × speedup) with a marginal drop ( 1 . 24 -3 . 18% ) (see Section 5.1).

We further extend the concept of MIMONets to Transformer architectures, where the calculation of attention scores poses additional challenges for the computation in superposition paradigm. To this end, we propose MIMOFormer, which relies on a 2D grid binding scheme for computing attention in superposition (Section 4). We derive probabilistic tail bounds on the distortion caused by interchannel interference and show that our method converges to noise-free attention in the limit of high dimension. Our method succeeds empirically ( ≥ 96 . 52% accuracy) at synthetic sequence modelling tasks [12], while previous work [13] fails ( ≤ 20 . 04% ). We also provide evaluations on the long range arena (LRA) dataset [14] in Section 5.2 using a MIMOFormer that is based on the Performer architecture [15]. Compared to the Performer, MIMOFormer maintains a high average accuracy with a marginal drop of 1 . 07% and 3 . 43% when handling two and four inputs at once, respectively.

MIMONets allow a dynamic trade-off at inference time between accuracy and speed, i.e., they offer an instantaneous on-demand switching between accuracy-throughput operating points using a

single set of fixed parameters . Experimental results in Section 5.1 show that a dynamic MIMOConv can seamlessly operate at various modes ( ≈ 1 -4 × speedup) while maintaining a high accuracy compared to the best static models ( ≤ 1 . 82% drop).

2 MIMONets enabling Computation in Superposition

The central idea behind MIMONets (see Figure 1) is to simultaneously pass multiple inputs as a superposition through a nonlinear function f θ parameterized by neural network weights θ . We isolate the individual inputs into separate protected channels by binding them with protection keys resulting in a key-value data structure [5-7].

Concept. Assuming two inputs ( x (1) and x (2) ), which can be generic embeddings either from images or natural language, we define a unique high-dimensional key ( a (1) and a (2) ) for each protected channel, drawn randomly at initialization. Owing to the Blessing of Dimensionality, randomly drawn high-dimensional vectors are quasi-orthogonal with high probability (see Appendix A). Consequently, binding ( ⊙ ) the inputs with these keys yields quasi-orthogonal key-value pairs ( x (1) ⊙ a (1) and x (2) ⊙ a (2) ), which enables one to superpose the pairs with low interference:

As discussed in Appendix A, s admits a noisy retrieval of x (1) and x (2) through unbinding:

To accelerate computing, inspired by the above-mentioned noisy retrieval, we pass the superposition s through a nonlinear function f θ with parameters θ , such as a neural network, before retrieval. The quasi-orthogonality of the bound inputs allows processing each in a separate protected subspace-all with a single function call. To be able to recover the first processed sample f θ ( x (1) ) from f θ ( s ) , we aim to find an unbinding key ˜ a (1) for which

The first approximation holds exact for linear f θ . As discussed in Section 3, a nonlinear f θ can still be encouraged to allow such an approximation through appropriate weight regularization techniques and well-suited activation functions. Further, by optimizing over unbinding keys ( ˜ a ( i ) ), the second estimation (Eq. (4)) can be achieved. Consequently, matching binding and unbinding keys ( a ( i ) and ˜ a ( i ) ) that confirm the approximation (Eq. (3) and (4)) set up a protected channel through the nonlinear function f θ ( s ) . Appendix A lists the design choices of the adopted VSA, which define the operations of key-value binding and unbinding, for all MIMONet variants presented in this work. In the case of image embeddings, we use circular convolution [6] for binding and Matrix Binding of Additive Terms (MBAT) [16] for unbinding. In the case of sequence tokens, we bind and unbind using the Hadamard product [17]. Binding and unbinding keys are always data-independent, i.e., they depend only on the index of the protected channel. See [18] for alternative binding and unbinding options.

Dynamic Inference. Setting up N protected channels through a neural network f θ gives almost a speedup of N × due to most computations taking place in superposition. However, as is explored empirically, increasing N adds inter-channel noise leading to a decrease in predictive accuracy. If a fixed trade-off is unsatisfactory, one can build a dynamic model capable of running a superposition of one up to N different inputs. By inserting the same input into multiple channels and averaging the output, one effectively forms an in-network ensemble, similar to [19,20]. Using all protected channels for different inputs leads to a fast but less accurate model, whereas using all protected channels for the same input yields a slower but accurate ensemble model. By partitioning the superposition channels on demand, arbitrary configurations in between may be reached. Note that our method can instantaneously adapt to the current computational demand, without loading different model weights from the main memory. To perform across slow and fast configurations, the model should randomly switch between them during training. See Appendix A for a more detailed explanation.

3 MIMOConv

This section presents how the MIMONets concept, introduced in Section 2, can be applied to construct a multiple-input-multiple-output CNN (MIMOConv) capable of simultaneously processing multiple images in superposition. The MIMOConv architecture is shown in Figure 2. Multiple input samples ( N ) are passed through the first convolutional layer, bound with a unique high-dimensional key based on Holographic Reduced Representations (HRR) [6], and superposed by element-wise addition. After passing the superposed tensors through the network's main CNN layers, we obtain a combined feature vector with information on all inputs. By unbinding with separately learned keys based on MBAT [16], which amounts to a matrix multiplication, we extract the individual processed information, which is then passed through a final fully-connected layer to produce logits for classification. In the following, we introduce our three main contributions that lead to a highly accurate MIMOConv.

Augmenting CNNs with locality-preserving variable bindings. We embrace a principled and transparent binding mechanism from HRR. Accordingly, binding is performed using circular convolution with a binding key of dimension D , drawn from an i.i.d. Gaussian distribution with zero mean and 1 /D variance. Instead of convolving the flattened image tensor, we repeatedly apply circular convolution between the binding key and each pixel volume spanning across the feature maps ( D × 1 × 1 ). This binding operation, which we call position-wise HRR (PWHRR), is translation equivariant and maintains locality, an essential property for subsequent layers with limited receptive fields. More concretely, binding and unbinding are performed as

with image tensors x ( k ) ∈ R D × W × H , hidden representation h ∈ R D ′ × W × H , binding key a ( k ) ∈ R D and unbinding key ˜ a ( k ) ∈ R D ′ × D ′ . Here, D,D ′ , W, and H denote the hidden dimension at binding, the hidden dimension at unbinding (generally differs from D ), image width, and image height, respectively. ∗ is the circular convolution, · the matrix multiplication, and k indexes the superposition channel. Unbinding is applied after the global (average) pooling to reduce computational costs. The binding keys can be either learned or fixed during training (see ablation study in Appendix E).

Embracing high dimensional embeddings. According to the Blessing of Dimensionality (see Appendix A), random vectors quickly become quasi-orthogonal as their dimension increases. To reduce interference between protected channels, we increase the number of feature maps by adopting Wide Residual Networks [11], the most commonly used CNN architecture to achieve state-of-the-art accuracy on CIFAR100 [21]. The input tensors are passed individually through the first convolutional layer before being superposed in a suitably high dimensional space. We set the number of feature maps after the first convolutional layer to D =64. This is 4 × more than the standard Wide-ResNet-28 [11], which results in improved training stability at a marginally higher compute cost (see Section 5.1).

Encouraging isometric layers. We aim to preserve the quasi-orthogonality of our protected channels as the superposition is passed through many network layers. To that end, residual connections are used and each subfunction g θ of type (strided) spatial convolution or activation function is made approximately inner-product preserving, i.e.,

Inspired by [22], we deploy a regularization to the CNN weights and use a parametric ReLU [23] activation function, a learnable affine combination between identity and ReLU. Those adjustments lead to a near-isometric network. The regularization for the individual CNN layers is determined by

with γ as hyperparameter. We use L ( W ) if C i > C o , else L ( W T ) . See Appendix B for more details.

4 MIMOFormer

This section presents MIMOFormer, which applies the principles of computation in superposition to dot-product self-attention [24]. Figure 3 shows a MIMOFormer layer with four protected channels, consisting of a single-head 2 attention block, a concatenation, a linear layer, an MLP, and a skip connection.

Merely superposing protected attention keys 3 and queries does not yield the desired result. As discussed in Appendix F, with scalar attention scores between pairs of tokens, vanilla dotproduct attention irreversibly combines attention scores of separate protected channels, effectively blurring the attention weights. By building instead on linear Transformers [15] [25], attention scores are not collapsed to scalars, thus enabling computation in superposition.

Despite being compatible with other linear transformers (such as DPFP [25]), for concreteness we discuss changes to the Performer's FAVOR+ attention block [15]. Enabling computation in superposition, we label the block as FA VOR+S.

Given attention keys ( k j ) L j =1 , queries ( q i ) L i =1 , and values ( v j ) L j =1 , FAVOR+ estimates dotproduct attention at sequence index i through

where ϕ : R D → R R + approximates the softmax kernel exp( ⟨ k j , q i ⟩ / √ D ) as an explicit inner product. Since computing ϕ has a computational complexity of O ( DR ) , the construction of A ∈ R D × R takes O ( LDR ) . Equally, multiplying A × ϕ ( q i ) ∀ i takes O ( LDR ) . Thus, FAVOR+ breaks the quadratic dependence on sequence length L of attention. The denominator B i is discussed in Appendix C.

Our extension, FAVOR+S, computes attention in superposition, yielding a square-root speedup in the number of protected channels. It encodes them in an M × N grid, distributing the computational burden among the setup of the value-key matrix A and its product with ϕ ( q i ) . Importantly, in the limit of high dimensional projections D our mechanism converges to exact self-attention, completely separating the protected channels from one another. In the following, we assume the token values v j , keys k j , and queries q j to be in protected subspaces:

where a ( m,n ) are i.i.d. bipolar vectors of Rademachers [17] and ( m,n ) denotes a channel.

MIMOFormer benefits from the low time complexity ( O ( D ) ) of the Hadamard product, especially since binding and unbinding are performed in every MIMOFormer layer. Our derivations rely on two estimates:

The approximation P , which improves with increasing R = dim( ϕ ( q i )) , is due to FA VOR+ and is quantified in [15]. On the other hand, the approximation H follows from:

Inter-channel distortion. The probability that inter-channel attention distorts the intended signal of the dot-product by a factor outside [1 -α, 1 + α ] has various upper bounds, most notably decaying exponentially w.r.t. Dα 2 cos 2 ( ∡ ( k ( u,n ) j , q ( u,n ) i )) / ( NM -1) 2 . See Appendix D for the full theorem.

4.1 FAVOR+S: Computing self-attention in superposition

We first discuss separately how to use a one-dimensional grid to carry out either the multiplication ( × ) or the construction of A in superposition. Finally, the integration into a 2D grid will be shown.

Placing multiple queries in superposition. We set up channels 1 , . . . , M by simultaneously generating a superposition in the construction of A s and of the queries to be applied. To avoid inter-channel attention we superpose value-key tensor products, i.e., we do not construct tensor products between superposed values and superposed keys:

We obtain a superposition of bound output values; hence, the cost of computing A × ϕ ( q i ) for all i is amortized across channels. However, the construction complexity of A s is increased M -fold to O ( LDR · M ) , hence the complexity per protected channel remains at O ( LDR ) .

Constructing value-key tensor products in superposition. Next, we demonstrate a value-key tensor product ( A s ) shared across all channels, but with a mere O ( LD ( R + N )) setup complexity. In O ( LND ) we compute ∑ N q =1 v ( q ) j and ∑ N w =1 k ( w ) j for all j . These are then (re)used to build A s .

̸

The output contains the n th channel together with noise. The operation A s × ϕ ( q ( n ) i ) , which takes O ( LDR ) , must be repeated N times to produce outputs for all N channels, causing a bottleneck.

Simultaneous superposition of queries and value-key tensor products using a 2D grid. Finally, we combine the two previously described approaches to encode the superposition channels in a 2D grid of size N × M . We multiply a constant matrix ( A s ) with features derived from a superposition of queries to get the superposition vector S ( n ) i

Computing the multiplication ×∀ i, n takes O ( LNDR ) . If we set M = N , we can evaluate S ( n ) i ∀ i, n using only O ( LNDR + LN 2 D ) instead of the usual O ( LDR · N 2 ) . Thus, one may achieve a speedup of O (min( √ N 2 , R )) compared to FAVOR+. Since R is normally in the hundreds [15], we can assume improvements of O ( √ N 2 ) for reasonably large N 2 = M · N . Eq. (19) simplifies to:

̸

4.2 Integrating FAVOR+S into MIMOFormer

As is apparent in Eq. (19), the query superposition is along a different axis ( M ) than the key and value superpositions ( N ). The output of attention, however, is only superposed along a single axis ( M ). To be able to set up superpositions along both axes ( M and N ) at the next layer, we require all channels (i.e., keys, queries, and values) in separation, i.e., not superposed, at the interface between MIMOFormer layers. We present two variants of MIMOFormer with different speedups.

The first computes in superposition exclusively during the attention mechanism. The individual tokens of the channel ( n, m ) are directly retrieved from S ( n ) i by unbinding with the key ˜ a ( n,m ) = a ( n,m ) , and the remaining computational steps within FAVOR+S are performed separately.

The second (and faster) MIMOFormer instance additionally performs the concatenation, the linear layer, as well as the MLP in superposition (shown in Figure 3). Unlike in the first variant, the skip connection around the attention block must account for the introduced superposition. To allow a potential embedding dimension mismatch, we instantiate two different sets of randomly drawn bipolar keys: one for the skip connection and post-MLP unbinding, and one for FAVOR+S binding. All keys are frozen during training; it is up to the trainable weights in the linear layer after concatenation to find the relationship between the binding and unbinding keys.

The function ϕ in the self-attention block consists of an R =256 dimensional projection and a ReLU activation [15]. Appendix C provides theoretical justification for using ReLU in ϕ and its benefits for MIMOFormer. Empirically, ReLU shows better numerical stability than unbiased softmax FAVOR+.

5 Empirical Results

This section evaluates the proposed MIMONets on various model architectures and benchmarks. Appendix E and Appendix F describe the experimental setup for MIMOConv and MIMONets, respectively. All experiments are repeated five times with a different random seed. We report the mean and standard deviations of accuracy to account for variability in training.

5.1 MIMOConv

CIFAR10 and CIFAR100. Our main baseline, which we adapt to MIMOConv, is a reproduced WideResNet-28-10 [11], i.e., a 28-layer CNN with the number of feature maps of each convolutional layer enlarged by 10 × . Moreover, as a stronger baseline, we include the isometry regularization in the training of WideResNet-28-10, and call the resulting network WideIsoNet-28-10. Both baselines demand 5.251 GMACs (Giga multiply-accumulate operations) per sample. MIMOConv's inference complexity per sample is 5 . 335 GMACs for N = 1 , 2 . 667 GMACs for N = 2 , and 1 . 334 GMACs for N = 4 ; hence, we get a ≈ N × speedup despite not accelerating the first and last layer. See Appendix E.

Table 1 shows the accuracy of the static MIMOConv, which is exclusively trained to support either 1, 2, or 4 channels. The MIMOConv with one channel ( N =1) outperforms both baselines, which may be attributed to regularizing effects of the key-value binding. MIMOConv with N = 2 channels still outperforms WideResNet-28-10, while reducing the inference complexity by 2 × . The complexity can be further reduced by increasing the number of superpositions to N = 4 at a slight accuracy drop of ≤ 3.18%, compared to WideResNet-28-10.

Next, we evaluate the dynamic partitioning of the superposition channels to select a speed-accuracy operating point instantaneously, which is a main feature of our approach and sets it apart from other static approaches that opt for a fixed performance point like model downsizing, quantization aware training, and pruning (see Appendix A for a discussion). We set up a model with four channels, but evaluate its performance in different configurations: a fast (4 inputs/pass), a normal (2 inputs/pass), and a slow mode (1 input/pass). The fast mode maps each input to one channel; the medium mode distributes two inputs over pairs of channels; and the slow mode uses all channels for the same input. The models are trained on 80% of the batches in fast mode and on 20% of the batches in slow mode. Appendix E provides more details on the trade-off between fast and slow mode training. As Table 1 shows, a single dynamic model can seamlessly switch between operation points while maintaining a high accuracy compared to the static models ( ≤ 1 . 82% drop).

The detailed ablation study in Appendix E makes the following findings: (1) isometry regularization improves accuracy for any number of channels; (2) training MIMOConv for more epochs closes the performance gap to the single-input baseline; (3) an appropriate number of feature maps (32 or 64) in the first layer stabilizes training; and (4) the binding keys can be frozen during training without loss (whereas unbinding keys are never frozen).

MNISTandSVHN. Figure 4 compares MIMOConv with DataMux [13] on the MNIST dataset. Even with a trivial downsizing for fair comparison from a 28-layer very-wide ( 10 × ) ResNet to a 10-layer narrow ( 1 × ) network, MIMOConv scales much better to high superposition channels ( N ) than DataMUX does. Indeed, our model shows an accuracy of 80 . 4% against their 52 . 9% in case of N = 16 superposition channels (highest number of channels reported by DataMUX for vision tasks), despite being computationally cheaper ( 0 . 47 MMAC/s vs. 0 . 65 MMAC/s). Also, DataMux's binding overhead results in a mere 1 . 35 × reduction in MACs compared to our 10 . 9 ×

as

N

goes from

1

to

16

. Ergo, our method scales better in accuracy and throughput as

N

increases.

1

2

4

8

16

Number of superposition channels (N)

60

70

80

90

100

Accuracy (%)

MIMOConv-10-1

DataMUX: CNN+Nonlinear (8x)

Finally, we tested MIMOConv on the SVHN dataset. Despite limited hyperparameter tuning, MIMOConv achieves a high accuracy of 97 . 17% ( N =1), and can maintain its performance with larger superpositions ( 97 . 05% and 96 . 84% for N = 2 and N = 4 , respectively).

5.2 MIMOFormer

LRA. We evaluate MIMOFormer on five tasks from LRA [14], and compare against the vanilla Transformer [24] and the Performer [15] using FAVOR+ attention with ReLU projection. Moreover, we also consider wide Transformer variants [26], consisting of only one layer but as many heads as their deep counterparts. Task-specific architectures and training hyperparameters are kept the same for the Performer and the MIMOFormer (see Appendix F).

Owing to an improved training setup, our replicated deep and wide Performer baselines substantially outperform the results reported in [14] (see Table 2). Moreover, MIMOFormer enables accurate computation in superposition for both deep and wide attention models. The performance drop is less pronounced in wide models (only 1 . 07% drop compared to Performer with N = 2 , att.), which may be attributed to the larger number of heads, increasing the effective dimension ( D tot = H · D head ).

When computing both attention and the MLP in superposition (att.+MLP), we observe better scaling (in N ) for wide models. Also, MIMOFormer reduces the gap to the baseline as the number of epochs increases (see Appendix F).

To stabilize training in the case of N = 4 , we implemented a curriculum training procedure where the number of superpositions is reduced to N '= N/ 2 during a warmup phase (1/6th of the training steps), improving the average accuracy of MIMOFormer in both wide and deep models.

Comparing against a reproduced DataMUX [13], MIMOFormer (att.) outperforms it on ListOps (38.08% vs. 30.54% accuracy) when using models of similar size and N =2, see Appendix F.

Synthetic sequence modeling. Table 3 reports the accuracy on two synthetic sequence modeling tasks, which Transformer alternatives such as S4 [27] have difficulties solving [12]. On these more nuanced NLP tasks, the accuracy of DataMUX [13] drops to 20 . 04% and 6 . 06% for N = 2 despite significant efforts in training, while MIMOFormer, at a score

of 96 . 52% and 99 . 40% respectively, succeeds. We attribute this difference in performance to attention score blurring in DataMux, discussed in Appendix F. Contrastingly, our method converges to exact attention without blurring. It is versatile and can be adjusted to other linear Transformers such as DPFP [25], achieving a score of 93 . 64% and 98 . 56% .

6 Related Work

So far, superposition principles have been applied in order to store the weights of multiple models in a single neural network [28-30], to circumvent catastrophic forgetting in continual learning [31,32], and to render symbolic reasoning tractable [33]. To address privacy concerns when running remote inference, single inputs were bound with random channels to implement pseudo-encryption [34]. Recently in [35], HRR was used to define an unconventional version of self-attention, whose attention scores are processed to a diagonal matrix. The value vectors are scaled according to their importance in the sequence instead of being combined in a weighted sum. In contrast to us, none of these works superpose multiple inputs into a data structure to speed up computation.

In [19, 20], an ensemble of CNN models was fit into one network. However, by only broadcasting a single input over the channels and by averaging all the outputs, this approach collapses to a single-input-single-output (SISO) network. On the contrary, we explore using protected channels for different inputs at inference, resulting in an actual multiple-input-multiple-output (MIMO) network.

There has also been a line of work to accelerate Transformers using inputs in superposition [13] [36]. DataMux [13] claims to retain high performance for language understanding tasks, even when using up to 40 inputs in superposition. However, none of the reported tasks require attention layers at all [37]. In Section 5.2 we show failure of their method when actual attention is required (see also Appendix F). MUX-PLMs [36] improves on DataMux with contextual binding and replaces token prefixes with unbinding keys, but does not address the blurry attention mechanism. In contrast to DataMUX and MUX-PLMs, our work approximates true attention and our theoretical derivations show convergence to actual dot-product attention as the dimension of attention projections increases, giving us an even stronger case for applicability to large language models.

7 Conclusion

We present MIMONets that simultaneously process multiple inputs by performing computation in superposition. MIMONets bind arbitrary inputs with high-dimensional keys, which projects them to orthogonal subspaces that, together with near-isometric subfunctions, guarantee low interference through all nonlinear layers. Unbinding with (learned) keys can safely retrieve information on individual channels. We provide two MIMONets instances, MIMOConv and MIMOFormer, that show the effectiveness of computation in superposition through two dominant operations in neural network architectures: convolution and attention. Further investigations could explore the MIMOcapability of architectures that contain additional nonlinearities (e.g., max-pooling) and use different input modalities. MIMONets could be suitable candidates to accelerate dynamically and on-demand the inference of foundation models [38].

References

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, pp. 1877-1901, 2020.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021, pp. 10 012-10 022.

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, 'A convnet for the 2020s,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 11 976-11 986.

R. W. Gayler, 'Vector symbolic architectures answer Jackendoff's challenges for cognitive neuroscience,' in Joint International Conference on Cognitive Science (ICCS/ASCS) , 2003.

T. A. Plate, 'Holographic reduced representations,' IEEE Transactions on Neural Networks , vol. 6, no. 3, pp. 623-641, 1995.

P. Kanerva, 'Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors,' Cognitive Computation , vol. 1, no. 2, pp. 139-159, 2009.

J. Neumann, 'Learning holistic transformation of HRR from examples,' in International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies (KES) , 2000, pp. 557-560.

--, 'Learning the systematic transformation of holographic reduced representations,' Cognitive Systems Research , vol. 3, no. 2, pp. 227-235, 2002.

D. Kleyko, M. Davies, E. P. Frady, P. Kanerva, S. J. Kent, B. A. Olshausen, E. Osipov, J. M. Rabaey, D. A. Rachkovskij, A. Rahimi, and F. T. Sommer, 'Vector symbolic architectures as a computing framework for emerging hardware,' Proceedings of the IEEE , vol. 110, no. 10, pp. 1538-1571, 2022.

S. Zagoruyko and N. Komodakis, 'Wide residual networks,' in Proceedings of the British Machine Vision Conference (BMVC) . BMVA Press, September 2016, pp. 87.1-87.12.

D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re, 'Hungry hungry hippos: Towards language modeling with state space models,' in The Eleventh International Conference on Learning Representations (ICLR) , 2022.

V. Murahari, C. Jimenez, R. Yang, and K. Narasimhan, 'DataMUX: Data multiplexing for neural networks,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 35, pp. 17 515-17 527, 2022.

Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, 'Long range arena: A benchmark for efficient transformers,' in International Conference on Learning Representations (ICLR) , 2021.

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser et al. , 'Rethinking attention with performers,' in International Conference on Learning Representations (ICLR) , 2020.

S. I. Gallant and T. W. Okaywe, 'Representing objects, relations, and sequences,' Neural Computation , vol. 25, no. 8, pp. 2038-2078, 2013.

R. W. Gayler, 'Multiplicative binding, representation operators & analogy,' in Advances in Analogy Research: Integration of Theory and Data from the Cognitive, Computational, and Neural Sciences , 1998.

D. Kleyko, D. Rachkovskij, E. Osipov, and A. Rahimi, 'A survey on hyperdimensional computing aka vector symbolic architectures, part I: Models and data transformations,' ACMComputing Surveys , vol. 55, no. 6, 2022.

M. Havasi, R. Jenatton, S. Fort, J. Z. Liu, J. Snoek, B. Lakshminarayanan, A. M. Dai, and D. Tran, 'Training independent subnetworks for robust prediction,' in International Conference on Learning Representations (ICLR) , 2021.

A. Ramé, R. Sun, and M. Cord, 'Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR) , 2021, pp. 823-833.

A. Krizhevsky, 'Learning multiple layers of features from tiny images,' University of Toronto , 2009.

H. Qi, C. You, X. Wang, Y. Ma, and J. Malik, 'Deep isometric learning for visual recognition,' in International Conference on Machine Learning (ICML) . PMLR, 2020, pp. 7824-7835.

K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 1026-1034.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 30, 2017.

I. Schlag, K. Irie, and J. Schmidhuber, 'Linear transformers are secretly fast weight programmers,' in International Conference on Machine Learning (ICML) . PMLR, 2021, pp. 9355-9366.

J. R. Brown, Y. Zhao, I. Shumailov, and R. D. Mullins, 'Wide attention is the way forward for transformers?' in NeurIPS '22 Workshop on All Things Attention: Bridging Different Perspectives on Attention , 2022.

A. Gu, K. Goel, and C. Re, 'Efficiently modeling long sequences with structured state spaces,' in International Conference on Learning Representations (ICLR) , 2022.

B. Cheung, A. Terekhov, Y. Chen, P. Agrawal, and B. Olshausen, 'Superposition of many models into one,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 32, 2019.

M. Hersche, P. Rupp, L. Benini, and A. Rahimi, 'Compressing subject-specific brain-computer interface models into one model by superposition in hyperdimensional space,' in 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE) . IEEE, 2020, pp. 246-251.

M. Zeman, E. Osipov, and Z. Bosni´ c, 'Compressed superposition of neural networks for deep learning in edge computing,' in 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2021, pp. 1-8.

M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi, M. Rastegari, J. Yosinski, and A. Farhadi, 'Supermasks in superposition,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, pp. 15 173-15 184, 2020.

M. Zeman, J. F. Pucer, I. Kononenko, and Z. Bosni´ c, 'Superformer: Continual learning superposition method for text classification,' Neural Networks , vol. 161, pp. 418-436, 2023.

M. Hersche, M. Zeqiri, L. Benini, A. Sebastian, and A. Rahimi, 'A neuro-vector-symbolic architecture for solving Raven's progressive matrices,' Nature Machine Intelligence , vol. 5, no. 4, pp. 363-375, 2023.

M. M. Alam, E. Raff, T. Oates, and J. Holt, 'Deploying convolutional networks on untrusted platforms using 2D holographic reduced representations,' in International Conference on Machine Learning (ICML) . PMLR, 2022, pp. 367-393.

--, 'Recasting self-attention with holographic reduced representations,' in Proceedings of 8TH SIGKDD International Workshop on Mining and Learning from Time Series - Deep Forecasting: Models, Interpretability, and Applications (MiLeTS 2022) , 2022.

V. Murahari, A. Deshpande, C. Jimenez, I. Shafran, M. Wang, Y. Cao, and K. Narasimhan, 'Muxplms: Data multiplexing for high-throughput language models,' in Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023, pp. 4540-4554.

M. Hassid, H. Peng, D. Rotem, J. Kasai, I. Montero, N. A. Smith, and R. Schwartz, 'How much does attention actually attend? Questioning the importance of attention in pretrained transformers,' in Findings of the Association for Computational Linguistics: EMNLP 2022 , 2022, pp. 1403-1416.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , 'On the opportunities and risks of foundation models,' arXiv preprint arXiv:2108.07258 , 2021.

Peer reviews:
comments: SUMMARY:
This study introduces Multiple-Input-Multiple-Output Neural Networks (MIMONets) that can process multiple inputs simultaneously, reducing computational cost. Two types of MIMONets, MIMOConv for CNNs and MIMOFormer for Transformers, are presented. MIMOConv can handle multiple image inputs with minimal accuracy loss, while MIMOFormer effectively calculates attention scores for two concurrent inputs. These models offer a dynamic balance between accuracy and processing speed, using a fixed set of parameters.

STRENGTHS:
MIMONets significantly improve the processing speed by handling multiple inputs simultaneously, reducing the computational cost per input. hey can be applied to various neural network architectures, including CNNs and Transformers.

WEAKNESSES:
1. The concept of superposition can lead to interference between inputs, which may affect the model's accuracy.
2. The integration of variable binding mechanisms and transformations for holistic processing may increase the complexity of the model, potentially making it harder to implement and understand.
3. While the speed of processing is improved, there is a noted drop in accuracy when handling multiple inputs, which may not be suitable for applications requiring very high precision.

QUESTIONS:
1. Can the MIMONets approach be generalized to other types of neural network architectures beyond CNNs and Transformers?

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main content of the article is about MIMONets, which are multiple-input-multiple-output neural networks that exploit computation superposition. By using fixed-width distributed representations in vector-symbolic architectures, MIMONets can represent a variable number of inputs in a data structure and process them holistically with nonlinear neural transformations. This leads to a significant reduction in computational burden per input and offers a dynamic trade-off between accuracy and throughput. The article presents two instances of MIMONets (MIMOConv and MIMOFormer), which apply the concept of computation in superposition to convolutional neural network (CNN) and Transformer architectures, respectively. Empirical evaluations show that MIMONets achieve significant speedups while maintaining high accuracy.\n\nSTRENGTHS:\nStrengths:\n1. The method cleverly combines multiple inputs into a single sample for inference, reducing the computational cost. This is highly meaningful for practical applications.\n2. The method achieves promising results in both CNN-based and Transformer-based structures, indicating its versatility and applicability.\n\nWEAKNESSES:\nWeaknesses:\n1. The experimental datasets and network applications in the study are relatively limited. It would be beneficial to apply the method to more datasets, such as ImageNet, and explore its effectiveness on a wider range of networks.\n2. When the number of stacked samples is 4, a noticeable performance drop is observed.\n\nQUESTIONS:\n1. In lines 86 and 87, why is it stated that n and $x^{(1)}$ are orthogonal?\n2. I don't fully understand how Dynamic Inference is performed.\n3. Are unbinding keys shared among all data or does each sample have a corresponding unbinding key?\n\nLIMITATIONS:\nI believe the limitations of the method mainly manifest in two aspects:\n1. Performance significantly decreases when a large number of samples are stacked.\n2. It remains uncertain whether similar performance can be maintained when applying the method to a wider range of structures.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes a novel method named multiple-input-multiple-output (MIMO) neural networks, which aims to achieve simultaneous inference for several inputs together by mixing them into one input. To this end, the authors invent one method to first encode the inputs, whose output can be decoded to give separate outputs. The proposed method is evaluated on both convolutional and attention models on CIFAR datasets and LRA framework.

STRENGTHS:
The proposed method to mix several inputs is novel and interesting. Also, the authors conducted simple analysis and several experiments.

WEAKNESSES:
The method is only verified on some small datasets like CIFAR. Also, the results in Table 1 seem to give a significant accuracy drop for multiple input cases, compared to using a single input. Also, it would be better if the author could provide more results on other tasks, like object detection or segmentation.

QUESTIONS:
Could the author provide more explanation on how to apply the proposed method in practice?

LIMITATIONS:
The authors did not provide limitations. I think the practical application of the proposed method can be limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUPDATE: scores updated based on rebuttal.\n\nThis paper proposes a method (MIMONets) for multiplexing multiple independent samples in superposition in such a way that one can train neural networks to simultaneously process those samples in training and inference. The method is adapted both for CNNs and Transformers and there is empirical evaluation for both that show the viability of the method.\n\nSTRENGTHS:\nThe paper is well written and easily understandable, and the method is relatively well grounded in foundations of previous works and new analysis.\n\nWEAKNESSES:\nThe authors claim that this is the first time this has been done, but they fail to mention some related work such as Murahari, Vishvak, et al. \"DataMUX: Data multiplexing for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 17515-17527. Please review that work and discuss similarities and differences, and if possible conduct experimental comparison. The DataMUX paper claims an impressive 40 samples multiplexing for the transformers, while MIMONet uses just 4. At least, the CNN part of the MIMONet work seems to give better empirical results, though.\n\nOne of the strongest motivators for the MIMONets is the ability to “dynamically” scale computation vs. accuracy at inference run-time using the same trained weights by changing the way input and output is processed, and by creating in-network ensembles if more resources are available. The authors should center the motivation more in this area unless they can have a strong argument to the “static” case.\n\nRegarding the static case, there is less convincing results. For example, since there is a drop in accuracy with multiplexing 2-4 samples compared to 1, the comparison should be done with a smaller baseline with the same accuracy, and the authors should include full FLOPS figures for the models, including the bind/unbind operation.\n\nAlso, for the static case, some comparison methods include pruning, quantization aware training, compression. It would be needed to compare these in terms of intOPS/FLOPS and power consumption estimates using models of the same accuracy.\n\nMIMOConv\n\nFor the MIMOConv model, the authors should include the mathematical description of bind/unbind including all dimensions and indexes.\n\nTable 1 does not clearly state the FLOPS / sample for each of the rows. \n\nTable 1 does not discuss what would be the FLOPS size of a baseline model that reaches the same accuracy as e.g., N=4 MIMOConv models.\n\nMIMOFormer\n\nThe authors should add some more information on the Transformer model, for example, in the Figure 3 it seems every layer does bind/unbind. This seems different from the CNN models, so that authors should describe this more in detail and give more background to these choices. What is the complexity addition in FLOPS from these bind/unbind operations? \n\nTable 2 (MIMOFormer results) is not fully clear what is the computational complexity (per sample) for each of the rows. For example, do the MIMOFormer models have higher complexity in FLOPS because of the bind/unbind operations? I assume that MIMOFormer N=4 is almost 4x more efficient pers sample than N=1, but it would be great to report full model FLOPS/sample for each of the rows.\n\nTable 2 seems to hint that the +MLP version performs worse (especially for N=4). But do I understand correctly that it also has less FLOPS since the -MLP version repeats the same MLP for each of the superimposed samples? This should be described. Would the +MLP version perform better if MLP was bigger?\n\nTable 2 is missing comparison to a non-MIMO Performer baseline that would have the same FLOPS complexity per sample.\n\nIn Table 2 there seems to be some tasks, such as retrieval, where N=4 seems to make the performance drop dramatically. Could the authors discuss these?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nComparison for DataMUX missing, some apples-to-oranges comparisons in the experimental section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}