DataMUX: Data Multiplexing for Neural Networks
Authors: Vishvak Murahari
Carlos E Jimenez
Runzhe Yang
Karthik R Narasimhan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural networks
Multiplexing
Efficient inference

Abstract: In this paper, we introduce \emph{data multiplexing} (DataMUX), a technique that enables deep neural networks to process multiple inputs simultaneously using a single compact representation. DataMUX demonstrates that neural networks are  capable of generating accurate predictions over \emph{mixtures} of inputs, resulting in increased inference throughput with minimal extra memory requirements. Our approach uses two key components -- 1) a multiplexing layer that performs a fixed linear transformation to each input before combining them to create a "mixed" representation of the same size as a single input, which is then processed by the base network, and 2) a demultiplexing layer that converts the base network's output back into independent representations before producing predictions for each input. We show the viability of DataMUX for different architectures (Transformers, and to a much lesser extent MLPs and CNNs) across six different tasks spanning sentence classification, named entity recognition and image classification. For instance, DataMUX for Transformers can multiplex up to 20x/40x inputs, achieving up to 11x/18x increase in inference throughput with absolute performance drops of $<2\%$ and $<4\%$ respectively compared to a vanilla Transformer on MNLI, a natural language inference task. We also provide a theoretical construction for multiplexing in self-attention networks and analyze the effect of various design elements in DataMUX.

DataMUX: Data Multiplexing for Neural Networks
Vishvak Murahari
Carlos E. Jimenez Department Computer Science
Department of Computer Science
Princeton University
Princeton University
murahari (prpaarroirinnneoo..... edu
carlosee@princetorr edu
Runzhe Yang
Karthik Narasimhan
Department of Computer Science
Department of Computer Science
Princeton University runzheyEprinceton.. edu
Princeton University
karthikm@@princetonn. edu
Abstract
In this paper, we introduce data multiplexing (DataMUX), technique that enables deep neural networks to process multiple inputs simultaneously using a single compact representation. DataMUX demonstrates that neural networks are capable generating accurate predictions over mixtures of inputs, resulting in increased inference throughput with minimal extra memory requirements. Our approach uses two key components - 1) a multiplexing layer that performs a fixed linear transformation to each input before combining them create 'mixed' repre- sentation of the same size as a single input, which is then processed by the base network, and 2) demulriplexing layer that converts the base network's output back into independent representations before producing predictions for each input. We show the viability of DataMUX for different architectures (Transformers, and much lesser extent MLPss and CNNs) across six different tasks spanning sentence classification, named entity recognition and image classification. For instance, DataMUX for Transformers can multiplex to 20x/40x inputs, achieving up 1xx18x increase in inference throughput with absolute performance drops of 2% and < 4% respectively compared to vanilla Transformer on MNLI, a natural language inference task. We also provide theoretical construction for multiplexing in self-attention networks and analyze the effect of various design elements DataMUX.
Introduction
Deep neural networks are very effective at modeling complex functions from real-valued vector inputs to vector outputs. However, recent studies have hinted that modern networks are vastly overparameterizee Kaplan et al. (2020) Allen-Zhu al. (2019a) and only require fraction their parameters for capturing many functions Frankle and Carbin (018); Frankle et al. (020). This raises natural question: if networks contain greater processing capacity than necessary, could be possible for them to model function over multiple inputs simultaneously, similar to how radio channels share bandwidth to carry multiple information streams at once? Such a capability could enable significant improvements in efficiency and inference throughput without requiring substantial increases in model size or computation.
In this paper, we propose Data Multiplexing (DataMUX)) technique that enables neural networks to process multiple inputs simultancously as a single "mixed" representation of the same size as
Code available at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Batch Procossing
32 instancee
DataMUX
instances
and they all come not neAr
Neural
he WaS hore loss...
you be able
8 220
MUX
Neural Net
DEMUX
1280 door
1280
200
Figure 1: Schematic illustrating our proposed DataMUX technique in comparison to traditional minibatch processing in neural networks. Here, DataMUX uses a multiplexing layer (MUX), to multiplex 40 instances into single representation, passing only 32 imputs the neural network. The demultiplexing layer (DEMUX) uses the output to generate predictions for 1280 instances.
single input, during both training and inference (Figure D. This is complementary to and different from minibatch processing that adds an extra batch dimension to the input, which in effect increases computational and memory requirements. Due to the use of a single combined representation, DataMUX can increase network inference throughput with minimal time and memory overhead.
The key challenge for DataMUX lies in effectively compressing a mixture of signals so that it can be processed by neural network without sacrificing task performance. To this end, our procedure involves the addition of a multiplexing layer and demultiplexnng layer to both ends of a vanilla neural network. The multiplexing layer applies fixed linear transformation each input before combining them to create a single "mixed" representation. This is then fed into the base network, which processes it to obtain "mixed" vector representation. The demultiplexing layer then converts this vector back into a set of vector representations corresponding to each original input, which is then used make final predictions for each instance. Our multiplexing and demultiplexing layers are end-to-end differentiable, which allows us to train the entire model jointly through standard gradient descent methods. We find that training time convergence increases almost linearly with the number of multiplexed in stances as training gets more challenging since the model has to make multiple simultaneous predictions. In order to train multiplexed models effectively, we also introduce a special token retrieval task as a pre- trainini objective.
We demonstrate the effectiveness of our DataMUX approach using three different architectures (Transformers Vaswani et al. (2077). MIPs, and CNNs) on six tasks spanning sentence classification, named entity recognition and image classification. Our results demonstrate that DataMUX models can successfully process between 2-40 input instances simultaneously, with minimal losses in task performance. For instance, on the sentiment analysis dataset SST-2, our multiplexed Transformer (T-MUX) can handle 40x inputs with only a 2% drop in accuracy compared to vanilla Transformer. To our best knowledge, this is the first demonstration of large-scale data multiplexing in Transform- ers. While our results on CNNs and MLPs for image classification are not as strong as those on Transformers, we present them to demonstrate the generality of our method and believe that better multiplexing methods may yet improve them in the future.
We further perform theoretical and empirical analyses better understand our results. First, show that it is possible to theoretically construct self-attention networks that can simultaneously process inputs in N independent subspaces. Next, we perform several analyses of various multiplexing strategies and the effect of attention heads and model size. Finally, we also measure throughput statistics to demonstrate the efficiency gains provided by DataMUX (up to 18x inference speedup). We believe multiplexing holds great promise in improving the efficiency and versatility of neural networks and hope that future research can develop improved strategies for DataMUX and expand its applicability to more architectures and tasks.
2 Related Work
Multiplexing in deep learning While multiplexing is a standard idea in signal processing Rabiner and Gold (1975), its use in deep learning has been limited. Lu et al.] (2020) develop MUXConv, a layer that multiplexes both spatial and channel information for convolutional neural networks
(CNNs). Instead of performing data multiplexing, these models multiplex different features within
the network and only process one input at a time. mixup Zhang et al. 2018) proposes training scheme where networks are trained with a convex combination of instances predict the instances' label distributions from this combined representation. They find that this scheme provides implicit regularization of models and improves generalization during inference. Unlike our work however,

---

mixup does not preserve the order of the combined instances and therefore mixed instances may not be used during inference. Another line of work uses multiple-input multiple-ouputt training to obtain multiple sub-networks within a neural network for relatively small number of inputs Havasi al. (202T); Ramé et al. (20TT) ;; Soffaei et al. (020). These approaches aim to improve single-instance performance through ensembling sub-networks' predictions. In contrast, we are motivated by achieving optimal performance for prediction on multiple inputs and improving model
throughput during inference.
Multiplexing in the brain On the biological side, several studies hint at the capability of the brain to multiplex and transmit tremendous amounts of information with single neural code and process them in parallel Bllumhagen et al. (20IT); Akam and Kullmann (2014): Pirschel and Kretzberg (016):Hong et al. (2016); Friedrich et al. (2004). Recently, Lankarany al. (2019) confirmed that neurons in the somatosensory cortex multiplex different types of features (low-contrast, high-contrast) using their timing and rate of synchronous and asynchronous spiking. Rezaei et al. (202T) further provide a computational model to explain multiplexing with an ensemble of homogeneous cortical neurons. These studies increasingly demonstrate that data multiplexing key feature of natural neural networks.
Over-paramattriiation in neural networks Our work also relates the over-parametterizaton of deep neural networks that has been investigated in several pieces of prior work Allen-Zhu et al. (2019b ab; Radhakrishnan et al.) (2020). The lottery ticket hypothesis Frankle and Carbin (2018);
Frankle et al. (2020): Malach et al. (2020) demonstrated that only a fraction of neurons in a trained deep model are sufficient to capture required function. Neural architecture search Zoph and Le (2016); Liu et al. (2018) Elsken et al. (2019) aims to balance performance with parameter efficiency. by discovering the most optimal architectures for each task. There have also been several efforts at leveraging this over-parameterrizaion effect for multi-task learning Hu and Singh (2021). Cheung et al.) (2019) propose method to superpose many models for different tasks into single neural network, and show that it helps mitigate catastrophic forgetting. Wortsman et al. 2020) is another work along the same lines which uses a base network, along with specialized sub-networks for different tasks. work possibly takes advantage of the over- parameterization existing in deep neural networks. However, to our knowledge, we are the first to utilize data multiplexing to simultancously process multiple inputs during both training and inference, and are able to show that fixed set of parameters can process a mixture of inputs (even up to 40) with minimal loss in accuracy.
Similar to these paradigms,
Method
Our primary goal is to process set of input instances simultaneously over shared neural network (multiplexing) with minimal overhead during inference. To this end, we design DataMUX consist of three components: multiplexer module to combine the multiple input instances into superposed representation, a neural network backbone to process mixed representations, and a demultiplexing module to disentangle the processed representations for individual prediction. A schematic illustrating the flow of representations is shown in Figure 2 We detail the general requirements for each component below, as well as specific implementation details used in our experiments.
3.1 Multiplexing
The multiplexer module, denoted 4 combines tuple of inputs, either images or sentences from a batch, (x) (x,,...,xx x**), for x' E Rd, into a more compact representation x I:N E Idd in an order- dependent way, which enables effective demultiplexing after processing, well distinguishing intra-sequence interactions in the case of sequenced inputs (e.g. token sequences). Towards this end, for each input x' with index i E [1] of the input tuple, the multiplexer module performs transformation of (Rd - Rd),, on the instance before finally averaging all inputs into single multiplexed representation as:
N = N
1
x I:N ((x
Σ 0 (x*).
(1)
For sequenced inputs (e.g. token sequences), we combine N sequences by multiplexing token- wise. That is for inputs of the form x" [w)Jee(,,L) where w) E Rd is token's 's input vector representation, this operation uses the same transformation o² for each token in the sequence before
3

---

averaging over each position across indices, such that x I:N = [w] 1:N bel.. where with R is multiplexed representation of N tokens at position j. i.e., x (f(W)..-. w) )EE(LE))-
For o²', we experiment with using either (1) linear projection with random fixed orthogonal matrix (denoted "Ortho"" or (2) the Hadamard product with a fixed Gaussian random vector (denoted "Had- mard", equivalent to linear map using diagonal matrix in our case). We hope these transformations map instances at different indices into distinguishable regions and consequently reduce interference between their representations, Finally, this multiplexed representation, x=:N, used input the neural network backbone, which is architecturally unchanged.
3.2 Demultiplexing
The output of the neural network backbone will be a multiplexed hidden representation h of the x input I:N, To make a prediction for each input, one can explicitly disentangle I:N into N individual hidden representations, h h'.....hi h", respectively. We first obtain each h' with a demultiplexing function &¹, i.e..
h' = ='=0'(h" (h""""). E
(2)
For sequenced input, demultiplexing is done position wise, i.e., hj (hhh.. for each position j.
Finally, predictions are made using shared task head on each inputs' respective individual hidden representation to prevent a substantial increase in the number of parameters and improve training efficiency. We use two alternatives for the demultiplexing function (
MLP Demuxing This strategy employs N MLPs to generate each indices' hidden representation as h' = (h 1-8). ). We use this method for both NLP and vision tasks. Although this method is conceptually simple, it adds learnable parameters proportional N.
2. Index Embeddings We generate index embeddings p', which are then concatenated to h! and transformed by a shared multi-layer network to generate each individual hidden representation, i.e., hy pL.aaarred (h):.V p To generate the index embeddings p', add sequence of N special tokens, called the prefix, to the beginning of each sequence of the input tuple. For multiplexing with N sequences, we add N corresponding prefixes, denoted prefix' for i E [1, N]. Each prefix consists of index token e in it's i'th position while the remaining tokens special pad token enad The prefix sequences then take on the following pattern:
prefix' = [c_, pad
pad
prefix² = [c 2 pad
padi
prefix N = [cPad
pad
We then prepend each sequence x² the input sequence with the corresponding prefix'. The tuple of prepended sequences is then passed to the multiplexing module. When finally generating individual hidden representations, we use the corresponding hidden representation each index token el as the index embedding p².
We use the Index Embeddings (.2)] demultiplexing strategy on language tasks for the Transformer architecture. The prefix tokens may implicitly enable the Transformer to do instance-specile computations when processing the multiplexed representation and further enable demultiplexing for large N.
3.3 Retrieval warm-up for multiplexing
For Transformer models, we find that naively adding the multiplexing and demultiplexing layers to the model fail to converge. This is likely because the gradients for individual instances from the task loss get mixed up in the backward pass. To overcome this, propose Retrieval warm-up self-supervised pre-training task to promote the ability of DataMUX models at distinguishing the order and the content of individual sequences in multiplexed representation. This task consists of retrieving the correct tokens and order for each position and sequence of the input tuple (Figure 4a). Although we could add this loss for every sentence in each position, memory constraints force to retrieve a token from one random sentence for each token position, yielding the following objective:
CRetieval (z (N..
Σ
log (w (h/h
(3)

---

B-ORDI-ORG D
h
+
Shared Head
Transfomer
h
penut
Muttiploxorr
mean
ratoo sold
Figure 2: DataMUX for Transformers: (Left) Given a tuple of N sentences (x 2 x ). each of length L, we first apply mutriplexing operation which performs a transformation o' on the embeddings of each sequence x', such that the same transformation is applied to every token in a sequence x'. The multiplexing operation then aggregates the sequences by averaging over each position, generating a single combined sequence L E I 4xd for embedding size d, which will be passed on to the central Transformer model. (Right) After processing. we perform demultiplexing operation the Transformer model's output h R. generate hidden representations h' h
corresponding to inputs x' = _/ respectively. We finally use these hidden representations to generate predictions for a particular task (e.g. named entity recognition (NER)) using a shared task prediction head.
where ht is a demultiplexed hidden representation of the j-th token in a randomly selected sentence with the index [[11, .N), generated using the methods described in Section 3.2 We use this objective to optimize all the parameters of the end-to-end multiplexed model. For sequenced inputs, we find the retrieval auxiliary objective crucial learning and report performance on the retrieval task in Section
4 Multiplexing for Transformers (T-MUX)
4.1 Experimental setup
Models We first evaluate the capabilities and limits of data multiplexing specifically for the Trans- former architecture on range of text classification tasks. We apply DataMUX on 12-layer Transformer based model with hidden dimension size of 768 and 12 self-attention heads built on the Huggingface Wolf et al. 2019 framework, and refer to the resulting model as T-MUX. We compare our T-MUX models to baselines: (B1) A 12-layer 768 hidden dimension vanilla Transformer. (B2) 12-layer 768 hidden dimension Transformer pretrained using the retrieval task described in Section 3.3 Though there is multiplexing done for B2 (meaning this operation could be solved by simply copying input tokens to the output layer) we find that the retrieval pre-training produces differences in performance and we show this baseline for completeness.
We also apply DataMUX to smaller models (see A2 and compare with similar baselines.
Tasks We evaluate our models and the baselines on two types of text classification tasks:
1. Token-level classification: This evaluates models' ability to perform token-level tasks mul- tiplexed inputs. This poses a particular challenge for data-multiplexing models since they must maintain high level of individual token disentanglement while also producing representations capable of solving the task. We evaluate token-level classification on the CoNLL-2003 Named Entity Recognition (NER) task [Sang and Meulder (2003).
2. Sentence-level classification: We evaluate models on a subset of the sentence-level classification tasks found in the General Language Understanding Evaluation (GLUE) benchmark [Wang al. (2019) the sentiment classification task SST-2 [Socher et al. (2013) the sentence similarity task HOP and the natural language inference tasks MNLI Williams al. 2018) and QNLI Wang et al. 2019): Rajpurkar et al. (2016). By evaluating on a variety of sentence-level tasks, we can gain a better sense of the capabilities of data multiplexing neural networks tasks that require aggregating contextual information. Similar to previous works, we prepend a [CLS] token to all sequences and learn task head on top of the demultiplexed [CLS] token representation.
Auxiliary retrieval objective The T-MUX models are all pre-trained using the retrieval warm-up on the Wikitext-103 dataset Mataset/Merity et 2017) as described in Section 3.3 addition, we also
https//data..uura.com//ttt Dataset Release-Questinn-Pee Question-Pairs
5

---

MNL (Aco.
NER (F1)
GNLI (Ace)
50
90
05
(Aoc.)
(Aok.)
02
60
70
20
10
Frctxt Ensedd
Figure 3: Multiplexing task evaluation for T-MUX. Across tasks, we demonstrate multiplexing up to 40 instances without significant drop in performance. Further, we show results here for Index Embedding demultiplexing as the MLP Demuxing method leads optimization instability for the Transformer architecture. We provide results for the MLP Demuxing method in Appendix A.6
continue to the retrieval task auxiliary objective during task training. The total loss is combination of the task loss and retrieval loss (we use a = 0.1 in our experiments):
L (1 L R.etrieeall:
4.2
Main results
(R1) Multiplexing leads to minimal drop in performance even for large N Figure 3 shows performance across four sentence classification tasks (MNLI, QNLI, QQP, SST2) and token-level classification task (NER). We observe that it is possible to multiplex up to 40 instances and maintain reasonable performance. For easier tasks like QQP, SST2 and QNLI, we observe that the drop in performance with increasing N is insignificant while for more difficult tasks like MNLI and NER, there is a trade-off between performance and N, with performance dropping 10%- 15% for 40 instances. Since we've encountered unstable optimization when using ML. Demuxing, we only provide results using Index Embeddings demultiplexing. We find that the multiplexing strategy does not impact performance across different tasks, even slightly increasing for small values of N (e.g. 2, 5). This increase may be attributable to implicit regularization à la mixup. We also did not notice much variance between fine-tuning runs and plot variance for the "Hadamard Index Embedding" setting in Section A.4
(R2) Perfect multiplexing on the retrieval warm-up task for large N Figure 4b sbows the test accuracy on the retrieval warm-up task described Section 3.3 We first note that across different multiplexing and demultiplexing strategies, models have a retrieval accuracy of nearly 100% for up to 20 instances, demonstrating the surprising ability of T-MUX to multiplex perfectly for large N. Note that this task does not require any aggregation of context across the sequence and thereby is much easier than sentence classification or token-level classification tasks. Therefore, performance on this warm-up task indicates soft upper bound on the number of instances we can multiplex for sentence and token-level classification tasks given a particular multiplexing and demultiplexing method.
(R3) Throughput can be increased multi-fold We measure throughput of our multiplexed model (Hadamard + Index Embed) across different number of instances by calculating inference speed for processing ~200000 samples on the MNLI dataset. We use four different batch sizes for all the configurations and take the max throughput (details in A.8). Figure 4 shows that multiplexing increases throughput many folds (18x for 40 instances. 11x for 20 instances). T-MUX enables superior throughput as batch size can be effectively increased by a factor of N. We would expect the speedup to scale linearly with N, however a large N corresponds to more prefix tokens which increases the sequence length. Therefore, having 40 instances leads to almost a 20x speedup as opposed to the expected 40x. Future work can potentially improve this speedup by designing better multiplexing and demultiplexing strategies.

---

8
nato
sald
red
sox
are
Retrieval (Ace.)
1.0
Total Runtime
25x
Throughput
f
Shared Retrieval Decoder
0.8c
23k
80
0.6x
13k
h
10k
Deruliiptexer
70
Ontio MEP Dersux
Orha role Erredd
0.2x
56
heccerard NUP Deran.
0.0X
2
8
10
20
10
20
40
x nato cold
60
N
N
+8 12. 3544
4L 755H
12L 768M
Figure 4: (Left a) Given the Transformer's combined output representation h generated from the input (x". x² "...,,""" the goal of the retrieval task to retrieve the original tokens from (a* ,,"", ....,,"", ). Since h= is order-dependent, retrieval requires distinguishing not just which tokens were originally input to a j L]. but also which sentence [1, .V] each belonged to. (Center b) Accuracy for the retrieval warm-up task. Surprisingly, models are able retrieve words with 100% accuracy up to 20 instances across most multiplexing and demultiplexing strategics. (Right e) Runtime and throughput numbers T-MUX models on 20K MNLI instances, normalized by hase model's 's performance without multiplexing. T-MUX (121/768H) can multiplex up to 40 instances leading to ~ 18x speedup.
MNLI (Ace.)
NER
MNLI
NER
005
12
53
10.0
42
65.2
10
10
25
2 Alllertin Fraeda
-8" 12 Ad Pineda
4L172811
-8- 12L 2047
4L 7581 Daps Pane
/2L D3M
Figure 5: (Lefft a) Understanding the role of attention heads in multiplexing. Reducing attention heads to 2 does not impact performance greatly, suggesting that attention heads are not core to multiplexing. (Right b) Multiplexing performance with smaller model capacities. Smaller models can multiplex up to 20 instances without significant drop in performance.
4.3
Analysis
(A1) The number of attention heads seems invariant multiplexing To understand the role of the number of attention heads in multiplexing, we train a variant of T-MUX with 2 self-attention heads per layer. We use T-MUX with the (Hadamard + Index Embed.) configuration. We find that reducing self-attention heads has minimal effect on performance on the retrieval warm-up task and achieves a retrieval accuracy ~ 100% up to N = 20. We then find that on token and sentence-level classification tasks, T-MUX with 2 self-attention heads performs comparably to T-MUX with 12 self-attention heads (Figure 5a
(A2) DataMUX also provides throughput boost with smaller Transformers We further investigate the applicability of DataMUX to smaller Transformer models. We choose two smaller Transformers to multiplex: 12 layer with hidden size of 384 (12L/ / 384H), 4 layer with hidden size of 768 (4L7 768H). Figure shows that these smaller T-MUX models can also multiplex up to 20 instances with competitive performance. Figure 4c illustrates the speedup from the smaller models. As the smaller models can only multiplex up to 20 instances with reasonable performance, we see that multiplexing with 20 instances provides even higher throughput of 25x, compared to only 18x for the full-sized T-MUX with 40 instances.
(A3) Performance varies more across different indices N increases The prediction of an instance is conditioned on the index of the instance. Figure 76 illustrates performance on MNLI for different indices across different choices of N for the (Hadamard Index Embed.) configuration. We observe that performance varies more across different indices for large N (For N = 40, performance varies across 10 percentage points).
(A4) The demultiplexed representation of an instance is robust to the other instances it is multiplexed with To analyze whether the representation of an instance changes with respect to the other samples it is multiplexed with, we randomly select 10 instances 21, 210 from the
We performed empirical study determine models that performed best, see A.7 for details.
7

---

2
5
N
20
10
15
10
-5
-10
10
-10
-15
-20
-20
-10
10
20
-10
0
10
-20
-10
0
10
N
20
40
20
20
10
-10
-10
-20
Figure T-SNE plots to understand how the demultiplexed representation of an instance changes with respect to the set of instances it is multiplexed with. Each color (10 in total) a single input sequence. Across different N, we find that the demultiplexed representation of instance is not significantly impacted by the set of instances it is multiplexed with.
MNLI dataset and multiplex each of them (separately) with 30 different sets of other instances vi. y2: ... viy for i E [1..... 31). This generates 30 different demultiplexed representations for each of the 10 selected samples. We then visualize the resulting points by reducing the 768 dimen- sional demultiplexed representation to a 200 dimensional vector with PCA (FR.S. (190)), followed by t-SNE van der Maaten and Hinton (2008) visualization. Figure 6 shows these clusters for different values of N. We find that across different values N, all the points corresponding to the same 24 are very close each other, which suggests that the representation of an instance is not significantly influenced by the set of instances is multiplexed with.
4.4 Theoretical construction for multiplexing in self-attention models
To provide some theoretical explanation for why Transformers may be amenable to multiplexing, we also detail below a construction for self-attention based neural networks which process multiplexed token embeddings in N independent subspaces. Our construction relies on particular structural properties of singular spaces linear transformations across layers we provide brief sketch below, with more details in Appendix A.3
In model with self-attention, the multi-head attention projects the queries, keys, and values à times with different, learned linear projections to dx. dx and dy dimensions respectively. Thus, given a sequence of d-dimensional multiplexed token embeddings (w! ]ee[1,2]. each head looks like:
head,())
w." W
(5)
(W/"WAN) w?...
E exp
Now, wi:N 1/2 E (w and let us assume each function &* be trained to project each embedding into subspace that is least dineaely-dependent with the others. So, if we define u. (k) = of (w("), assume (u, (k) u, (0))) se 0 for all pairs of indices + k' and all positions t. To preserve this independent subspace structure after self-attention, we will first need
(W (+) W.Y. (P) (t) (W." w.) !! o.
(6)
8

---

This is achievable if the eigenvectors of W. wy can be grouped into N non-overlapping subsets fri (1) ,...., "..,,,))), ... (rl (r(N) ....., I' } where rt are orthonormal vectors (since the Gramian is real symmetric), and span the same input subspaces, denoted as D ..... D.N. In this case, the vector (N) Vi ..... iit in dual subspaces
after transformed by W.' can be expressed as sum of N vectors (1) Di...., DV which are independent of each other. The linear maps from D to Dÿ still allows rich operation on each component of multiplex input without interference by other components.
In addition to decompress-able value vectors, we can set the query and key matrices, 1º and W.K. to have some subsets of right and left singular vectors such that span D ..... DA and Dy..... D.y Then, we can show that the inner product of the query and keys of the i-th head can be rewritten as:
Di....,
W K W:N)
W w W.N.
Σ
(R)
where T,,,r' is scalar only depending on the k-th input sequence. Thus, the self-attention operation at each position can be seen as retrieving values based on the average of query-key similarity scores of N sequences.
(k)
(1) = Σ
exp
En
(A) Tacee
M
Σ exp E. Faiee /ide/Vdd)
This average retrieval using soft-max could be desired property as implicit regularization. However, if we want perfect non-interference in retrieval, the network always has an option to specialize each (k')
head to only focus on one input sequence, by setting Ti,t,'' = for all K' + k, which is easily
achievable by controlling singular values of W.º W.K. More details of this construction are provided in Appendix A.3] Figure 4.3 suggests that number of attention heads do not significantly impact our empirical multiplexnng results. Therefore our theoretical construction does not fully explain what models learn empirically but nonetheless provides an existence proof for multiplexing under certain assumptions.
5
Multiplexing for MLPs and CNNs
We also investigate multiplexing for multilayer perceptrons (MLPs) and convolutional neural networks (CNNs). Certain cases of data multiplexing have been explored for convolutional architectures image classification as techniques for robustness and data augmentation Ramé et al. (202T): Haaaas et al. 2021;; Soflaei et al. (2200) These works implicitly employed the frontend layers of convolutional net as multiplexing layers and suggest that a convolutional neural network can learn most 3-4 independent subnetworks concurrently Havasi al. (02T). Since Transformers can be multiplexed for up to 40 instances without a severe performance drop, we explore the Mut scheme for MLPs and CNNs on the MNIST image classification task LeCun and Cortes (2005 While our results for these architectures are not as strongly positive as for T-MUX, we believe that future work on better multiplexing strategies can make them more viable.
Multiplexing for MLPs Figure 7a (solid lines) illustrates performance of DataMUX for MLPs with various multiplexing strategies, paired with the MLP demultiplexing strategy. As a baseline, we show multiplexing using the identity transformation before combining instances (MLP baseline). Since this transformation does not preserve the order of the multiplexed instances, accuracy expectedly decreases on the order of 1 1/N. Multiplexing with random orthogonal matrices (MIP + Ortho) works for up to instances with an accuracy of ~ 78%, compared to 95% for the baseline without multiplexing. Since random orthogonal projections cannot separate inputs perfectly, also experiment with a set of N low-rank independent transformations for multiplexing (MLP + LowRank) and observe that this improves performance slightly for larger N (by 5% for N=8).
Multiplexing for CNNs Results for CNN multiplexing are shown in Figure 7 (dashed lines). Like in the MLP case, all methods shown use the MLP demultiplexing strategy. We notice that the baselines (CNN baseline) has a roughly N-lold performance drop because of the unidentifiability of input order. The "Ortho" transformation performs quite poorly for CNNs (only ~ 56% for N 8).
IImplementation details can be found in Appendix A.10

---

MNIST (Aoc.)
Demux Index Performance
625
NLP (eneire))
60
KLP Citte
MNLI (Acce
KLP LowForkk
(Taosinr)
$
40
18
2
5
20
40
N
Figure 7: (Left a) Average test accuracy for multiplexing MLP and CNN for N = (1, 4, 16) on the MNIST classification task for different multiplexing strategies. MLPs can be multiplexed up to N = 8 without significant drop in performance while CNNs only be multiplexed up 4 instances. (Right b) Performance results for different demultiplexing output indices. Results are reported for Transformer model with the Hadamard product multiplexing and Index Embeddings demultiplcxing. Variance of performance across diffcrent indices with increasing N.
This is likely because this transformation destroys the property of spatial locality that CNNs rely We therefore also explore N two-layer convolutional networks with tanh activation out multiplexing transformations o² (CNN + Nonlinear). We note that even with this transformation. the dimensionality of the multiplexed representation still remains equal to the dimensionality of single input. We find that for < 4, performance is above 80%, which significantly better than CNN+Ortho, but performance drops rapidly for N 4. We show results of other multiplexing strategies for CNN in Appendix A.IT
Overall, multiplexing for MLPs and CNNs seems be more challenging than Transformers as evidenced by the sharper performance drops with increasing N. However, we believe these numbers still non-tri trivial and show the potential for multiplexing in CNNs and MLPs, with multiplexing and demultiplexing strategies better suited for these architectures.
Discussion
In this work, we have shown that neural networks can be trained to multiplex, i.e. process multiple inputs using single "mixed" representation. We introduced DataMUX, novel setting for training and inference with multiplexing, and demonstrate its viability using a variety of network architectures and tasks. Our results reveal the surprising ability of neural networks to predict up 40 input instances simultaneously using data multiplexing. Our efforts have sought to show the potential for data multiplexing using methods that require limited computational overhead or added learned parameters; achieving results that incur minimal degradation in performance while simultancously increasing system throughput dramatically.
While the scope of this paper is to demonstrate the general ability and potential for neural networks to use data multiplexing, there are several avenues for future research investigate the limits and theoretical implications of DataMUX. Directions of particular interest include: large-scale pre- training, multi-modal processing, multi- lingual models, different mechanisms of multiplexing and demultiplexing, and a more rigorous understanding of the architectural, data, and training conditions enabling data multiplexing.
7
Acknowledgements
We thank Ameet Deshpande, Shunyu Yao, Jens Tuyls, Tianyu Gao and Mengzhou Xia for their valuable feedback on early drafts and encouragement throughout the course of this project, and the anonymous reviewers for their suggestions on improving the paper.
References
Thomas Akam and Dimitri Kullmann. 2014. Oseillatory multiplexing of population codes for selective communication in the mammalian brain. Nature Reviews Neuroscience, 11-2).
10

---

Z Y Li, and Y Liang. 2019a. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. 2019b. A convergence theory for deep learning via over-paramettriiztion. In International Conference on Machine Learning, pages 242-252. PMLR.
Francisca Blumhagen, Peixin Zhu, Jennifer Shum, Yan-Ping Zhang Schärer, Emre Yaksi, Karl Deisseroth, and Rainer W Friedrich. 2011. Neuronal filtering of multiplexed odour representations. Nature, 479(7374):4933 498.
Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. 2019. Superposition of many models into one. In Advances Neurul Information Processing Systems, volume 32. Curran Associates, Inc.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. The Journal of Machine Learning Research,
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR.
Rainer W Friedrich, Christopher Habermann, and Gilles Laurent. 2004. Multiplexing using synchrony in the zebrafish olfactory bulb. Nature neuroscience, 7(8):862-877 871.
Karl Pearson F.R.S. 1901. Liii lines and planes of closest fit systems of points in space The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(115559--572
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Laksh- minarayanan, Andrew Mingbo Dai, and Dustin Tran. 2021. Training independent subnetworks for robust prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Sungho Hong, Mario Negrello, Mare Junker, Aleksandra Smilgin, Peter Thier, and Erik De Schutter. 2016. Multiplexed coding by cerebellar purkinje neurons. Elife, 33100
Ronghang Hu and Amanpreet Singh. 2021. Unit: Multimodal multitask learning with unified transformer. arXiv preprint wrXiv:2102.10772
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Milad Lankarany, Dhekra Al-Basha Stéphanie Ratté, and Steven A. Prescott. 2019. Differentially synchronized spiking enables multiplexed neural coding Proceedings of the National Academy of Sciences,
Yann LeCun, Bernhard E. Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D. Jackel. 1989. Handwritten digit recognition with back propaaaronn network. In Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pages 396-404. Morgan Kaufmann.
Yann LeCun and Corinna Cortes. 2005. The mnist database of handwritten digits.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pages 19-34.
Zhichao Lu, Kalyanmoy Deb, and Vishnu Naresh Boddeti. 2020. Muxconv: Information multiplexing in convolutional neural networks. 2020 IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12041-12050.

---

Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. 2020. Proving the lottery ticket hypothesis: Pruning all you need. In International Conference on Machine Learning. pages 6682-6691. PMLR.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. ArXiv, abs/ 109.609497733
Friederice Pirschel and Jutta Kretzberg, 2016. Multiplexed population coding of stimulus properties by leech mechanosensory cells. Journal of Neuroscience, 36(13):3636-367,
Lawrence R Rabiner and Bernard Gold. 1975. Theory and application of digital signal processing. Englewood Cliffs: Prentice-Hall.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. 2020. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences, (1tp)22222222772770
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392.
Association for Computational Linguistics.
Alexandre Ramé, Rémy Sun, and Matthieu Cord. 2021. Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks ICCV, abs/2103.06132.
Mohammad R Rezaei, Milos R Popovie, Steven Prescott, and Milad Lankarany, 2021. Synchrony- division neural multiplexing: An encoding model. medRxiv.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the con1l-2003 shared task: Language-independent named entity recognition. CoNLL.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Mamning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631-1642.
Masoumeh Soflaei, Hongyu Guo, Ali Al-Bashabsheh, Yongyi Mao, and Richong Zhang. 2020. Aggregated learning: A vector- quaettiquatizztiionn approach to learning neural network classifiers. In The Uhirty-Fourthe AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty--econd Innovative Applications of Artificial Intelligence Conference. IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5810-5817. AAAI Press.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using L-sne Journal of Machine Learning Research, (86):2579-266)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACI-HETT
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the--n natural language processing. ArXiv, 191./791...
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. 2020. Supermasks in superposition. In Advances in Neural Information Processing Systems, volume 33, pages 15173-15184. Curran Associates, Inc.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
12

---

Barret Zoph and Quoe V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint 1r115:611..0157.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Refer to
A.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Refer
to
A.2
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code. data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] Refer to supplementary material
(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)? [Yes] Refer to A.9 and A.10
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Refer to A.4
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Refer to A.8
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zGavL3Jisox", "reviewer": "Reviewer_DLX3", "summary": "The basic idea is as follows: multiplex multiple inputs into one input of the same size, then feed the model's output into a module at the end that untangles the multiplexed representations and makes a prediction for each input individually.\n\nThe multiplexing module randomly projects inputs then does an element-wise average. Demultiplexing is achieved by feeding the outputs into a demultiplexing function, which is either a set of N MLPs (where N = number of multiplexed inputs), or by prepending each input sequence with an index token such that a single demultiplexing head can use the index embedding to distinguish between inputs. To work with sequenced inputs, multiplexed models need to be pretrained on a self-supervised “retrieval” task which requires selecting the correct token for each possible position across the set of input sequences. The authors primarily evaluate their method on transformers and NLP tasks, and find that multiplexing results in modest accuracy drops but large throughput increases.", "questions": "Overparameterization/unused model capacity are presented as motivations for the development of and reason for success of the method. However, I don't think the analyses in the paper justify this sufficiently. First, the results for the small and baseline models should be compared directly. The following experiments would more thoroughly answer this question: Reduce the model size until performance begins to decline substantially. If model capacity is why multiplexing \"works\", then the multiplexing models' performance should decline sooner than the non-multiplexing models' performance. The same result should be observed when pruning model weights: performance as a function of weights pruned should be more favorable for non-multiplexed models than for multiplexed models.\n\nThe theoretical examination of multiplexing is interesting. I encourage the authors to test these theoretical predictions: for example, do the singular values of the query and key matrices differ between the baseline and retrieval-pretrained networks in the manner predicted by the theoretical examination? Furthermore, if reducing the number of attention heads has little effect on multiplexing capability, does it even make sense to focus the theory on the role of attention heads in multiplexing?\n\nThe authors demonstrate that pretraining is necessary for learnable multiplexing, and that retrieval pretraining can affect downstream task performance, but they don’t demonstrate that retrieval pretraining per-se is necessary for learnable multiplexing. While I suspect it is, I think it would be useful to show that other types of pretraining do not affect the learnability of multiplexing.\n\nThis method is presented as providing a resource savings due to the inference speedup. However, assessing the resource savings is difficult for a number of reasons. First, the pretraining protocol does not seem to be discussed at all other than a description of the task. Second, multiplexing causes an accuracy decrease relative to baseline models. Is it possible to compensate for this decrease by e.g. training longer? It would be worthwhile to see accuracy vs. training time plots for the different training configurations.\n\nIs it justified to compute speedups for small transformers relative to the baseline size transformer? This confounds the effect of transformer size on speedup with the effect of the multiplexing in small models on speedup.\n\nIs it possible to learn the multiplex projection?\n", "limitations": "The authors do a satisfactory job addressing the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: A clever method that I am somewhat surprised the authors were able to make work. The results are clearly presented. The inference speedups appear meaningful.\n\nWeaknesses: I'm not sure how relevant the theory is. Some of the stated motivations are not very thoroughly addressed by the analyses. Total resource savings is unclear.\n\nI think the paper is acceptable in its current form, and I think it would be excellent if revised appropriately.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FaxLPQejG5", "reviewer": "Reviewer_BEtA", "summary": "This paper proposed DataMUX, a method to allow DL models to process multiple inputs at the same time using a single representation. They conduct their experiments on token and sentence level classification for Transformer, MNIST for CNN and MLPs.", "questions": "1. From the experiment, it seems like a single representation can actually contain much information from multiple inputs, which is even enough to decompress it(demultiplexing). So is it possible that a traditional Bert [CLS] token can achieve the same thing with multiplexing pretraining?\n2. From section 4.4, even though the self-attention-based models are suitable for multiplexing in certain cases, it seems like it still requires multiplexing pretraining to reach the desired parameters. Therefore, I guess multiplexing and demultiplexing layers cannot just be added to any type of common language model right(e.g. Bert, Roberta)?", "limitations": "The authors addressed the limitations and potential negative impact of their work in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n  1. The motivation here is sufficient, especially the connection to biological perspectives.\n  2. Their experiment results show data multiplexing can speed up the throughput while maintaining performance.\n  3. It is great to have a theoretical explanation of the proposed method.\n\nWeaknesses:\n\n  1. Please refer to the questions\n\nI would recommend this paper be accepted. Overall, I think this paper has solid motivation from both theoretical and intuitive perspectives. The authors conducted many experiments to show that DataMUX is effective in many tasks. I think there is a lot to explore with the connection between over-parameterization and multiplexing, too, that would be interesting to other researchers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "93_Yb6N154c", "reviewer": "Reviewer_N4gg", "summary": "The authors propose to train a single encoder (mainly Transformer, but also CNN, MLP) to process multiple samples in parallel with the goal of approximating batch-processing without the computational overhead added by default batchwise processing. To this end, they first mix the input samples together (and add order tokens), then apply the encoder, implicitly forcing it to learn a form of parallel processing of this joint representation, and then de-mix the representation to obtain the prediction for each individual sample. Experiments indicate that Transformers may be flexible enough to process up to 20 samples in parallel (giving a 11x speed increase) while only losing 2-6pp.  \nperformance.\n", "questions": "- If I understand correctly, the goal is that the processing of each input should not influence each other input. Does this hold theoretically (e.g., decompositional Jacobians) or experimentally (e.g., resampling inputs and recording differences in the output)?  \nMore drastically formulated, how much can the prediction of one sample be influenced by the others in its batch? Does this open an ethically concernable weakness to attacks?\n- In this paper, the architectures (Transformers, CNNs) are viewed from a universal approximator \"black-box\" perspective. Which inductive biases implemented by these architectures motivate this claim, and which threaten to deteriorate the desired independence?\n- The model is trained on a fixed batch size. Can we hand over an incomplete batch to the model, e.g., by zero-padding or repeating the input (like we can for normal batch-processing)?\n- The two plots in Figure 4c show the same, only with inverted axes. I would find it more useful to use the space for additional information, such as increase in memory complexity (which was claimed to \"not rise substantially\"). Maybe consider a Pareto chart. This would also allow showing several baselines, i.e., B1 and B2 transformers with several batch sizes\n- I do not quite understand the Retrieval warm up: Why do we want to predict the whole input representations from the output, as opposed to just their order token?\n- Please provide a reference to the Index Embeddings Demultiplexing strategy in line 157.\n- A minor cosmetic critique is that plots are referenced to as, e.g., 4a, 4b, while the plots themselves are not labeled with a and b\n- Will code to reproduce the experiments be made available upon acceptance?\n", "limitations": "- Most technical and experimental limitations were adressed humbly and openly (see strength 5)\n- The paper opens a field with diverse positive and negative ethical implications which remained unadressed (see weakness 6)\n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I see the main value of this work in formulating a useful, novel task and taking a first shot at it (conditional on it being a new task, which I cannot confidently ascertain since it is not my primary\nfield)\n- If the model is able to guarantee independent processing of the inputs, this might have impact on causality and disentanglement literature, as a form of independent mechanisms\n- The main hypothesis (efficiency increase without big drops in\nperformance) is experimentally well investigated for several Transformers and language tasks\n- The authors also introduce small methodological novelties (e.g., retrieval warm-up)\n- Fluently written and error-free language, in particular very informative headings, and honest about how broad or limited the claims and evidence are\n\n\nWeaknesses:\n- The important side hypothesis (inputs should not influence each\nother) is only speculated about (\"we hope these transformations [...] reduce interference\", \"may implicitly enable [...] instance-specific computations\"), with only limited experimental (toy case in 4.2(R2)) and theoretical support (4.4 argues that the family of functions a Transformer comprises contains some with independent processing, but not whether these are actually picked during training) (see also question 1)\n- Another side claim is that vanilla Transformers with high batchsize (i.e., achieving high throughput) are limited due to their memory consumption, but the memory consumption overhead of the proposed method is not measured and compared to them (see also question 4)\n- The experiments for MLPs and CNNs on MNIST are very simplistic. I appreciate them as outlook, but would appreciate being more humble with them in abstract (like, e.g., done in the introduction)\n- The results presented in the abstract appear cherrypicked since the weaker B1 was chosen as baseline\n- The experiments appear not to be repeated on several random seeds\n- The paper opens a novel field with both ethical chances (reduction of computational consumption) and risks (attacks due to confounded processing), and would benefit strongly from discussing them\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7K88uk3Nmke", "reviewer": "Reviewer_RBia", "summary": "The authors explore the idea of data multiplexing to accelerate both training and inferencing of neural networks. A data multiplexer is added at the input of the network to multiplex or \"mix\" N samples into a single one. The network processes this single sample, and then the final answer is demultiplexed near the output of the network back to N answers. Potentially, the method can achieve a theoretical speedup of N times over the original network. The multiplexer and demultiplexer are differentiable and hence trainable. The multiplexer applies a linear transform to each sample and averages per position in a sequence. The demultiplexer is applied for each position to recover the N sequences. During training gradients seem to be superposed or mixed up, failing to converge for Transformers. An auxiliary loss is proposed to overcome this issue. For transformers, the authors share an explanation why is possible to mutiplex the data. In the experimental section, the idea is evaluated on NER, sentiment analysis, sentence similarity, and NLI. The models include Transformers, CNNs and MLPs. The results show typically a drop in performance when N increases.\n\n", "questions": "* What is the name of the model used from the Huggingface hub? \n* What does cause in some cases that for increasing N, the model improves in performance (e.g., SST-2 N=40 for Ortho+Index Emb)?\n* What is the original performance of the model (i.e., N=1) in each plot in Figure 3? ", "limitations": "To the best of my understanding, limitations have been addressed. ", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of data multiplexing is not new (as pointed out in the text), the application to deep neural networks seems novel. This leads to a very interesting work presented in the paper. This work setups the ground for further improvements and novel methods in this community as well. In addition, I appreciate the practicality of the method, including the discussion on the transformers and the analytical analysis on their properties for exploiting multiplexing. The experiments are convincing and show the limitations when N is large, suggesting there is room for improvement.\n\nThe paper is nicely written and clear, as well as supported by nice diagrams. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept