Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The basic idea is as follows: multiplex multiple inputs into one input of the same size, then feed the model's output into a module at the end that untangles the multiplexed representations and makes a prediction for each input individually.

The multiplexing module randomly projects inputs then does an element-wise average. Demultiplexing is achieved by feeding the outputs into a demultiplexing function, which is either a set of N MLPs (where N = number of multiplexed inputs), or by prepending each input sequence with an index token such that a single demultiplexing head can use the index embedding to distinguish between inputs. To work with sequenced inputs, multiplexed models need to be pretrained on a self-supervised “retrieval” task which requires selecting the correct token for each possible position across the set of input sequences. The authors primarily evaluate their method on transformers and NLP tasks, and find that multiplexing results in modest accuracy drops but large throughput increases.
strengths_and_weaknesses: Strengths: A clever method that I am somewhat surprised the authors were able to make work. The results are clearly presented. The inference speedups appear meaningful.

Weaknesses: I'm not sure how relevant the theory is. Some of the stated motivations are not very thoroughly addressed by the analyses. Total resource savings is unclear.

I think the paper is acceptable in its current form, and I think it would be excellent if revised appropriately.
questions: Overparameterization/unused model capacity are presented as motivations for the development of and reason for success of the method. However, I don't think the analyses in the paper justify this sufficiently. First, the results for the small and baseline models should be compared directly. The following experiments would more thoroughly answer this question: Reduce the model size until performance begins to decline substantially. If model capacity is why multiplexing "works", then the multiplexing models' performance should decline sooner than the non-multiplexing models' performance. The same result should be observed when pruning model weights: performance as a function of weights pruned should be more favorable for non-multiplexed models than for multiplexed models.

The theoretical examination of multiplexing is interesting. I encourage the authors to test these theoretical predictions: for example, do the singular values of the query and key matrices differ between the baseline and retrieval-pretrained networks in the manner predicted by the theoretical examination? Furthermore, if reducing the number of attention heads has little effect on multiplexing capability, does it even make sense to focus the theory on the role of attention heads in multiplexing?

The authors demonstrate that pretraining is necessary for learnable multiplexing, and that retrieval pretraining can affect downstream task performance, but they don’t demonstrate that retrieval pretraining per-se is necessary for learnable multiplexing. While I suspect it is, I think it would be useful to show that other types of pretraining do not affect the learnability of multiplexing.

This method is presented as providing a resource savings due to the inference speedup. However, assessing the resource savings is difficult for a number of reasons. First, the pretraining protocol does not seem to be discussed at all other than a description of the task. Second, multiplexing causes an accuracy decrease relative to baseline models. Is it possible to compensate for this decrease by e.g. training longer? It would be worthwhile to see accuracy vs. training time plots for the different training configurations.

Is it justified to compute speedups for small transformers relative to the baseline size transformer? This confounds the effect of transformer size on speedup with the effect of the multiplexing in small models on speedup.

Is it possible to learn the multiplex projection?

limitations: The authors do a satisfactory job addressing the limitations of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors explore the idea of data multiplexing to accelerate both training and inferencing of neural networks. A data multiplexer is added at the input of the network to multiplex or "mix" N samples into a single one. The network processes this single sample, and then the final answer is demultiplexed near the output of the network back to N answers. Potentially, the method can achieve a theoretical speedup of N times over the original network. The multiplexer and demultiplexer are differentiable and hence trainable. The multiplexer applies a linear transform to each sample and averages per position in a sequence. The demultiplexer is applied for each position to recover the N sequences. During training gradients seem to be superposed or mixed up, failing to converge for Transformers. An auxiliary loss is proposed to overcome this issue. For transformers, the authors share an explanation why is possible to mutiplex the data. In the experimental section, the idea is evaluated on NER, sentiment analysis, sentence similarity, and NLI. The models include Transformers, CNNs and MLPs. The results show typically a drop in performance when N increases.


strengths_and_weaknesses: The idea of data multiplexing is not new (as pointed out in the text), the application to deep neural networks seems novel. This leads to a very interesting work presented in the paper. This work setups the ground for further improvements and novel methods in this community as well. In addition, I appreciate the practicality of the method, including the discussion on the transformers and the analytical analysis on their properties for exploiting multiplexing. The experiments are convincing and show the limitations when N is large, suggesting there is room for improvement.

The paper is nicely written and clear, as well as supported by nice diagrams. 


questions: * What is the name of the model used from the Huggingface hub? 
* What does cause in some cases that for increasing N, the model improves in performance (e.g., SST-2 N=40 for Ortho+Index Emb)?
* What is the original performance of the model (i.e., N=1) in each plot in Figure 3? 
limitations: To the best of my understanding, limitations have been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed DataMUX, a method to allow DL models to process multiple inputs at the same time using a single representation. They conduct their experiments on token and sentence level classification for Transformer, MNIST for CNN and MLPs.
strengths_and_weaknesses: Strengths:
  1. The motivation here is sufficient, especially the connection to biological perspectives.
  2. Their experiment results show data multiplexing can speed up the throughput while maintaining performance.
  3. It is great to have a theoretical explanation of the proposed method.

Weaknesses:

  1. Please refer to the questions

I would recommend this paper be accepted. Overall, I think this paper has solid motivation from both theoretical and intuitive perspectives. The authors conducted many experiments to show that DataMUX is effective in many tasks. I think there is a lot to explore with the connection between over-parameterization and multiplexing, too, that would be interesting to other researchers.
questions: 1. From the experiment, it seems like a single representation can actually contain much information from multiple inputs, which is even enough to decompress it(demultiplexing). So is it possible that a traditional Bert [CLS] token can achieve the same thing with multiplexing pretraining?
2. From section 4.4, even though the self-attention-based models are suitable for multiplexing in certain cases, it seems like it still requires multiplexing pretraining to reach the desired parameters. Therefore, I guess multiplexing and demultiplexing layers cannot just be added to any type of common language model right(e.g. Bert, Roberta)?
limitations: The authors addressed the limitations and potential negative impact of their work in the appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to train a single encoder (mainly Transformer, but also CNN, MLP) to process multiple samples in parallel with the goal of approximating batch-processing without the computational overhead added by default batchwise processing. To this end, they first mix the input samples together (and add order tokens), then apply the encoder, implicitly forcing it to learn a form of parallel processing of this joint representation, and then de-mix the representation to obtain the prediction for each individual sample. Experiments indicate that Transformers may be flexible enough to process up to 20 samples in parallel (giving a 11x speed increase) while only losing 2-6pp.  
performance.

strengths_and_weaknesses: Strengths:
- I see the main value of this work in formulating a useful, novel task and taking a first shot at it (conditional on it being a new task, which I cannot confidently ascertain since it is not my primary
field)
- If the model is able to guarantee independent processing of the inputs, this might have impact on causality and disentanglement literature, as a form of independent mechanisms
- The main hypothesis (efficiency increase without big drops in
performance) is experimentally well investigated for several Transformers and language tasks
- The authors also introduce small methodological novelties (e.g., retrieval warm-up)
- Fluently written and error-free language, in particular very informative headings, and honest about how broad or limited the claims and evidence are


Weaknesses:
- The important side hypothesis (inputs should not influence each
other) is only speculated about ("we hope these transformations [...] reduce interference", "may implicitly enable [...] instance-specific computations"), with only limited experimental (toy case in 4.2(R2)) and theoretical support (4.4 argues that the family of functions a Transformer comprises contains some with independent processing, but not whether these are actually picked during training) (see also question 1)
- Another side claim is that vanilla Transformers with high batchsize (i.e., achieving high throughput) are limited due to their memory consumption, but the memory consumption overhead of the proposed method is not measured and compared to them (see also question 4)
- The experiments for MLPs and CNNs on MNIST are very simplistic. I appreciate them as outlook, but would appreciate being more humble with them in abstract (like, e.g., done in the introduction)
- The results presented in the abstract appear cherrypicked since the weaker B1 was chosen as baseline
- The experiments appear not to be repeated on several random seeds
- The paper opens a novel field with both ethical chances (reduction of computational consumption) and risks (attacks due to confounded processing), and would benefit strongly from discussing them

questions: - If I understand correctly, the goal is that the processing of each input should not influence each other input. Does this hold theoretically (e.g., decompositional Jacobians) or experimentally (e.g., resampling inputs and recording differences in the output)?  
More drastically formulated, how much can the prediction of one sample be influenced by the others in its batch? Does this open an ethically concernable weakness to attacks?
- In this paper, the architectures (Transformers, CNNs) are viewed from a universal approximator "black-box" perspective. Which inductive biases implemented by these architectures motivate this claim, and which threaten to deteriorate the desired independence?
- The model is trained on a fixed batch size. Can we hand over an incomplete batch to the model, e.g., by zero-padding or repeating the input (like we can for normal batch-processing)?
- The two plots in Figure 4c show the same, only with inverted axes. I would find it more useful to use the space for additional information, such as increase in memory complexity (which was claimed to "not rise substantially"). Maybe consider a Pareto chart. This would also allow showing several baselines, i.e., B1 and B2 transformers with several batch sizes
- I do not quite understand the Retrieval warm up: Why do we want to predict the whole input representations from the output, as opposed to just their order token?
- Please provide a reference to the Index Embeddings Demultiplexing strategy in line 157.
- A minor cosmetic critique is that plots are referenced to as, e.g., 4a, 4b, while the plots themselves are not labeled with a and b
- Will code to reproduce the experiments be made available upon acceptance?

limitations: - Most technical and experimental limitations were adressed humbly and openly (see strength 5)
- The paper opens a field with diverse positive and negative ethical implications which remained unadressed (see weakness 6)
review_title: null

Meta-review: metareview: The paper applies DataMUX  for deep learning approach, which enables the network to process multiple inputs.  The idea is very interesting and the paper is well written.  All three reviews suggested the paper should be acceptable. 
confidence: Certain
Decision: Accept