Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Similar to MIMO in wireless communication, multi-input multi-output training improves network performance by optimizing multiple subnetworks simultaneously. Previous MIMO network architecture uses CNN. This paper proposes MixViT, the first MIMO framework for vision transformers which takes advantage of ViTs’ inherent mechanisms to share features between subnetworks. Unlike MIMO CNN, MixViT only separates subnetworks in the last layers leveraging a source attribution that ties tokens to specific subnetworks. The paper shows that MixViT can have significant gains across multiple architectures (ConViT, CaiT) and datasets (CIFAR, TinyImageNet, ImageNet-100, and ImageNet-1k).
strength_and_weaknesses: Strength
1. This paper proposes MixViT, the first MIMO framework using vision transformers which takes advantage of ViTs’ inherent mechanisms to share features between subnetworks. 

2. The paper shows that MixViT can have significant gains across multiple architectures (ConViT, CaiT) and datasets (CIFAR, TinyImageNet, ImageNet-100, and ImageNet-1k).

Weaknesses
1. It is not clear how MIMO training helps representation learning. The paper lacks analysis on this. Does MIMO training improve hard example classification accuracy? Does it better cluster images into different clusters? Does most of the gains come from input mixing such as CutMix?

2. The evaluation is on relatively small datasets. The paper should evaluate on larger datasets such as ImageNet.
clarity,_quality,_novelty_and_reproducibility: The paper uses certain phrases without clear definition, e.g. in "MixViT modifies the traditional MIMO structure to take full advantage of ViTs’ propensity to mutualize features between subnetworks while still retaining the advantage of training distinct predictions", it is not clear exactly what mutualize features mean. Can you show an example?

It seems MIMO networks using vision transformer is new. 

summary_of_the_review: MIMO networks with vision transformer seems to novel. However, the paper does not provide deeper analysis on how it helps representation learning, besides the data augmentation benefits from input mixing, e.g. CutMix. The evaluation is also inadequate as it does not provide results on large datasets such as ImageNet.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a transformer-based MIMO framework, called MixViT. At training time, two subsets of tokens from two images are taken as input and the model is tasked to predict labels for the two input images. At test time, the same input is passed to the model multiple times with different sourced tokens. The features in the earlier layers are shared while two separate encoders, called source attribution, are introduced in the later layer for the input-dependent feature extraction. In this way, the model is expected to achieve feature sharing and separation for multi-prediction. Improved results are obtained on image classification tasks compared with the single input counterparts and baselines. 
strength_and_weaknesses: Strength
- Consistent improvements are achieved on the task of image classification.

Weaknesses
- The technical contribution seems limited. The input mixing for training is simply adopted from CutMix, and the source attribution appears to be quite straightforward. 
- The model needs to run multiple times for the inference, causing a significant increase in the computational cost.
- The model is claimed to be multi-input and multi-output. However, it only supports two inputs and outputs, and the model needs to be changed for a different number of inputs, and it appears to be non-trivial to extend this model for more than two inputs and outputs. Hence, the model is not scalable and flexible for a different number of inputs and outputs.
- As this is an architecture paper, it would be more convincing to also validate the effectiveness on other tasks like segmentation and detection, for which the model seems not directly applicable.
clarity,_quality,_novelty_and_reproducibility: - The motivation is clear and the method is technically sound, but the technical contribution is somewhat limited as discussed above.
- The writing is generally clear.
summary_of_the_review: Overall, I think the proposed method sounds interesting and good performance is also achieved. However, the technical contribution is somewhat limited as discussed above, and the weaknesses overwhelm the strength. Hence, I lean to recommend rejection based on its current shape.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to bring MIMO for CNNs to vision transformers. Specifically, it designs a source attributing module at last layer to separate the input source and perform classification for different input. Experiments are mainly conducted on small-scale datasets, CIFAR, Imagenet-100 to show its effectiveness. 

strength_and_weaknesses:  Strength:
		Many small-scale datasets are used for validating the results. And the paper proposes one working example of MIMO for vision transformers. 

	Weaknesses:
		The paper introduces one way of bring MIMO to vision transformers, which is quite straightforward and novelty-limited, 
Source attribution is a direct analogy of using different input projection in existing MIMO.  But there exist many other natural designs that can potentially work for vision transformers. The study space in the paper is not enough to show this proposed way is better.  i.e. important baselines are missing. For example, use different patch embedding for different inputs (similar to different project for inputs in MixMo paper). 
		Again, this specific design of late source embedding is not thoroughly studied. Early source embedding with carefully designs comparison is missing. Or middle source embedding, etc, this design should be carefully studied.
		Experiments are mainly conducted on small-scale datasets, ability to adapt to other large-scale datasets and large models remain unknown. Table 3 shows CaiT S-24 and M-24 results are 82.3 and 82.7. However, table 3 in CaiT paper shows s-24, m-24 results are 82.7 and 83.4 respectively. Can authors check the numbers here? And provides some large model (CaIT-M, or other large popular ViT variants) results would be better.  

clarity,_quality,_novelty_and_reproducibility: The clarity is good, novelty is limited as the method is a direct adaption of existing MIMO to vision transformers. Reproducibility is good as authors provides many implementation details. 
summary_of_the_review: 
Overall the paper is a direct adaption of MIMO to vision transformers. However, the design space is not thoroughly studied, and current results are only on small-scale dataset and small models, raising the concern regarding if current design is optimal and can adapt to other large models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the use of the multiple-input multiple-output (MIMO) paradigm for vision transformers. The authors propose to train a ViT by using two images A and B at a time – the two images are tokenized and the tokens are “mixed” by randomly selecting tokens from image A or B. The mixed tokens are then passed through the ViT, and a proposed “source attribution” layer at the top allows the network to predict the classes of both images A and B. Experimentally, the authors show that 1) their network beats previous small networks reported in the literature on small-scale datasets (table 1), 2) the proposed MIMO architecture gives improvements across datasets and architectures (table 2 and 3) and 3) that the proposed MIMO architecture beats previous MIMO architectures for ViTs (table 6).


strength_and_weaknesses: Strengths:

* The proposed method generally gives solid gains in known benchmarks.
* The authors give clear ablation experiments which show that their methods are better than other MIMO methods for VITs (table 6).

Weaknesses:

* The novelty is low.
* Some results do not control for the training recipe, e.g. numbers in table 1 might have vastly different hyperparameters. 
* It is not clear if the method improves SOTA models.
* The paper mostly focuses on small-scale datasets. It is known that VITs work best for large-scale datasets.
* The clarity when explaining the proposed architecture could be improved.

clarity,_quality,_novelty_and_reproducibility: The proposed method is not clearly explained, and the figures do not really help. Most captions are only a sentence or so – see e.g. Figure 3. This makes the figures hard to interpret. I would encourage the authors to add some PyTorch/NumPy/Jax pseudo-code for the network architecture. 

For table 1, the hyperparameters might be so different that it is hard to compare the methods. The number of parameters can also be misleading since many authors change the resolution when training networks – and higher resolution typically leads to better scores.

I would also like to ask, are all numbers in table 3 from you? Do the methods in table 3 all use the same hyperparameters? If not, the different performances might simply come from different training recipes instead of the MIMO architecture. I would encourage the authors to use the original code from the VIT authors and then make sure that the proposed method can bring clear improvements over the official code. I am not convinced that the proposed method would actually improve SOTA methods. The best result would be to improve the SOTA number for e.g. Vit-base on imagenet. 

Mixing two input images on the token level seems like a simple adaptation of cut mix. Could you add some ablation experiments that test how the method compares against cut mix and if the improvements from the two methods are orthogonal?


summary_of_the_review: The authors present a natural extension of the MIMO framework for ViTs and show how this can improve performance for various vision datasets. While the empirical results are positive, it is not clear that all factors are controlled for and that the proposed method would actually improve SOTA models. Furthermore, the novelty is relatively low and the proposed method is not explained clearly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The authors propose a way to train a practical MIMO method that has feature sharing. They use the ViT architecture and image classification benchmarks for their experiments. The main idea is to mix the tokens in the input, use a shared encoder (that allows for feature sharing), and use source attributions in the final layers to perform classification.

Strengths: 
- Clever use to ViT model's ability to share features. 
- MixViT shows improvements on many small-scale image classification benchmarks (Table 1). 

Weaknesses: 
- A majority of the reviewers found issues with the writing and presentation of the paper, and remain unconvinced even after the author response. On reading the paper, the AC agrees that the important details only become clear after reading the author response. We would encourage the authors to improve the presentation
- Training recipes in Table 1 are not controlled for, which makes the comparison unclear.


justification_for_why_not_higher_score: The rating of the paper could be improved by the presentation quality and improving the main experimental setting in Table 1.
Decision: Reject