Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new CNN architecture: ConvMixer, which integrates some techniques/techniques used in the recently proposed visual transformer model/mlp model. ConvMixer is simple and effective, and it also prompts us to think deeper about where is key to improving performance in the vision transformer and mlp models.
main_review: Pros:
Well written. Clear motivation and the problem raised in this work is very interesting.

Cons:

a. Although the proposed ConvMixer illustrates very promising results compared with its counterparts, such as DeiT, MLPMixer, and ResNets, the main claim of this work:  "the patch representation may be the most critical component" has not been sufficiently well validated: 
- Compared to other previous convolution-based network architectures, such as ResNet, the ConvMixer has too many factors beyond the patch representation itself that may lead to performance differences, such as data augmentation, appropriate hyper-parameters, optimizer and other macro/micro network architectures, For example, the ConvMixer adopts the spatial-conv + pointwise-conv as its basic building block, which is also an unfair comparison with ResNets. In addition, through carefully tuning, [1] reports a much higher ResNet performance than this work.

- Compared to vision transformer/mlp-based networks, this work only considers some computation/parameter unfriendly architectures, such as DeiT, while ignoring some recent progress in this field, like Swin. 

- Lack of comparison for downstream tasks, such as object detection/semantic segmentation. This is necessary to fully understand the importance of patch representation in a broader range of tasks.

These under-explored factors challenge the conclusion/connection of this work that the 'patch representation' is the key.
 
[1] ResNet strikes back: An improved training procedure in timm

summary_of_the_review: This work is well written and presents a very interesting conjecture/conclusion that the patch representation may be the key to the success of visual transformers and visual Milp. However, its research is not strong enough to support its claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed a new design for the image classification task named ConvMixer, which brings the idea from CNN to Visual Transformer. Unlike previous ConvNets, Transformer-based models, and MLP-based models, ConvMixer simply applies depth-wise (with skip connection) and point-wise convolutions on the patches. The key point authors claimed is that using patch embeddings is a powerful and important takeaway besides the architecture design. Experiments on ImageNet demonstrate the effectiveness of the proposed model.
main_review: Strengths: 
1.	The model is interesting and simple, which would inspire the vision community a lot.
2.	The results are promising, especially on the accuracy. Even a simple depth-wise / point-wise convnet without resolution reduction can achieve a promising result.
3.	An interesting but not discussed finding is that, different from VIT  and MLP-mixer (and variants) that emphasize the global context, this work did not introduce global context (although the receptive field is large enough). A close look at ResNet and BagNets [1] would be interesting.

Weaknesses: 
1.	Typos, e.g. , “simiar” in page 2; “do ConvMixers” in page 8, etc.
2.	At the end of Section 1, the authors claim that “This suggests that, at least to some extent, the patch representation itself may be the most critical component to the superior performance of newer architectures like Vision Transformers.” I don’t think this is true, and the authors have not fully verified such a claim. There are many other factors, like learnable parameters, effective receptive fields, etc. This statement seems to be overclaimed.
3.	Most importantly, I don’t think this method achieves “better” results than others.  i), Comparing with ResNet152, the throughout is 89 vs. 872. The slow speed of ConvMixer is not acceptable in real-world applications; ii) Besides, the performance of ResNet would be higher than reported if trained with appropriate strategies [2].  iii) Small patch properly can achieve better performance.  I would suggest authors try some other Transformer models, like Swin Transformer [3].
4.	In future work of page 4, authors claim that “We are optimistic that a deeper ConvMixer with larger patches ... and hyperparameter tuning”. However, this may not be true. Please see Appendix A. I have not seen the strong superiority of ConvMixer over the ResNet in the accuracy with similar parameters. And the ConvMixer’s speed is far below the ResNet.
5.	In VITs and MLPs (including their variants), we have two skip connections (it is not called residual connection as shown in Fig 2) in a block. However, authors designed with only one (tuned on cifar10). I just want to know if we add two skip connections in ConvMixer, what is the result for ImageNet.
6.	I’m not a fan of cifar10 dataset due to the dataset size and inevitable randomness in results. Considering the limited computational resources, some other datasets may be helpful (like cifar100, Tiny-ImageNet, etc.) or authors can report the mean/std results of 5 runs on cifar10.
7.	I could not get the idea of “A note on paper length” paragraph in a conference paper, considering some of the important experiments are missing. It is nothing to do with the main idea of ConvMixer.
8.	Some important ablation studies are missing, like the influence of depth, kernel size, patch sizes, etc (I mean detailed ablation studies). I would also be interested in the performance of ConvMixer on some downstream tasks like detection. 

[1] Brendel, Wieland, and Matthias Bethge. "Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet." International Conference on Learning Representations. 2018.

[2] Wightman, Ross, Hugo Touvron, and Hervé Jégou. "ResNet strikes back: An improved training procedure in timm." arXiv preprint arXiv:2110.00476 (2021).

[3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103.14030 (2021).

summary_of_the_review: I recommend reject due to the weaknesses above, and I will consider updating my score if seeing a strong rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a very simple architecture which consist of patching the input image and then applying a combination of depth-wise and point-wise convolutions. In the paper, authors evaluate the performance of this model when used for image classification. In their main experiment, they train the architecture using only the ImageNet-1k dataset. On that experiment, they show how their simple architecture is competitive with state-of-the-art architectures. 
main_review: **Summary**: Overall, I believe this is a good paper. It consist on a simple model which is very well described and one single experiment which shows well the advantages of the paper. However, I believe the authors are missing a few points which are worth addressing before the paper is published.

**A note of the paper length**: I understand the message from the authors about the paper length, but as a reviewer, I needed to check the Appendix many times to produce a complete review of the paper. Although I can agree this is a good discussion to have as a community, I don't think the note from the authors is fair as if reviewers only looked at the main paper (which is supposed to be sufficient for a full understanding of the paper), many relevant points would be missed. For this reason, I would recommend the authors to move to the main paper some sections which provide relevant insights (for instance, Appendix A gives very relevant details). 

**Strengths**:
1. The model is very simple and effective. I agree with authors that this is a relevant piece of research for the community, as it disentangles a bit the performance improvement from the transformer architecture.

2. In terms of reproducibility, I believe this can be a great baseline for future research. Code is simple and authors seem to imply that it does not need a lot of hyper-parameter tuning to make it work.

3. The model description is clear and well written. I think the figures and the text are very clear and the community will really appreciate that.

4. The experimental results on ImageNet-1k are very good. Specially, when taking into account the parameter count, the model seems to show very good performance overall.


**Weaknesses and proposals for improvement**:

1. Patches and architecture: authors introduce the model as validation of the idea of the patches being important for final performance on SOTA architecture such as transformers (see title and abstract). However, in the paper they introduce the idea of using patches in convolutional networks along with a novel architecture, but they never decouple the gains of this two contributions in the experimental section. If authors want to claim that patches are relevant for final performance, in my opinion they should ablate only that change in an already known architecture. 

2. Although the experimental results in the paper are encouraging, in my opinion authors are missing a very relevant point about pre-training. It is clear that transformers do need large scale pre-training for them to get high performance. I would be very curious to see this model operating on that data regime. Otherwise, I am not sure if we can conclude much, as it was already known that these large models were not very good with regimes without much data. 

3. The first paragraph of the experimental section is a bit confusing for me. If authors want to include the CIFAR experiments in the paper, they should just include that table in the main paper. If they think those experiments are not relevant, then they should not start the experimental section by discussing those experiments. But providing the table in the Appendix but some discussion in the main paper is very confusing for the reader. 

4. Other than large-scale pre-training, I would be curious to see the performance of this model on other classification task. Authors include CIFAR, which is a small dataset. But I think it would be useful for the community to benchmark the proposed model to other classification tasks such as scene classification, fine-grained recognition, etc. Essentially, evaluate the ability of the model to perform on other domains and make sure the results are not overfitting to object classification.

5. Following up with the previous points, isotropic models should be good to perform segmentation-like tasks, as they keep most of the input resolution (except for the patching). I am wondering if that could be an additional experiment to include. 

6. Looking at the results, it is clear that the model has lower throughput than its competitors. Figure 1 revolves around the parameter count of the model, but I believe it would be fair for the authors to include also some discussion about the throughput. I realised they had included some in the Appendix, but I believe authors should include a proper discussion on the effects of this low throughput in the main paper, as I believe it's one of the main weaknesses of the model. 

summary_of_the_review: I believe the paper is good and it should be accepted. It will be a relevant contribution to the community as a baseline as well as a good datapoint to improve our understanding of SOTA models. However, I believe the paper would be more complete if authors evaluated in additional task. Furthermore, I think authors should address some relevant points when comparing to other models such as data scale and throughput.
review_title: null

Meta-review: This paper observes that a fully-convolutional model in the style of recent MLP-Mixer and ViT variants can have surprisingly good initial performance. As this paper attracts certain amount of attentions, three expert reviewers have provided very detailed and serious comments, and two actively engaged with author discussions. AC also carefully read the paper as well as all discussion threads.

AC agrees the authors should not be penalized by not achieving the best performance, nor not comparing with very recent work. The main legitimate critiques, however, focus on three aspects: (1) over-claimed contribution; (2) experiment solidness/competitiveness; and (3) writing completeness/clarity.

First, this paper established an interesting ablation experiment that a very simple model, that uses only standard convolutions to achieve the mixing steps, can roughly "do the work". However, AC disagrees this is a very "surprisingly new" result, on top of MLP-mixer: given convolutions are increasingly re-injected into ViTs to gain the vision inductive bias, their similar role in MLP-mixer should be expected too. Moreover, as in general agreement by reviewers, the paper title might have over-claimed - the authors cannot directly prove this concept "patch is the most critical component" yet. The authors later also agreed and changed some confusing wording, which is a good move (but also, making their contribution now even less obvious). 

Second, this method does not achieve noteworthy competitive results compared to others, in order to justify its merit (simplicity alone is good to have, but insufficient to justify a strong work). Importantly, it has been pointed out by two reviewers that the model throughput is much worse than the competitors. AC also noticed that the comparison was not very rigorous, e.g., comparing ConvMixer patch size 7 with DeiT-B 16 patch size 16 doesn't help draw much fair informative conclusion. The cifar-10 results alone did not provide strong support and were later de-emphaszied by authors too. 

Third, while NOT being the main reason of rejection, AC personally suggests the authors to responsibly enrich their main text, and to remove the  “A note on paper length” paragraph. The authors intentionally kept the paper length unusually short. Reviewers generally dislike this idea. Being an innovative writer is good, but very relevant details and discussions were left in the supplemental as a result. Especially, AC agrees the whole section A and part of section B of the supplemental should have been in the main paper at very least.

In summary, the authors strive to tell an interesting story, but it is not yet a well settled story. The experiments are not solid enough to support their bold claims. The authors are suggested to improve their work further by taking into account reviewer comments.
Decision: Reject