Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors attempt to generalize linear layers with layers in which weights are produced by a function on inputs, motivated by the success of Transformers. Motivated by the CP decomposition, they extend the proposed layer to be more computationally efficient, and suggest a variant of a convolutional neural network architecture. They validate that the proposed architecture works on par with common approaches in image classification and object detection benchmarks.
strength_and_weaknesses: - the proposed generalization of a linear layer provides a simple novel view on convolutional and Transformer networks

Experimental validation is lacking.
- while the authors propose a generalization of linear layers and Transformers, the evaluation is done only on computer vision benchmarks. Applying the proposed approach to non-computer vision tasks would provide stronger evidence for the claims
- in the puzzle reconstruction experiment, it is not clear what information is provided by the Convolutional, Spatial mix, Pooling and Channel mix curves. As the task requires global receptive field, it is expected that the layers without it would be able to the reconstruction task. A convolutional network with more layers should be able to handle the task as well. A comparison with linear layer only should be sufficient to validate the claims.
- in object detection, 1x schedule is not sufficient for experimental validation, as some networks might need more iterations to reach peak performance.
- citations [1] and [2] are missing. These can also serve as stronger baselines for the experiments.
- it is not clear is the results in Table 1 are statistically significant, or are within std of training with different random seed.

[1] Wang et al., Non-local Neural Networks
[2] Hu et al., Squeeze-and-Excitation Networks
clarity,_quality,_novelty_and_reproducibility: - the paper is well written and is easy to follow
- the proposed architecture is simple and should be easy to implement and reproduce the experiments.
summary_of_the_review: While the attempt to generalize linear layers and connect with Transformers is valuable, it seems like the resulting architecture is simply another variation of convolutional neural networks. A stronger and more thorough experimental validation is needed to validate the claims and novelty.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed ChoP’D Formers based on CP decomposition representation. Some modification is implemented on the proposed "Former". And experiments show some comparable results.
strength_and_weaknesses: Pros:

- Have a try to unify nowadays models.

Cons:

This paper seems to be organized by (1)firstly unifying various networks; (2) then denoting networks with a CP format; (3) lastly designing ChoP’D to achieve a good performance on benchmarks. However, I have some concerns:
  - The unifying strategy is not strict and not novel. The paper claims to unify convolutions, fully-connected layers, and transformers into a category "Former". 
    - The Transformers are not suitable for this category since they have non-linear structures (namely self-attention that has softmax function). In addition, this paper claims "Without lack of generality, we omit at this stage the Layernorm (LN) applied before every block and the residual connections". As the softmax and LN functions are both important components in a network, omitting them is not suitable. Therefore, it is unreasonable using such loose conditions to group Transformer and CNNs into one category.
    - For the CNNs and fully-connected layers, [1] has already unified them. And [2] also gives tensorial CNNs a unified representation to initialize them in one scheme. Therefore, the idea of unifying CNNs is not novel.
  - Unifying models into a CP format is already proposed by [3]. Equation (5) in this paper is the same as Equation (2) in [3]. Thus, such CP representation is also not novel.
  - For the modification of the Former to construct ChoP’D Formers, there seems some limited novelty.

- Unreadable writing:
  - This paper uses improper uppercase word formats like "Dynamic and Spatially Adaptive". Simply using them in lowercase is ok. For example, "Dynamic and Spatially Adaptive" -> "dynamic and spatially adaptive";
  - Grammar errors like "Vision Transformers (ViT) success has long ..." -> "The success of Vision Transformers (ViT) has long...";
  - Overlong paragraphs:
    - "Puzzle Reconstruction" on Page 7;
    - "Classification, Segmentation, Detection" on Page 8.
  - Inconsistent descriptions like "fully-connected" and "Fully Connected";
  - The quotation mark in Latex is written as `` '';
  - The citation should use citep and citet.

- For the performance, the proposed ChoP’D Formers seem only derives some comparable results to baselines, which are not significant. So, what are the remarkable advantages of ChoP’D?


Overall, regarding the above points, I vote for "rejection".

[1] Kohei, Hayashi, et al. "Exploring unexplored tensor network decompositions for convolutional neural networks." Advances in Neural Information Processing Systems 32 (2019).

[2] Yu, Pan, et al. "A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks." International Conference on Machine Learning. PMLR, 2022.

[3] Jean, Kossaifi, et al. "Factorized higher-order cnns with an application to spatio-temporal emotion estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
clarity,_quality,_novelty_and_reproducibility: Clarity: Poor.

Quality: Poor.

Novelty: Medium.

Reproducibility: N/A.
summary_of_the_review: The paper proposed a newly designed model based on CP decomposition. However, I still have some concerns as mentioned above. Therefore, I tend to give "rejection".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper first delivers a general five-dimensional tensor operator that can generalize many batched single-input layers that our community is frequently using. Then, the authors decompose the tensor with CP-decomposition; using this low-rank approximation of the tensor, they propose an architecture Chop'D former. Another low-rank approximation of self-attention is used (SAT) to lower the complexity. The architecture's performance was on par or better on the benchmarks like COCO detection/segmentation and ImageNet classification for small parameter models (~30M)
strength_and_weaknesses: - (C1) It was a pleasure to read a paper trying to minimize mathematical obfuscation by employing concise einstein notation.
- (C2) I agree with the claim that the five-dimensional tensor W in the paper is general enough to represent all trending models of the community as described.
- (C3) A study for how low-rank r affects the performance of ChoP'D former is still needed. I even failed to find what r the authors used for the experiments in the main manuscript and the supplement material.
- (C4) The tensor W is only for a single-input layer. I am curious how the multiple-input layer, like cross-attention layers, could be formulated with Einstien-tensor notation. Plus, if possible, how can they be CP-decomposed (and its effectiveness too.) Maybe converting the original transformer architecture (with a transformer decoder) for machine translation would be a great starting point.
- (C5) Performances for major downstream tasks: COCO detection/segmentation and ImageNet classification are notably good. However, there is a report [1] that approximation methods tend to perform better on the small-parameter regime and worse on large-parameter models. Thus I want to see how Chop’D former works on at least a BERT-base scale (12 blocks).
- (C6) Typo: wrong double quote character on page 2.

[1] Tay, Yi, et al. "Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?." *arXiv preprint arXiv:2207.10551* (2022).
clarity,_quality,_novelty_and_reproducibility: - (Clarity) The paper is clearly written and easy to follow.
- (Quality) The experiments were well organized and clearly addressed the research questions brought throughout the paper.
- (Novelty) The novelty is limited in the sense that technically what the paper proposed is a low-rank approximation of existing architectures.
- (Reproducibility) Source code is not provided, but implementation details and the pseudo-code in the appendix provide reproducibility.
summary_of_the_review: It would be nice if some of my questions (see C3-C5) were answered. I want to give this paper a borderline accept recommendation because I believe the contributions that this simple and universal notation will bring to the community would outweigh the limited novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel neural network layer architecture, which is derived by a tensor rank decomposition (CP decomposition) to a generalized version of a fully connected layer (namely, weights tensor of the FC that is created as a general function of the input). This new layer by design can be dynamic and spatially adaptive, it can create global receptive fields, and can mix channel information, therefore can accommodate the tensor operations for many of the existing known NN layers, such as self-attention or convolution layers.

While the full version of the proposed generalized fully connected layer is costly and has many parameters, its CP decomposion version has reduced computational complexity that makes it comparable with SOTA vision backbones, such as Swin-Transformer and ConvNextT models.  The authors conduct experiments on ImageNet-1K and show comparable top-1 accuracy as well. 

strength_and_weaknesses: Strength:

(*) The paper proposes an interesting perspective on formulation of neural network layers

(*) The idea of forming the proposed generalized fully connected, and then apply CP decomposition is novel as far as I know, and interesting. The modifications proposed for the decomposed components to make it more practical, such as using SAT, or combining gating operators, are also contributing to the technical novelty of the paper. 

(*) On the image classification task, the performance of the proposed model is comparable to SOTA. 
    
Weaknesses: 

(*) The paper proposes a novel layer and model that can be used for image classification, but it there is not enough discussion comparing it to recent ideas that go beyond ViT (e.g Swin-Transformer and ConvNextT models). It could be interesting to examine these models with your notation and perspective as well, since these achieve similar (or even better) efficiency and accuracy for the classification task.  

(*) More experiments comparing the proposed model as a backbone for visual tasks might be useful for better evaluation against other models, such as experiments for detection and segmentation. 


clarity,_quality,_novelty_and_reproducibility: The paper is clear. Main ideas seem to be novel. Most parts seem to be reproducible. 
summary_of_the_review: Novel architecture based on interesting perspective on formulation of neural network layers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The purpose of the paper is to propose a more general architecture than transformers. The proposed architecture is adaptive according to the different instance axis, spatial dimension and channels. In order to limit the complexity of such an approach the paper adopts a decomposition of the different operations. The paper evaluates in image classification, segmentation and detection.
strength_and_weaknesses: Strengths:
- The subject treated in the paper is interesting as well as the different operations used in the decomposition to have dunamic layer on the different axis.
- The paper is quite well written and easy to follow.

Weakness:

- Hierarchical aspect: The decomposition of the dynamic block is interesting. However the hierarchical aspect of the architecture adds complexity and noise to the analysis. As we can see in Appendix Table 2 in order to design ChopForme we have to define the different pooling stage, the width and the depth of each. This complexify the architecture in comparison to vanilla transformers. It would be better to do an ablation of the architecture without the hierarchical aspect.

- Tasks: 
The paper proposes an architecture presented as general but there are only tasks of computer vision. It would be interesting to complete it with NLP tasks to see if the architecture is competitive with transformers.

- Missing SOTA architecures: The architectures reported for comparison in the paper are not SOTA. In Table 1 appendix and Table 2 main paper. The following architecture should be added: EfficientNet-v2 [1], CoatNet [2], LeViT[3], MaxViT [4] The results of DeiT can be updated with those presented in the DeiT III paper [5].

[1] Tan et al., EfficientNetV2: Smaller Models and Faster Training
[2] Dai et al., CoAtNet: Marrying Convolution and Attention for All Data Sizes
[3] Graham et al., LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
[4] Tu et al., MaxViT: Multi-Axis Vision Transformer
[5] Touvron et al., DeiT III: Revenge of the ViT

- Missing metrics: Several metrics are missing in the different tables to evaluate the tradeoffs of the proposed architecture as well as the ease of use. In addition to parameters and FLOPs, peak memory and speed should be reported.

- Only small architectures:  The paper considered only relatively small architectures. Ablations are done with the tiny version and the largest model reported is smaller than ViT-B. General architectures like transformers generally scale very well so it would be interesting to add results with larger architectures to see if the method scales. The comparison Table 2 in the main paper are only with small architectures.

- Overfitting evaluation: For the evaluation on ImageNet there is no measure of overfitting. But there is no separate validation set for ImageNet. It is necessary to add evaluation on ImageNet v2 in order to measure the level of overfitting of the proposed method. Especially since the more general an architecture is, the higher the risk of overfitting.

- Segmentation & Detection: There is no large comparison in segmentation and detection of the proposed method with other approaches like Swin or Deit III. It is important to make this kind of comparison in order to determine if the architecture is competitive on different tasks
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. However, it is not clear that this new kind of architecture will be competitive with other general architectures such as transformers which are very efficient on different computer vision and NLP tasks and scale extremely well.The results seem reproducible.
summary_of_the_review: The idea of the paper is interesting but it lacks a lot of analysis to show that this approach is competitive with well established general architectures like transformers. Indeed, there is no experiment in the paper that shows the advantage of the proposed architecture over the transformers on different vision tasks like segmentation, detection and image classification. There is no experience that shows that the proposed architecture generalizes to other domains like NLP tasks and that the models scale well.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors propose a new neural network architecture based on a tensor-rank decomposition (CP decomposition) which can generalize convolution and self-attention layers. In particular, the authors test the efficacy of their method on puzzle reconstruction with MNIST, detection and segmentation on COCO, and image classification on ImageNet. The proposed layer achieves competitive and at times favorable performance given a computational budget across an array of these vision tasks.

The reviewers commented positively on the novel manner for unifying convolutional and Transformer layers, the application of the CP decomposition, and competitive numbers on ImageNet image classification. The reviewers also expressed concerns about (1) the overall novelty of the resulting architecture, (2)  a lack of experiments on non-computer vision tasks, (3) relationship with previous works on unifying CNN’s and Transformers, and (4) lack of strong improvement over previous methods. During subsequent discussion, the primary issues about (3) and (4) were not resolved and no reviewer raised their score to champion acceptance of the paper.

Although I sympathize with the desire to only evaluate on vision tasks to make the problem more tractable for a conference-length submission, I am concerned that (3) and (4) specifically. To address (4) the authors do need to provide additional clarification in the manuscript about the relationship with prior work attempting to generalize across layers and architectures. With regard to (3), one item of discussion focused on appropriate comparisons and which works were contemporaneous with the present work. Given a submission deadline of May 28, 2022, several of the papers highlighted – e.g. CoatNet (NeurIPS 2021), EfficientNetv2 (ICML 2021); MaxViT (ECCV 2022) – were all accepted prior to this cutoff deadline and should be considered valid prior work when arguing whether this model achieves SOTA performance. Given that SOTA performance is one of the central contributions in the Introduction, there should at the minimum be additional clarity from the authors about what aspects are being considered SOTA with respect to what baselines. Because of all of these unresolved issues, we can not accept this paper to this conference. I would instead encourage the authors to revise their manuscript and their claims accordingly and resubmit to a future venue.

justification_for_why_not_higher_score: Weak baselines. Minimal empirical gains.
justification_for_why_not_lower_score: n/a
Decision: Reject