Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a model for selecting test-time data augmentation to boost a classification model. The proposed method is based on RNN to gradually output multiple augmentations, selected from a predefined augmentation set. The idea is based on loss prediction for finding suitable augmentations. In each step of RNN prediction, the model outputs a predicted loss value and then select the best augmentation. To show the effectiveness of the proposed method, this paper conducts experiments on CIFAR-100 and ImageNet datasets, and compare this method with many other test-time augmentation strategies.
strength_and_weaknesses: Strength:
+ This paper presents a method, which is able to produce multiple data augmentations with single forward pass.

Weakness:
- The novelty of this work is not surprisingly strong. The proposed model is different to previous architecture, however, applying RNN to produce multiple output is a common method.

- From the experimental results, it is hard to observe clearly better results than other TTA methods, for example Horizontal-Flip. 

- How do you train the loss predictor. This is unclear how to train a loss predictor on specific datasets.

- How to select the predefined candidate transformations. For individual instances, different transformations are required. How do you prepare the transformation set for the best performance for various datasets. 

- Need citation after the first sentence in section 3.

clarity,_quality,_novelty_and_reproducibility: The overall quality of this paper is not strong enough. There are some issues unclear to me. I don't know how to train the loss predictor and cannot reproduce the results after reading this paper.
summary_of_the_review: This paper does not have a clear merit to be accepted by ICLR. They present an architecture for test-time augmentation, however, no clear benefit from the proposed method can be achieved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for learning data augmentations for improving test performance in the presence of distribution shift. The method uses an RNN based augmentation module rather than a feedforward module that may be run multiple times. Experiments show that this method seems promising on standard corrupted image benchmarks.
strength_and_weaknesses: Strengths
---

- Experiment seem to show that the method holds promise for achieving favorable performance and efficiency.
- The method is relatively simple and easy to understand.

Weaknesses
---

- The paper is difficult to follow in numerous places (see below).
- It is unclear whether the proposed idea is significant enough to warrant publication at ICLR.
clarity,_quality,_novelty_and_reproducibility: Quality
---

I expected to see a comparison to cyclic TTA in the experiments, but as far as I can tell, that is missing. The experiments do seem to show some improvements compared to other baselines, but the improvements are generally small and it is unclear how competitive these baselines truly are.

Clarity
---

I do not understand why the authors refer to their method with the term "cascade" when simply referring to it as RNN-based would be much simpler and clearer. Eq (2) seems to only make sense if y is a real value, which it is not in classification. I was not able to follow Section 3.3 detailing the training strategy, and given that there is still room in the main paper, I would suggest focusing on providing more detail in that section to help clarify these important points. Generally, the vocabulary used throughout ("ascendancy", "rough", etc.) unnecessarily complicates the exposition and I would recommend being more consistent and rigorous with the writing.

Originality
---

It seems like the main novelty of the proposed method is in using an RNN to output sequential augmentations. This does not seem significantly novel by itself. As noted, cyclic TTA also outputs sequential augmentations but is more computationally expensive due to working in image space. This seems to lead to a natural comparison: cyclic TTA, but with the same backbone that embeds the image into the lower dimensional feature space. In general, comparisons such as these (and standard cyclic TTA) would go a long way in strengthening the empirical aspects of the paper, which are needed due to the lack of a significantly novel technical contribution.
summary_of_the_review: In summary, there are several important improvements that can be made as laid out above. In the paper's current state, I am recommending reject.

Edit after author response
---

Thank you for the response, and apologies for the late reply. My recommendation remains unchanged. It seems that the other reviewers and I agree that this paper will benefit significantly from another full reviewing round before publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a test time augmentation using a cascade loss prediction method which only requires a single forward pass of the transformation predictor to select multiple transformations. In contrast to the repeated usage of one loss predictor in cyclic-TTA, the proposed method uses RNN to capture the semantic information in each iteration and predict transformations without intermediate images. Experiments are conducted on various architectures to explain the trade-off between computational cost and model performance.
strength_and_weaknesses: Strength
The idea of using RNN seems natural and useful..
The computational cost of the proposed method is showed impervious with the iteration number on Resnet-50.
The paper is well-organized and easy to follow 

Weakness
The experiments are not convincing enough. The paper does not compare with any SOTA methods listed in the related works.
As the results show in Table2, the proposed method has no obvious advantage over Single-TTA. 
It's necessary to compare the evaluation results and the calculation cost simultaneously to get a convincing conclusion, while both of them are insufficient in this paper.

clarity,_quality,_novelty_and_reproducibility: * Clarity:4
* Quality:3
* Novelty:3
* Reproducibility:3

summary_of_the_review: Overall, the main idea of this paper is interesting but the experiments are not convincing. Therefore, I suggest a rejection for this paper unless more experiments are complemented.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: To cope with the task of TTA, this paper proposes to replace the simple version or the cyclic version of the loss predictor with a cascaded version utilizing an RNN architecture. The authors evaluate their method on CIFAR-100 and ImageNet.
strength_and_weaknesses: Strength:
1. This paper is well-written and easy to follow
2. This paper is well-motivated and the motivation of this paper is reasonable.

Weaknesses:
1. My first concern w.r.t. this paper lies in its limited novelty. While I understand that compared with the previous method that uses either the simple version or the cyclic version of the loss predictor, this paper does demonstrate its advantages. However, it seems to me that this paper technically just applies RNN which is not a new technique already. Thus, while it is reasonable to choose to use RNN here, from my perspective, this does not bring enough novelty as no specific designs have been made for the task of TTA.
2. I am also a bit confused about why the authors suddenly jumped to the usage of ResNet-50. Specifically, intuitively, I think that the task of the loss predictor is not a very hard task. Thus, the usage of a lightweight backbone EfficientNet-B0 is adequate and reasonable from my perspective. Thus, while the authors do demonstrate that their method can be much more efficient than cyclic predictor with ResNet-50 as the backbone, I am a bit confused why ResNet-50 should be used as the backbone in the first place. Note that the improvement of efficiency is quite small on EfficientNet-B0, and it seems from the experimental section, the usage of ResNet-50 does not lead to a significant performance enhancement compared to EfficientNet-B0.
3. At last, I suggest the authors make their evaluation metric more clearly in the experiment section. Specifically, the name of the metric and whether a small or a large number leads to better performance can be included in the table. The current title of the table with only "Clean" and "Corrupt" there brings me a slight reading difficulty.
clarity,_quality,_novelty_and_reproducibility: I believe that the presentation of this paper is clear, and enough details have been provided to reproduce this method. However, I have concerns about its novelty.
summary_of_the_review: Overall, I have concerns about the novelty of this paper. Besides, while I agree that this paper is more efficient than the cyclic version, it seems that this efficiency is only significant when a large model is used as the backbone. However, it seems that a small model as the backbone is already enough in this task.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: We appreciate the authors for addressing our comments and concerns and revising certain parts of the paper to address some of them. While the simple method proposed is more efficient than related TTA methods proposed by others before, our general concern is that its novelty is not significant enough for publication in ICLR. Moreover, from the experiment results, its improvement over other TTA methods is also not very significant. We hope the authors find our comments useful to improve their work for future submission.

justification_for_why_not_higher_score: The paper is far below the acceptance threshold.

justification_for_why_not_lower_score: N/A
Decision: Reject