Learning Test Time Augmentation with Cascade Loss Prediction
Authors: Siyang Pan
Jiaqian Yu
Dongwook Lee
Qiang Wang
ChangBeom Park
ByungIn Yoo
Venue: Submitted to ICLR 2023 (2023)

Abstract: Data augmentation has been a successful common practice for improving the performance of deep neural network during training stage. In recent years, studies on test time augmentation (TTA) have also been promising due to its effectiveness on improving the robustness against out-of-distribution data at inference. Instead of simply adopting pre-defined handcrafted geometric operations such as croping and flipping, recent TTA methods learn predictive transformations which are supposed to provide the best performance gain on each test sample. However, the desired iteration number of transformation is proportional to the inference time of the predictor, and the gain by ensembling multiple augmented inputs still requires additional forward pass of the target model. In this paper, we propose a cascade method for test time augmentation prediction. It only requires a single forward pass of the transformation predictor, while can output multiple desirable transformations iteratively. These transformations will then be adopted sequentially on the test sample at once before the target model inference. The experimental results show that our method provides a better trade-off between computational cost and overall performance at test time, and shows significant improvement compared to existing methods.

Under review as a conference paper at ICLR 2023
LEARNING TEST TIME AUGMENTATION WITH CAS- CADE Loss PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation has been successful common practice for improving the per- formance of deep neural network during training stage. In recent years, studies on test time augmentation (TTA) have also been promising due its effectiveness on improving the robustness against out out-of-distribution data at inference. Instead of simply adopting pre-defined handcrafted geometric operations such as cropping and flipping. recent TTA methods learn predictive transformations which are sup- posed to provide the best performance gain on each test sample. However, the desired iteration number of transformation is proportional to the inference time of the predictor, and the gain by ensembling multiple augmented inputs still requires additional forward pass of the target model. In this paper, propose cascade method for test time augmentation prediction. It only requires a single forward pass of the transformation predictor, while can output multiple desirable trans- formations iteratively. These transformations will then be adopted sequentially on the test sample once before the target model inference. The experimental results show that our method provides a better trade-off between computational cost and overall performance at test time, and shows significant improvement compared existing methods.
INTRODUCTION
Robustness in artificial intelligence system has been recognized as an important topic in recent years, especially for the application scenario closely related to human life or health, such biometrics, autonomous driving, medical diagnosis, virtual and augmented reality and so on. Though heavily rely on training data, AI models in real-world will inevitably encounter unforeseen cireumstances, which requires not only high performance from the aspect of accuracy, but also high robustness from the aspect of generalization.
Data augmentation has been a successful strategy for improving the robustness in many deep learn- ing model training applications. During training stage, various transformations adopted on the input samples thus to expend the diversity of the training space without truly collecting novel data. In the community of computer vision, some basic augmentation operations commonly used such as rotation, zoom-in and -out, cropping, flipping, translation, blur, contrast, etc. Some ad vancee techniques also explore sub-instance level operations such as mixing samples together (Zhang al.,
2018 Veiess & Taylor 2017 Hendrycks et al.] 2019), or learnable augmentation search strate- gies (Cubuk et al.] [019 Lim et al., 2019 Hataya al., 2020) Zheng et al., (021). While data augmentation during training time brings much benefit, the challenge would lie in the training cost
and the difficulty, given the continually increasing size of the training dataset.
On the other hand, training time augmentation cannot solve the issue once for all. In general, we consider that the performance on in-distribution data as the standard accuracy, the performance on the out-of-distribution data the robustness the generalization. Specifically, we consider some corruption will occur at test time that is unknown a priori. Consequently, such kind of corruption cannot be explicitly learnt during training stage e.g. by adopting certain data augmentation that attempts to explore the unknown data distribution.
Test time augmentation (TTA) is defined transforming samples before inference test time. Con- ventional TTA always requires averaging multiple predictions over different augmented test samples to obtain final prediction. The major performance gain of conventional TTA methods heavily lies

---

Under review as a conference paper at ICLR 2023
in the ensembling mechanism (Lakshminarayanan et al. 2017), which inevitably requires multiple forward passes of the inference model. Recent studies on learnable TTA methods put more focus on how to select the best transformation policies at each inference, i.c. the supposed provide the largest performance gain compared to no transformation (Kim et al. 2020; Chun et al. 2022). By adopting instance-level transformation policies, these methods show significant improvement for corrupted (out-of-distribute)
data without harm on clean (in-distribution) data. However, there exist still several limitations: (i) most methods still require model ensembling on different predic- tions to achieve the best performance; (ii) the desired iteration number of transformation before the inference proportional the cost of the transformation predictor, which limits the variety of the transformation; (iii) the transformation policy search is still under-explored thus leads to sub-optimal performance.
In this paper, we propose a cascade loss prediction method that, for the first time, only requires a sin- gle forward pass of the transformation predictor, while can output multiple desirable transformations iteratively. Our contribution can be summarized as follows:
a novel cascade test time augmentation with sequential predictions by a single forward pass.
better trade-off between target model performance and inference cost with the first com- patibility and analysis on various network architectures.
a better exploration on the test data space which leads to state-of-the-art performance against various corruption benchmark.
2 RELATED WORKS
General Data Augmentation Traditional data augmentation aims at enlarging training datasets to improve predictive performance. Recent works explore more diverse strategies of data augmentation such by mixing up the features and their corresponding labels (Zhang et al. 2018), by cutting out some random certain area of mixed samples Vries & Taylor| (017). or by cutting out then mixing up those samples with different strategies (Yun et al. [019:] Han et al. 2022). On the other hand,
there are some studies on trainable augmentation policy (Cubuk et al. 2019 Lim et al. 2019 Hataya et al. 2020; Zheng et al. 2021). They focus rather the exploration of larger data space and the automatic learning strategy for efficient training. These techniques are commonly used in many state-of ohette-arttt models for their benefit on both accuracy and calibrations, bringing performance
gain on standard benchmarks such as CIFAR (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).
Robustness Sufficient augmentation is also a successful practice to improve didtr--dditributionn robustness. [Hendrycks & Dietterich (2018) built the first benchmark for eval- uating model robustness given different image corruption at test time. [Hendrycks et al.] (2019) proposed a simple data processing method to improve robustness; it augments training samples by mixing weighted random transformation operations and learns distribution similarity between the original samples and the augmented samples. [Wen et al. (2020) argued that simple model ensem- bles top with such augmentation will degrade the performance, and then proposed improved variant that dismisses the ones with high uncertainty. Zhang et al. (2021) proposed to adapt the model parameters by minimizing the entropy of the model's average output distribution across the augmentations, at test time. Whereas the inference becomes expensive due to its augmentation and adaptation procedure, thus limits the usability for other models tasks.
Test Time Augmentation Given a trained model, conventional test time augmentation is often car- ried out together with model ensembling, that is at inference with different augmented test samples, such as the conventional transformations e.g. cropping or flipping- Lyzhov et al. (2020) demon- strated that test time augmentation policies can be learned and introduced a greedy method for learning policy of test time augmentation. (Shanmugam et al. (2021) analyzed when and why test time augmentation works and presented a learning based method for aggregating test time augmen- tations. Kim et al. (2020) selected suitable transformations for a test input based their proposed loss predictor; without high additional computational cost, it carried out instance-level transforma- tion at inference for the first time. However, the proposed method only explores one single trans-
2

---

Under review as a conference paper at ICLR 2023
formation for each sample, while requires model ensembles achieve the performance target model with multiple transformations. Recently, [Chun et al. (2022) proposed cyclic search for suitable transformations with the use of the entropy weight method, thus extend the instance-level augmentation to larger data space. Whereas, the proposed cyclic mechanism carried out on the entire loss predictor thus the model size is limited to be lightweight so to achieve reasonable applicability.
METHOD
Loss prediction find suitable transformations is an efficient search policy for test time augmen- tation (Kim et al. 2020) Chun et al. 2022). In this section, we introduce our method of a cascade loss prediction that outputs a succession multiple transformations describe the general test time augmentation and the loss prediction pipeline in Section 3.] Then, our Cascade-TTA method in detail explained in Section 3.22 The cascade style contributes to the flexibility as well as the advantage in terms of calculation cost. Particularly in Section introduce the training method for our cascade loss predictor.
stretch. To begin with, we
3.3 we
3.1 LOSS PREDICTION
Given a trained target model Otarget and an input image I, the predictive result is based on the output y of the target model:
y Izorge(
(1)
Let / T2, ..., T denote the candidate set of augmentations, test time augmentation is then conducted as:
Vtta
barget (T((x)),
(2)
where l: indicates the size for ensemble effect. Conventional TTA methods usually carry out straight- forward transformation for T. such as cropping and flipping (He et al. 2016 Krizhevsky et al. 2017). indiscriminatingly performed
every input image. Assume we have an N -siized pre-defined trans- formation operation set, T = {41, l2 ..., fv Especially, lia is included cate no transformation operation is done. In conventional TTA, only one transformation is adopted each time on z, thus the size of the candidate set is | = N.
as ident ity to indi-
In this paper, define that each T, is sequence of independent augmentations as
T TTE, Tias .. Ti..: Ti,
where is the iteration number of transformations, L 2 1; T is one single transformation from the pre-defined space. Thus conventional TTA with L = 1 is a special case by this definition. In general, the transformation space is | N 1 1.
Following Kim et al. (2020), we propose to find the optimal Ti in an instance- manner:
T Fr. î2. .... tz] (z), ig e T,,
where feaa stands for the learned search criteria.
Given a trained Otarget, when L = 1, the loss value on augmented samples L, (O (f))))((()))))))))))))(()) can signify the quality of the transformations (Kim al.] 2020). where ý indicates the ground- truth of for the target model. Thus, the selection on r is straightforward with the exact loss values. The loss predictor, denoted by Opp can be trained independently order to estimate the loss values corresponding to each pre-defined transformation (Kim et al. 2020):
feaa
Otp -==1-
(S)
The loss predictor takes charge telling by which transformations the target model achieves best performance. Since the output of loss predictor represents the quality ranking of the transformations, the benefit from ensemble effect is also possible. A cyclic version of the loss predictor has been proposed deal with severely computted test samples (Chun et al. 2022). namely in the case of
3

---

Under review as a conference paper at ICLR 2023
Praditedd Loss
Pood-died
0.5;; 02
0.35 4.5
Laaes Preductro
Thrret Modi
Farget Mode
0.2
1/+
1/r
OG
Badcooe
Stekkd RNNNoil
Stacaed RVNN
Thrget Mixil
layer
layer
Preckcteed
Loss
12y
Figure 1: Illustration of different loss predictors at inference stage. The selection from pre-defined transformations is based on the assumption that lower predicted loss value corresponds to better transformation. (a) The single loss predictor for the best transformation when
1, selecting from T {4, ..., }. }- (b) The cyclic version of the loss predictor. L steps of the single loss predictor form cycle and produce L-sequenced transformations. (c) The cascade loss predictor, requiring only a single of forward pass prediction with L transformations outputted. The backbone is used once while the stacked RNN-cell and the FC layer produce the outputs itcratively.
>> 1. Multiple repeated usage of the loss predictor forms cycle, deeming the transformed image again as an input:
times
Sena Opp( .). O
(6)
In the cyclic process, the loss predictor takes the intermediate transformed images in each iteration.
3.2 CASCADE LOSS PREDICTION
Multiple iterative transformations on single test sample improves the potential TTA. With L > 1, more operations are adequately performed to be better appropriate for the target model. Different from simple and plain repetition on the loss predictor, our method focusses on how to produce succession of transformations once with single network:
fut Ocip:
where Oelp is our novel cascade loss predictor performing merely once with no limitation on L.
As the ensemble is simple to implement in practical, we will take k = 1 in the sequel for simplicity. Figure I shows the overview of the single loss predictor (Kim al., 2020). the cyclic version (Chun et al. 2022) and our cascade loss predictor respectively. As noted earlier, the single implementation only caters for T, and the cyclic version just calls the loss predictor multiple times block-wise.
In this paper, we propose cascade architecture as shown in Figure [Tc). It uses recurrent neu- ral network (RNN) to capture the semantic information the transformed image in each iteration, and realizes predicting iterative transformations with no need to take advantage of the intermediate transformed images. Significantly, only single forward pass of the cascade loss predictor is re- quired to obtain L desired transformations iteratively. Without the tedious process of re-inputting the transformed image into the loss predictor, the proposed cascade network just accepts once the original input but provides succession of appropriate transformations. On this occasion,

---

Under review as a conference paper at ICLR 2023
Algorithm Inference of our cascade predictor
Inputs: An input test image ø1
Output: A succession of transformations T
1: hi Backbone(z1)
2: Ti < arg min
3: T [T]]
4: for each i E [2, L) do
5: if T;-1 is lid then
6: break end if
7:
8:
l 4 IN
9:
T T (FC(
10:
T [Td]
end for
able to directly perform the obtained L-sequenced transformation = [TL, T2, TL] at a stretch, and straightforward get the final augmented sample (zx) that should be fed into the target model. None the intermediate transformed images are substantially utilized for the cascade network.
The proposed cascade loss predictor consists of three parts including the backbone, the stacked RNN-cell and the FC layer. The direct and concise prediction on the successive transformations comes from the RNN architecture of the cascade network. In other words, the stacked RNN-cells process the dependencies through the cascade loss predictions. Inspired by the tremendous success of RNN models in sequence processing (Bahdanau al. 2014 Yang et al. 2018)), we put forward reasonable RNN-based loss predictor for sequential transformations generation.
We present the inference procedure of the cascade predictor shown in Algorithm I h, denotes the feature of state at each iteration. In the first iteration, we take advantage of the backbone feature / for transformation prediction. From the second iteration, instead of using the explicitly augmented sample i.e. = TT-1(T-2( 71 (z1)...) )), we apply the iterative hidden state as following:
h;
where is the embedding network to embed the transformations to feature space. Then the optimal T, at this iteration selected by the minimum predicted loss value from a linear regressor. We propose to use two stopping criteria: (i) tia is achieved; (ii) maximum iteration L. See Appendix A for more details on the procedure the cascade prediction.
For efficiency, the EfficientNet-BOO (Tan & Le 2019) with modification Yoo & Kweon 2019) is
often used as the backbone the loss predictor (Kim et al.) 2020 Chun et al. 2022). whose cost is relatively negligible to the target model. The downsizing operation into by 64 pixels for ImageNet dataset (Deg et al. 2009) further reduces the computation. However, light backbones intuitively lack of representative ability, especially for the multiple iterative predictions. As the complexity of backbone increases, it is also important to consider the trade-off between the loss prediction cost and the improved performance. We will show in Section 4.3 the analysis and experimental results on different complexity of backbones for the cascade loss predictor.
3.3 PRACTICAL TRAINING STRATEGY
We illustrate the training method of Cascade-TTA in Figure 2 Given training image I1, the ground-truth is defined as L vectors corresponding to L training iterations. Specifically, each of the vectors consists of N loss values, which calculated from the target model regarding augmented images as inputs by the pre-defined transformations. For iteration i, we note the ground-truth loss value vector as:
1, (Otargee (mx)), ú)) .j E [1,
where z; is the intermediate image output from the last iteration and ty (2:) indicates the transformed one. Note that L, indicates the loss function for the target model which has been all fixed. Different from the inference stage, I, needs to be explicitly produced for the ground-trut -tuutththhhhhh generation.
In practice, the intermediate image I; for iteration is augmented by the pre-defined transformations T = (11, (2, l2,,,, -... ), and each of the transformed image 15(x;) is in parallel fed into the target model
5

---

Under review as a conference paper at ICLR 2023
Stacke
Slackd RNN-oell
Backbone
RNN-oal
FC layer
FC layer
PC Layer
Modil
Tasget
dess:
mpn Corndalisa
Traryt Model
losdy
Corniation
Speeraa
Rankingg loss
Ratikap loss
Modld
Target
Joss
Terget Model
loong
Softnax
Saftraat
Ex
Theret Made
Midkl
Kandon
Bandom selection
stlaction
Figure 2: Illustration of the training procedure of the cascade loss predictor. The shaded area is the single ground-truth generator, which iterated several times. Note that even when training the predictor, the target model still remains fixed.
to obtain the loss value (O (1, (x;() 9). Before training the loss predictor, loss values are generated from the trained target model by using Li the loss function of the target model at its training stage. The generated loss values are then gathered to produce 1, and processed by softmax function. Concretely, in the first iteration, the training sample xn is transformed to [ty (x1)), which are fed into the target model for loss value calculation and eventually contribute to the first ground- truth vector 1, For the diversity and balance of training data, we randomly assign a transformation To (except tia) for the current image Z;. For instance, a random transformation Ti E 7 is assigned for 0ı at the end of the first iteration, so the intermediate image for the second iteration is defined as T2 = Ti (z1). During the second iteration, similarly the target model takes the transformed {t, (I2)) and produces the second ground truth vector 12. And so on, for each of the iterations, sequentially we can get all of the ground-truth vectors (1,) On the other hand, vector of predicted loss values produced by the FC layer in each iteration, so our cascade loss predictor optimizes the correlation between the predicted and ground-truth vectors in all training iterations.
The training iteration number can be determined independent the test time iteration number. Spearman correlation ranking loss (Engilberge al. (2019) is used for optimization in all iterations together, which is a better description the transformation quality than the exact ones. We regard the iteration number part of batch size during training the cascade loss predictor.
EXPERIMENTAL RESULTS
4.1 CIFAR 100
CIFAR-100 benchmark KKrizhessky al. 2009) is a widely-used classification dataset. A total of 60000 images with 32 by 32 pixels belong to 100 classes. The corruption version of CIFAR- 100- (Hendrycks & Dietterich 2018) introduced for evaluation, and the corrupted variant consists of a total of 19 kinds with severities. The error rate for clean data and the average corruption error, CE.- D define the evaluation criteria, where E denotes the Top-1 error rate on corruption e with severity s. The pre-defined transformations used in all of our experiments consist of 12 kinds of operations. See Appendix B for details.
As shown in Table T experiments are conducted on the comparison between Cascade-TTi TTA and ex- isting TTA methods. Results for both clean and corrupted data are presented. The two target models that we use are both augmented with AugMix (Hendrycks et al. 2019) with different architecture of Rest--42N (Zagoruyko & Komodakis 2016) and ResNeXt-29 (Xie al. 2017). Due

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation result CIFAR-100(CC) dataset. Metric for corrupted set average corruption
enror. The Cascade-TTA results are shown with L = 3.
Target Model
TTA method
Target Model Cost
Clean Corrupt
Center-Crop
23.00
35.34
Horizontal-Flip
22.36
34.38
5-Crops
5
22.97
35.16
Wide-ResNet
Random TTA
27.86
40,89
Cascade-TTA
23.08
34.12
Center-Crop
20.41
33.51
Horizontal-Fllip
2
19.82
32.90
5-Crops
5
20.11
33,26
ResNext
Random-TTh
25.51
38.94
Cascade-TIX
20.44
31.99
to the low resolution of CIFAR-100 images, simply use modified EffcientNet-BOO as the back- bone of the cascade loss predictor. The conventional augmentation strategies such as Center-Crop, Horizontal-Fli and 5-Crops have always been used in previous applications. In spite of the mul- tiple target model cost for ensemble in Horizonta-Fliip and 5-Crops, the improved performance is still limited. Random-TTA means choose random transformation mentations. The degraded performance indicates the sufficiency of the transformation diversity. Our proposed Cascade-Ti -TAA outperforms not only the methods with same computation cost, but also outperforms the methods with larger cost of target model.
of the pre-defined aug-
4.2 IMAGENET
ILSVRC 2012 classification benchmark (ImageNet) (Deng et al. 2009) consists 1.2 million im- ages of 1000 classes. The corrupted variant of ImageNet-C (Hendrycks & Dietterich] 2018) is also used for out-of-distributiinn evaluation. The error rate for clean data and the mean corruption error (mCE) metric for corrupted data is calculated for evaluation criteria. Table shows the perfor- mance of different TTA methods on ResNet-50, but with different train-time augmentation. Here also implement and ResNet-50. For target models trained with Standard, our performance by a clear margin exceeds the single version of the loss predictor, no matter which the backbone is used. Additionally, ensem- ble of two transformed images by Cascade- TTA with multiple cost of target model, produces lower emror rate up to expectations. For target models trained with AugMix, Cascade-TTh also achieves best performance on both of the backbones.
backbones of the loss predictor with different complexity, EfficientNet-BO
4.3 BACKBONE VS PERFORMANCE
We also explore on various network architectures to dig into the trade-off between backbone com- plexity and performance. Especially, show in Figure the calculation cost different iteration when using an EfficientNet-BO0 or a ResNet-50. As we can see, there exists essential difference on the calculation cost between Cyclic TTA (Chun al.) 2022) and Cascade-TTA, With the peated usage of the loss predictor for Cyclic TTA, the cost explicitly multiplies, greatly requiring L times of the backbone cost. For our proposed Cascade-TTA the cost contrary mainly depends on the stacked RNN-cell, which concretely contains one time of the backbone and L 1 times of the stacked RNN-cell. It is important note that the stacked RNN-cell is generally light enough to rival EfficientNet-OO.
We carry out experiments with diverse backbones to explore the relation between the complexity of backbone and the improved performance. Table shows the performance of Cascade- TTA with kinds of backbones for cascade loss predictor, ranging from EfficientNet-BO to ShuffleNetv2 (Ma) al. 2018) and ResNet-50. The input resolution is adaptively adjusted as 64 by 64 pixels on each EfficientNet family backbone. We choose to use the trained target model with ResNet-50 as backbone and Standard training data augmentation. Mcanwhile, results of different maximum iteration numbers are presented in proper order for compare. Experiments show that with light backbones, short length of iteration efficiently improve the performance, but the increasing

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluation result ImageNet(-C) dataset with target model ResNet-50. The Cascade- TTA results are shown with L = 2. Train-Aug
TTA method
Target Model Cost
Clean
Corrupt
Center-Crop
24.14
77.54
Horizontal-Flip
23.76
76.50
5-Crops
23.57
76.08
Random-TTA
26.82
81.55
EfficientNet-BO Single-TTA**
24.19
75.09
Standard
EfiicientNet-00 Cascade-
24.17
74.20
EfficientNet-BO Cascade -TA
2
24.16
74.05
ResNet-50 Single-TTA*
24.16
74.46
ResNet-50 Cascade-TTA
24.20
74.33
ResNet-50 Cascade-TTA
24.14
74.02
Center-Crop
22.39
66.57
Horizontal-Flip
2
22.14
65.84
5-Crops
21.70
65.02
Candom-TTA
24.15
70.58
AugMix
EfiicientNet-B BO Pascade-TTA
22.37
64.87
ResNet-50 Cascade- TTA
22.38
64.49
Single-TTA* indicates the work of Kim et al. 2020).
We re-implemented Cyclic TTA but did receive results Chun
2022 presents.
EfficientNet-BB
ResNet-50
1,000
6
FLOPs(M)
FLOPs(M)
800
4
600
400
2
2
3
2
3
iteration
iteration
Figure 3: Iteration Number vs. Calculation Cost. The blue line stands for the usage of Cyclic TTA while the red line represents our method. Left: EfficientNet-BO as backbone, the cost of Cyelic TTA is marginally more costy, Right: ResNet-50 as backbone, the cost increases sharply along with the iteration number Cyclic TTA while our Cascade- MAscade-TTA is almost impervious.
iterations soon start to depress the improvement even the cost of predictor expands. Therefore, the light backbones for the loss predictor limit the improvement from the iteration number. We assume that it is due to the insufficient representation ability for the growing itcrations of prediction. However, there no such issue when using large backbones. The increase on iteration stability provides greater benefits, and the extra cost from RNN-cell is still light. Eventually the improvement exceeds light backbones with sufficient iteration. In addition, experimental results show that the largest backbone as ResNet-50 provides the best performance with long enough iterations. As consequence, when desiring optimal performance gain out of the loss predictor, long iteration and large backbone offer the best alternative. In this case, Cascade- -TA compared with Cyclic TTA, requires merely the extra cost of the light RNN-cells instead of the multiple large backbones, so Cascade-TTA provides the best tr with significant improvement.
8

---

Under review as a conference paper at ICLR 2023
Table 3: The ImageNet-C results of Cascade-TTA backbones with different complexity. The first, second, and third iteration of results are shown for the trend.
FLOPs(M)
Clean
Corrupt
TTA backbone
L
L
=
L
ElficientNet-Bi
2.265
4.498
6.317
24.17
24.17
24.18
74.70
74.20
74.73
ettcc
3.273
5.590
7.452
24.19
24.20
24.20
74.48
74.32
75.66
EfficientNet-B B4
5.674
8.247
10.240
24.16
24.18
24.18
74.66
74.49
75.07
EfficientNet-BB B8
16.330
19.581
21.922
24.21
24.21
24.20
74.61
74.66
76.13
ShuffleNetv2
48.692
51.152
53.087
24.19
24.18
24.18
74.80
74.46
74.19
ResNet-50
337.308
339.767
341.702
24.20
24.20
24.20
74.64
74.33
74.05
5 DISCUSSION AND CONCLUSION
The visualization results are shown in Figure 4 The images transformed iteration by iteration and eventually are correctly classified by the target model.
Botar
Supportt
Xindy
Spockis Nove
SS
12 Slappmo 822
Poud
Cdla
Muatdy
Figure 4: Visualization of selected samples on ImageNet when L 3. First column: corrupted images from ImageNet-C: Second to fourth column: transformed images performed by our method iteratively. The corruption way the selected transformation is annotated below each image.
To conclude, in this paper, propose novel test time augmentation using cascade loss predic- tion. For the first time, multiple transformations can be predicted iteratively with one single forward pass of the predictor. The cascade predictor is computational efficient and compatible to various network architectures with limited additional cost, thus holds a promising applicability. Due to the fact that the training space is exponential to the pre-defined type of transformations, we propose practical training strategy to train the proposed cascade predictors. Experimental results validate the effectiveness of the proposed method. We suppose that enlarging the pre-defined transformation space could further upgrade the performance, while an efficient training strategy is essential, which could be expected future work.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:/409.0477 2014.
Sewhan Chun, Jae Young Lee, and Junmo Kim. Cyclic test time augmentation with entropy weight method. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.

---

Under review as a conference paper at ICLR 2023
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 123, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24822 255. leee, 2009.
Terrance Deieies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
04552, 2017.
Martin Engilberge, Louis Chevallier, Patrick Pérez, and Matthieu Cord. Sodeep: a sorting deep net to learn ranking loss surrogates. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10792-118001, 2019.
Junlin Han, Pengfei Fang, Weihao Li, Jie Hong, Mohammad Ali Armin, Ian Reid, Lars Petersson, and Hongdong Li. You only cut once: Boosting data augmentation with single cut. Proceed- ings of the 39th International Conference on Machine Learning, pp. 8196-8212, 2022.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation, In European Conference on Computer Vision, pp. 1-16. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi-- narayaman. Augmix: simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2019.
Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning loss for test-time augmentation. In Advances in Neural Information Processing Systems, volume 33, pp. 4163 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL 9ef2ed4b
neurips. wtw.eripps.cpppppr////0/.. pttpppppppp//////ille/// pdf
Sungbin Lim, I1doo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
Alexander Lyzhov, Yuliya Molchanova, Arsenii Ashukha, Dmitry Molchanov, and Dmitry Vetrov. Greedy policy search: A simple baseline for learnable test-time augmentation. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 1308-1317, 2020.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1214-1223, October 2021.
10

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, Pp. 6105 6114. PMLR, 2019.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W Dusenberry. Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. Sgm: sequence generation model for lulti-label classification. arXiv preprint 1rXiv:180.004880 2018.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp- 6023 6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. arXiv preprint arXiv:21 2X.v:2100009506, 2021.
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep autoaugment. In International Conference on Learning Representations, 2021.
A DETAILED ON THE CASCADE PREDICTION
Here we explain more details on how the cascade prediction works.
As shown in Algorithm] | Ict æ1 be a test sample. In the very first iteration, the cascade loss predictor takes 21 without transformations as input and produces h out of the backbone. Here we define the last layer of the backbone as a pooling layer, so hı is a vector representing the feature of x1. Then the FC layer takes h to perform the loss prediction, and selects the most appropriate transformation as TI. On the other hand, the stacked RNN-cell comes into play from the second iteration. Instead of fetching the transformed 222 = Ti (αi) substantially, the stacked RNN-cell takes /1 as the hidden state and the embedded т₁ input. Here we embed the transformation by learning vector to represent it. After the two representations are processed by the stacked RNN-cell, the feature of z2 can be still obtained as the output h2, without any processing by the backbone repeatedly. The FC layer is called again to perform the second time of loss prediction, taking h2 as input and selecting 72 as output. As we particularly employ the stacked RNN-cell instead of single cell, the vector fi is cut into several segments and respectively put into the single cells at different levels. The FC layer re-concatenates the several hidden states before the linear. In doing so, the deducible cascade fashion comes into existence. Aside from the old-fashioned loss prediction in the first iteration, for each iteration the hidden state h;-1 and the embedded selected transformation T--1 from the last iteration are inputted into the stacked RNN-cell, to provide the current hidden state h, And then the FC layer is responsible to predict the appropriate transformation т; from hG. Each state of h, simulates the feature of z; 77--177-2( (z1)...)) )). Eventually the cascade loss predictor is implemented continuously until two exit signals. One is the predicted best transformation as i ident i y and the other is the predetermined ceiling iteration number L. The former indicates the best condition of current image while the latter prevents endless prediction.
RNN kind of network with feedback loops, dynamically connects last output with current in- put, which greatly matches the searching process for iterative transformations. Given the ground- truth label of on with I. iterations, T [T) T2........ Tr.,, during ith iteration, the cascade loss

---

Under review as a conference paper at ICLR 2023
predictor aims at finding тІ. The determining factor of lies in the loss values of transformed I = (== while 7,-1 is corresponding to the prediction in the last iteration thus Ti-1 is supposed be inputted into the current iteration. In addition. RNN can memorize previous states of xi-1- Therefore, the utilization of RNN cascade loss prediction reasonably finds appropriate transformations iteratively.
B PRE-DEFINED TRANSFORMATIONS
We pre-defined transformations great extend following Kim et al. (2020); Chun al. (2022)
With bit difference on the magnitudes for better performance on iterative prediction, there are total of 12 kinds of pre-defined transformations in our experiments, including: identity, rotation, zoom, contrast, sharpness and saturation. Rotation refers to rotating the image 20 or -20 degrees the center point: Zoom refers to resizing the image to 0.8 1.2 times and then cropping for the original size; Contrast refers to adaptively streching the image to uniform contrast level; Sharpness refers to adjusting the sharpness of the image to 0.2, 0.5, 3.0 or 4.0; Saturation refers changing the color saturation of the image to 0.5 or 2.0.
12


Reviews:
[{"review_id": "xHxPoxFtlve", "reviewer": "Reviewer_yeBc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a test time augmentation using a cascade loss prediction method which only requires a single forward pass of the transformation predictor to select multiple transformations. In contrast to the repeated usage of one loss predictor in cyclic-TTA, the proposed method uses RNN to capture the semantic information in each iteration and predict transformations without intermediate images. Experiments are conducted on various architectures to explain the trade-off between computational cost and model performance.", "summary_of_the_review": "Overall, the main idea of this paper is interesting but the experiments are not convincing. Therefore, I suggest a rejection for this paper unless more experiments are complemented.\n\n", "strength_and_weaknesses": "Strength\nThe idea of using RNN seems natural and useful..\nThe computational cost of the proposed method is showed impervious with the iteration number on Resnet-50.\nThe paper is well-organized and easy to follow \n\nWeakness\nThe experiments are not convincing enough. The paper does not compare with any SOTA methods listed in the related works.\nAs the results show in Table2, the proposed method has no obvious advantage over Single-TTA. \nIt's necessary to compare the evaluation results and the calculation cost simultaneously to get a convincing conclusion, while both of them are insufficient in this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "u6dwiZ_wo-Y", "reviewer": "Reviewer_2Tn9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for learning data augmentations for improving test performance in the presence of distribution shift. The method uses an RNN based augmentation module rather than a feedforward module that may be run multiple times. Experiments show that this method seems promising on standard corrupted image benchmarks.", "summary_of_the_review": "In summary, there are several important improvements that can be made as laid out above. In the paper's current state, I am recommending reject.\n\nEdit after author response\n---\n\nThank you for the response, and apologies for the late reply. My recommendation remains unchanged. It seems that the other reviewers and I agree that this paper will benefit significantly from another full reviewing round before publication.", "strength_and_weaknesses": "Strengths\n---\n\n- Experiment seem to show that the method holds promise for achieving favorable performance and efficiency.\n- The method is relatively simple and easy to understand.\n\nWeaknesses\n---\n\n- The paper is difficult to follow in numerous places (see below).\n- It is unclear whether the proposed idea is significant enough to warrant publication at ICLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Nhk0lDVql9e", "reviewer": "Reviewer_N36J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To cope with the task of TTA, this paper proposes to replace the simple version or the cyclic version of the loss predictor with a cascaded version utilizing an RNN architecture. The authors evaluate their method on CIFAR-100 and ImageNet.", "summary_of_the_review": "Overall, I have concerns about the novelty of this paper. Besides, while I agree that this paper is more efficient than the cyclic version, it seems that this efficiency is only significant when a large model is used as the backbone. However, it seems that a small model as the backbone is already enough in this task.", "strength_and_weaknesses": "Strength:\n1. This paper is well-written and easy to follow\n2. This paper is well-motivated and the motivation of this paper is reasonable.\n\nWeaknesses:\n1. My first concern w.r.t. this paper lies in its limited novelty. While I understand that compared with the previous method that uses either the simple version or the cyclic version of the loss predictor, this paper does demonstrate its advantages. However, it seems to me that this paper technically just applies RNN which is not a new technique already. Thus, while it is reasonable to choose to use RNN here, from my perspective, this does not bring enough novelty as no specific designs have been made for the task of TTA.\n2. I am also a bit confused about why the authors suddenly jumped to the usage of ResNet-50. Specifically, intuitively, I think that the task of the loss predictor is not a very hard task. Thus, the usage of a lightweight backbone EfficientNet-B0 is adequate and reasonable from my perspective. Thus, while the authors do demonstrate that their method can be much more efficient than cyclic predictor with ResNet-50 as the backbone, I am a bit confused why ResNet-50 should be used as the backbone in the first place. Note that the improvement of efficiency is quite small on EfficientNet-B0, and it seems from the experimental section, the usage of ResNet-50 does not lead to a significant performance enhancement compared to EfficientNet-B0.\n3. At last, I suggest the authors make their evaluation metric more clearly in the experiment section. Specifically, the name of the metric and whether a small or a large number leads to better performance can be included in the table. The current title of the table with only \"Clean\" and \"Corrupt\" there brings me a slight reading difficulty.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Whb_ItUlFP", "reviewer": "Reviewer_6hqy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a model for selecting test-time data augmentation to boost a classification model. The proposed method is based on RNN to gradually output multiple augmentations, selected from a predefined augmentation set. The idea is based on loss prediction for finding suitable augmentations. In each step of RNN prediction, the model outputs a predicted loss value and then select the best augmentation. To show the effectiveness of the proposed method, this paper conducts experiments on CIFAR-100 and ImageNet datasets, and compare this method with many other test-time augmentation strategies.", "summary_of_the_review": "This paper does not have a clear merit to be accepted by ICLR. They present an architecture for test-time augmentation, however, no clear benefit from the proposed method can be achieved.", "strength_and_weaknesses": "Strength:\n+ This paper presents a method, which is able to produce multiple data augmentations with single forward pass.\n\nWeakness:\n- The novelty of this work is not surprisingly strong. The proposed model is different to previous architecture, however, applying RNN to produce multiple output is a common method.\n\n- From the experimental results, it is hard to observe clearly better results than other TTA methods, for example Horizontal-Flip. \n\n- How do you train the loss predictor. This is unclear how to train a loss predictor on specific datasets.\n\n- How to select the predefined candidate transformations. For individual instances, different transformations are required. How do you prepare the transformation set for the best performance for various datasets. \n\n- Need citation after the first sentence in section 3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject