Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Authors: Laura Yu Zheng
Wenjie Wei
Tony Wu
Jacob Clements
Shreelekha Revankar
Andre Harrison
Yu Shen
Ming Lin
Venue: ICML 2025 poster (2025)
Keywords: image segmentation
augmentation
robustness
heuristic
natural corruptions
adverse weather
Primary area: applications->computer_vision
TL;DR: We estimate sensitivity curves for different perturbation functions at intervals during train-time, and sample these curves for sensitivity-based augmentation.

Abstract: Achieving robustness in image segmentation models is challenging due to the fine-grained nature of pixel-level classification. These models, which are crucial for many real-time perception applications, particularly struggle when faced with natural corruptions in the wild for autonomous systems. While sensitivity analysis can help us understand how input variables influence model outputs, its application to natural and uncontrollable corruptions in training data is computationally expensive. In this work, we present an adaptive, sensitivity-guided augmentation method to enhance robustness against natural corruptions. Our sensitivity analysis on average runs 10 times faster and requires about 200 times less storage than previous sensitivity analysis, enabling practical, on-the-fly estimation during training for a model-free augmentation policy. With minimal fine-tuning, our sensitivity-guided augmentation method achieves improved robustness on both real-world and synthetic datasets compared to state-of-the-art data augmentation techniques in image segmentation.

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Laura Zheng'
Wenjie Wei Tony Wu Jacob Clements Yu Shen 14
Shreelekha Revankar 12 Andre Harrison 3
Ming C. Lin
Abstract
Training
Achieving robustness in image segmentation mod- els challenging due to the fine- geee-graned nature of pixel-level classification. These models, which are crucial for many real-time perception appli- cations, particularly struggle when faced with natural coruptiins in the wild for autonomous systems. While sensitivity analysis can help us understand how input variables influence model outputs, its application to natural and uncontrol- lable corruptions in training data is computation- ally expensive. this work, we present an adap- tive, sensitivity-guided augmentation method enhance robustness against natural corruptions. Our sensitivity analysis on average runs 10x faster and requires about 200x less storage than previous sensitivity analysis, enabling practical, on-the-fly estimation during training for model-free aug- mentation policy. With minimal fine-tuning, our sensitivity-guuie guidded augmentation method achieves improved robustness on both real-world and syn- thetic datasets compared to ifate-of-the-arr data augmentation techniques in image segmentation. Code implementation for this work can be found at:
Extimate 8, -
Model Accuracy (M4g - .MAg)
Segmentor
Ditaset Distance from Clean (KID -K1Dg)
InceptionV33
(a
Update until consergence
Figure I. Overview of online sensitivity analysis. We conduct adaptive sensitivity analysis using our fast analysis algorithm after warmup period clean data, then solve for L discrete perturba- tion intensitics, or levels, per perturbation type which the model is sensitive to. Finally, we augment training by sampling from the computed perturbation levels. Sampling weights are determined based off model performance on sensitive levels, where worse- performing levels are given higher probahility of heing sampled.
ronmental lighting, camera quality, and user handling. Simi- larly, autonomous vehicles and outdoor robots will need to operate under wide range of adverse weather conditions that are difficult simulate accurately. Even in medical imaging, where conditions are more controlled, factors such slight movements introduce blur, affecting segmen- tation results. While poor-quality examples can sometimes be discarded and re-captured, such solutions are costly or impractical, especially in large-scale, ubiquitous use cases, with limited resources, or during real-time inference (e.g.. failure in a navigating robot). Addressing these natural cor- ruptions is challenging because they are hard capture ahead of time in predictable and controllable way, sim- ulate parameterize, yet they significantly impact model performance.
1. Introduction
Segmentation models are crucial in many applications, but they often face unpredictable and uncontrollable natural variations that can degrade their performance. For instance, mobile applications using segmentation for image recon- struction may encounter diverse noises due varying envi-
One common approach to enhance robustness against such corruptions data augmentation, which artificially in- creases the diversity of training data by applying transfor- mations existing samples. While data augmentation is convenient and resource- -ffoureeefffiiient,, its effectiveness depends selecting the most beneficial angmentations. Ideally, we should determine which augmentation a given model is most sensitive to and focus on those to improve performance in other words, sensitivity analysis. However, traditional sensitivity analysis methods are computationally expensive
Department of Computer Science, University of Maryland, College Park, MD, USA Depaatment of Computer Science, Cor- nell University. Ithaca, New York, USA 'DEVCOM Army Re- search Laboratory, Adelphi, Maryland, USA 'Adobe Research San Jose, California, USA. Correspondence to: Laura Zheng <lyzheng@ udd.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
and resource-intensive (Shen et al., 2021), shown in Ta- ble 1, making them impractical for large-scale or real-time applications. Existing methods like AutoAugment (Cubuk et al., 2019) and DugmAntoAAgment attempt to optimize augmentation policies by training separate models, which adds significant overhead. Other state-of-the-art techniques rely on random augmentations (Cubuk et al., 2020; Muller & Hutter, 2021; Hendrycks et al.. 2020), which are scalable but may not target the most impactful transformations.
explored problem, especially in image classification. Cur- rently, image classification has a robust suite of bench- marks, including evaluation on both synthetic and natu- ral corruptions (Hendrycks et al., 2020; Yi et al., 2021; Dong et al., 2020). Many works study correlations be- tween image corruptions and various factors (Mintun et al., 2021; Hendrycks & Gimpel, 2017). Additionally, a popu- lar approach to increasing robustness in the general case is through targeted adversarial training (Xiaogang Xu & Jia, 2021; Shu et al., 2021). Several approaches target model architecture (Schneider et al., 2020; Saikia et al., 2021; My- ronenko & Hatamizadeh, 2020). Other approaches achieve robustmess to natural corruptions via the data pipeline. Data augmentations are a popular method for increasing out-of- distribution robustness and many have now become stan- dard practice (Geirhos et al., 2019; Rusak et al., 2020), Hendrycks et al. highlight that existing methods for gener- alization may not be consistently effective, emphasizing the need for robustness through addressing multiple distribution shifts (Hendrycks et al., 2021). In our work, we focus on studying and improving robustness in the context of seman- tic segmentation models due to natural corruptions using insights from previous work. Among findings from other works, we distinguish that our work focuses improving natural corruption robustness in semantic segmentation, a common computer vision task with unique properties.
In this paper, we propose scalable, fast sensitivity-guides augmentation approach for robustifying segmentation mod- els against natural corruptions, including those not explic- itly involved during training. Our approach performs a lightweight, online sensitivity analysis during training to identify the geometric and photometrie perturbations, shown to be effective as "basis perturbations" (Shen et al., 2021), to which the model is most sensitive. In contrast to (Shen et al., 2021), our sensitivity analysis is adaptive and signifi- cantly less resource intensive, allowing for practical imple- mentation without the need for offline models or extensive computation. Figure shows a high-level overview of our augmentation pipeline. Our method bridges the gap be- tween the efficiency of random augmentation techniques and the effectiveness of policy-based augmentations guided by sensitivity analysis. Despite our focus on segmentation, our approach is general and can be applied to other tasks, architectures, or domains without significant modifications.
Data Augmentation Techniques. Data augmentation meth- ods generate variants of the original training data to improve model generalization capabilities. These variants do not change the inherent semantic meaning of the image, and transformed images are typically still recognizable by hu- mans. Within data augmentation methods, CutMix and Aug- Mix are widely-used augmentation techniques that augment images by mixing variants of the same image (Hendrycks et al., 2020; Yun et al., 2019). Conversely, (Franchi et al., 2021) introduces segmentation specific augmentation ap- proaches which utilize superpixels, or clusters of similar pixels, to maintain semantic object information. Other data augmentation methods have utilized augmentation policies based on neural networks to select productive augmenta- tions (Olsson et al., 2021; Cubuk et al., 2019; Zheng et al., 2022), while other works have explored data augmentation for domain-specinn tasks (Zhao et al., 2019; Zhang et al., 2023). For example, (Zhao al., 2019) explores learned data augmentation for biomedical segmentation tasks via labeling of synthesized samples with a single brain atlas. (Zhang et al., 2023) explores data augmentation in specifi- cally brain segmentation via combining multiple brain scan samples, similarly to Augmix and Cutmix. However, this work is reliant on additional annotations to augment re- gions of interest. In our work, we present a generalizable augmentation technique and show that performance boosts generalize well out-of-the-bor -ho several domains.
In experiments, we achieve up to 6.20% relative mloU improvement in snowy weather and up to a 3.85% relative mloU improvement in rainy weather compared to the next- best method in zero-shot adverse weather evaluation on state-of-the-aar architectures. We also show improvements on synthetic benchmarks and increased data efficiency (up to 200x) compared to other augmentation methods as the size of the training set changes.
Our contributions are summarized as follows:
1. An efficient adaptive sensitivity analysis method for online model evaluation that iteratively approximates model sensitivity curves for significant speedup;
2. A comprchensive, censitivity-guided framework that systematically improves the robustness of learning- based segmentation models;
3. Evaluation and analysis of our method on unseen syn- thetically perturbed samples, naturally corrupted sam- ples, and ablated contributing factors to robustification.
2. Related Works
Robustification Against Natural Corruptions. The ef-
fect of natural corruptions on deep learning tasks is well-
2

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Method
SA Time
Data Gen Time
Storage
Algorithm Trrining with Sensitivity-Informed Augmenta-
tion.
AdvSteer
15.5= min
48 hours
2.4 TB
Data:
Training dataset X,, Validation dataset X., Valida- tion Rate u. SA Rate TsA
Ours
9.6.+0.2 min
12 GB
Result: Trained semantic segmentation model
Table 1. Runtime and Storage Comparison on Sensitivity Anal-
Ny + 0; // Number of validation rounds
ysis of AdvSteer (Shen et al., 2021) vs. Ours. Our approach
f() Identity());
// Aug transform
enables the practical use of sensitivity analysis in online training
3 Initialize network weights 0
as an augmentation policy. We compute each mean and standard
deviation value in "SA Time" across 4 runs. Each sensitivity anal- ysis iteration computes curves for 24 different augmentations at 5 levels each, for total of 120 evaluation passes. Advsteer
4 for i + 1...max iter;
1/ Training loop
5
do
6
Ess t DataLoader(()) if pf is initialized then // Sample aug PDF
requires an offline data generation stage for each dataset, whilst
7
s>ps:
ours is entirely online. Computed SA Time does not factor in data generation time. Ours runs about 9.3x faster and takes 2 less storage in isolation.
8
end
2 [I
if i % Tv 0 then
if i % TSA == 0; // Update SA Curve then
3. Methodology
12
In general, sensitivity analysis examines how small fluctu- 13 ations in the inputs affects the outputs of a system. our 14 augmentation approach, the key idea is that sensitivity anal- ysis can be used to sample augmentations uniformly with respect to "impact on model performance", opposed to sampling uniformly across the "parameterized augmenta- 17 tion space".
levels + 0: // Store all a values metrics + 0: // Store all metrics for each augmentation type f do a, 00,00 + SensitivityAnallsiitt 0);
// Appendix: Algorithm 2 levels.append(r,) metrics.append(ace)) end
levels = levels.oor(): // Sort based descending metrics
To quantify this for a given deep learning model, we need a metric for model performance and a metric for image degradation which is consistent across augmentation types.
20
Pj + BetaBinom(idx(/)) 0.75, 1.0) ;
// Categorical PDF by Acc
Choosing model performance metric is straightforward;
any bounded measure of accuracy (MA) where higher 21 end
values are better suffices. As for the image degradation 22 for Toi + DataLoader(X.): Validation
metric. we use Kernel Inception Distance (KID), introduced 23 do Compute clean validation metrics
by (Binkowski et al., 2018) to reduce bias towards sample size. At a high level, we use KID to measure the "distance"
end end
between an original dataset and its perturbed version. KID does so by passing both datasets through a generalized Inception model, and computing the square Maximum Mean Discrepancy (MMD) between their respective features. The reduced sample size bias of KID allows us to approximate the image degradation metric without iterating through the full validation set.
26
27 end
find set of increasing, nontrivial augmentation intensities < 02 <... az that maximize sensitivity. We define the local changes in accuracy and KID between consecutive intensities as:
By sampling augmentations to which the model is sensitive, we can improve robustness. We define the sensitivity of the model to changes in augmentation intensity as the ratio of the change in model accuracy to the change in KID:
A M Mlaa, a(;)) MA(ar-1) M A(as)
(2)
A AKID(a, a1-1)) DKw (#a, [| DKw(( (Ra, (etem))
Drm Dxm( (aan(aaaallsttao [[
sensitivity AMA
(1)
AKID
Here, M MA(a)) is the model accuracy at augmentation inten- sity a, and DKID [=ecaa)) is the KID between the aug- mented data intensity a and the original clean data. The normalization in A AKID ensures that KID values are com- parable across different augmentation types.
Our goal is to identify augmentation intensities that result in high sensitivity- that is, small changes in the augmenta- tion (as measured by KID) lead to large changes in model performance (MA). This indicates that the model is particu- larly sensitive to those augmentations, and training on them could improve robustness. To formalize this, we seek to
We then formulate an objective function Q to find the set
3

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
of intensities that maximizes sensitivity while ensuring ade- quate spacing between them:
values is visualized in Appendix Figure 13. Below, show the full training routine involving Sensitivity Analysis in Algorithm 1.
Q = arg max min AMA(as, 01-1)
Resource improvement over previous work in sensitivity analysis. Previous sensitivity analysis methods (Shen et al., 2021) compute g(a) using a uniformly sampled set of a values across the entire augmentation space. This approach requires evaluating the model at many intensities and of- ten necessitates offline generation of augmented datasets for each intensity and augmentation type. As a result, the storage complexity becomes the size of the original dataset multiplied by the number of augmentation types and inten- sities, leading to substantial storage demands.
oi 2S15L
-AKID 11-1) (4)
+M(a; ai-1)
In this equation, the term A AMa,,, a111) represents the change in model accuracy between intensities 0; and aq- We subtract A (KTD 0;-1) to favor intensity intervals where accuracy drops more than the image degradation in- creases, thus indicating higher sensitivity. Furthermore, the regularization term X(a; - 1)- (with À > 0) encourages spacing between intensities, preventing them from being too close together. In our implementation,
In contrast, our method performs sensitivity analysis online during training and adaptively samples intensities based on the model's 's responses. By estimating g(ax) iteratively and focusing only on necessary intensities, we eliminate the need for pre-generating augmented datasets. As result, our approach only adds about 0.2 (number of updates) * (eval- uation time) amount of time to the total training pipeline, making the use of sensitivity analysis practical for on-the-fly augmentation policy during training.
2.
Our objective seeks maximize the minimum value of this expression across all intervals, ensuring that even the least favorable interval is optimized.
To simplify the optimization, we introduce function g(x):
DKID(TrN [| g(a) = 1- - MA(aa DKID ( ooom |[Telan))
4. Experiments
+ Aa
(5)
Hardware. Each experiment is conducted on four NVIDIA RTX A4000 GPUs and 16 AMD Epyc 16-core processors. Sensitivity analysis experiments are conducted on one GPU and 4 processors.
The set of a values which fulfills Q has the following property: g(az) g(0z) g(as)= g(az2
g(an.) - y(ar.-1); in other words, optimal values are produced at equal intervals along the function g. Since g(a) is approximately monotonically increasing (as M A(a) de- creases and DKI ( =ctean) increases with increasing a), and its values lie within a known range, we can approximate the solution as:
Experiment Setup. For evaluation on real-world corrup- tions and data effiency, we train all experiments with the Segformer (Xie et al., 2021) backbone, robust and state-of- the-art architecture for segmentation. Experiments in down- stream fine-tuning from foundation model SAM (Kirillov et al., 2023), are restricted to their original VIT (Dosovit- skiy et al., 2021) architectures as the backbone. All experi- ments, with the exception of downstream fine-tuning experi- ments, are trained for 160k iterations regardless of approach, and only the best-performing checkpoints by mloU (mean -ovecctionnvve--niiin by class) are used for evaluation in results. Fine-tuning experiments initialized from foun- dation weights are trained for 80k iterations. Experiments within each table are run with the same hyperparameters with respect learning configuration, initialization, and architecture. Additionally, nearly all models share the same set of augmentations, with the exception of IDBH (Li & Spratling, 2023), which uses an additional two augmenta- tions (RandomFlip and RandomErase). We include these two additional augmentations in IDBH experiments to stay faithful to original release implementations by authors. We use official implementations for each method, and fix the random seed for each experiment such that they are repro-
Gmaxx .2) T,
0g g-d
i I,...,L
L
(6)
where Gmax is the maximum value of g(a) over the range of a, and --1 is the inverse function. Since we choose x 2 in our implementation, Gmax 2.
However, since we cannot explicitly compute g- due to g(ax being unknown in closed form, we iteratively esti- mate the values of a; using methods like the Piecewise Cubic Hermite Interpolating Polynomial (PCHIP), which is spline estimation technique. By sampling few initial points and fitting an interpolating function, we can estimate the intensities that satisfy our objective. A proof for equal spacing can be found in Appendix Section A. We show the pscudocode for sensitivity analysis in Algorithm of the appendix. Additionally, the iterative process for solving a
4

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
DU
DU
FTLM
truc
bloycle e
bicycle
(a) IDBH.
(b) Ours.
Figure 2. Special case on ACDC prediction: windshield wiper occlusion. We observe a special case of natural corruptions in rainy weather which cannot be directly simulated by the existing set of perturbations: physical occlusion by windshield wipers. We compare our method to recent SOTA in augmentation for robustness, IDBH (Li & Spratling, 2023). While IDBH involves random occlusion during training, ours does not. Our augmentation approach achieves comparable qualitative results with smaller set of augmentations. Additional visualizations can be found in Appendix Section F.
Fog
Rain
Night
Snow
Method
aAcef
mloUt
aAcet
mIoUt
aAce
mloUt
aAcef
mloUt
Baseline
89.70
55.10
87.41
42.82
54.39
14.89
83.23
41.22
AugMix
89.76
57.79
89.28
47.53
56.64
17.35
83.34
43.94
Augment
77.06
56.18
75.52
42.66
57.14
20.65
71.83
40.9
RandAug
88.24
53.99
86.92
43.10
56.03
18.08
83.35
41.86
Tri
85.79
55.16
84.35
41.26
54.52
17.02
77.99
42.64
VIPAug
92.04
60.28
89.19
46.41
61.10
17.40
85.72
45.04
IDBH
89.79
60.79
86.93
45.64
54.76
18.41
83.88
45.35
Ours
90.20
62.50
88.87
49.36
58.85
20.72
83.39
48.16
Table Evaluation of zero-shot adverse weather performance across data augmentation techniques. We evaluate each data aug- mentation method across four different weather scenarios from the Adverse Conditions Dataset with Correspondences (ACDC) (Sakaridis et al., 2021) dataset. Each model is trained only with clean Cityscapes data with the Segformer (Xic ct al., 2021) backbone. Our method, highlighted grey, maintains best performance across nearly all metrics three out of four scenarios, with relative mloU improvement over the next best method of up to 2.81% % fog. 3.85% on rain, and 6,20% snow.
ducible. More hyperparameter details for experiments can be found in Appendix Section G. Full experiment configu- rations will be released alongside the code implementation for full reproducibility results.
et al., 2020), AutoAugment (Cubuk et al., 2019), RandAug- (Muller &
ment (Cubuk et al., 2020), and Hutter, 2021), VIPAug (Lee et al., 2024), and IDBH (Li & Spratling, 2023). The Inception model used to compute KID in our method pre-trained on ImageNet; likewise, for policy-based augmentation techniques such Augmix and AutoAugment, the policies also based off of ImageNet. Different from other policy-based methods, our approach estimates the current model's sensitivity to perturbations relative to the Inception model pre-trained on ImageNet and utilizes the information in augmentation sampling; no additional "policy" parameters are trained. On real-world dataset evaluation for unseen weather and domain gap narios, method shows improvements performing model across almost all metrics. We include qualitative visualization of our model versus several other methods Figure 7 of the appendix, which shows inference on rainy weather sample. Amongst all methods, a common failure mode the presence of windshield wipers in rainy weather. A visualization of this can be found Appendix Section J.
Metrics. We use three different metrics for evaluating the performance of segmentation model: absolute pixel curacy (aAcc), mean pixel accuracy (mAcc), and mean intersection--ver--uui (mloU). Mean values are taken over object classes aAce will be more susceptible to class imbalances, although is the most intuitive.
4.1. Evaluation on Real-World Corruptions
To evaluate the robustness of our model in visual and graph- ics applications, we test on real-world adverse samples. While real-world adverse samples in most datasets are dif- ficult obtain, there are numerous real-world datasets for driving representing different cities and adverse weather scenarios.
the next best
We compare our results to seven methods: baseline model augmentation performed, AugMix (Hendrycks
where no
A break-down the performance on the ACDC dataset by
5

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Clean
Basis Aug
AdvSteer
IN-C
Dataset
Type
Method
aAcet
mloli1
aAsc1
mloU?
aAcc?
mloU]
aAcct
mloU1
TrivialAug
75.420
32,580
69.559
27.083
41.783
9.188
61.495
18.668
ADE20K
General
IDBH
76.220
33,950
72.752
30.651
40.557
9.475
61.971
19,091
Ouns
76.110
33.790
74.285
31.922
43,075
9.628
61.280
18.721
TrivialAug
90.050
57.900
87,837
52, 340
75.350
20.338
82.884
36,090
VOC2012
General
IDBH
90.610
60.570
89,262
56.876
20,810
81.819
36,933
Ours
90,800
61.140
89,555
58.183
69.690
21.470
82,519
38,834
TrivialAng
84,360
77.649
55.763
55.817
34,282
55.866
36.967
POTSDAM
Actial
IDBH
84,280
68.690
79.392
63.757
22.675
14.975
46,413
30.125
Ours
84,550
68.450
82,590
66,065
44,817
29.983
51,275
36,416
TrivialAng
98.730
69.180
97.317
51. 31.800
85,59 85.598
22.225
97.363
46.502
A212Haze
UGV
IDBH
98. 680
69. 300
98.346
64.615
85,545
19:490
97.
45,970
98.790
70.290
98.613
67.919
89,482
21.843
97.407
49,805
TrivialAug
95,570
74,300
86.117
56.952
69.785
30,593
82.664
44,332
Citysaaes
Driving
IDBH
95.530
73,930
93.160
68.052
71.932
29.388
83.041
Ours
95,780
75.530
94.305
71,5339
68 08.468
28,070
82,435
45,066
TrivialAug
98.890
62,000
97.939
49.237
97.243
32.182
98.425
51.512
Synapse
Medical
IDBH
99.150
67.720
98.912
63.550 63.504
95.143
29.760
98.486
53.475
Ours
99,250
71,380
99.082
68.828
90.282
30:310
96.779
56.013
Table Performance evaluation our method vs. SOTA on synthetic scenarios across different datasets. We evaluate our method and SOTA on ADE20K (Zhou et al., 2019). VOC2012 (Everingham et al., 2012), POTSDAM (for Photogammetry & Sensing). A212Haze (Narayanan et al., 2023), Cityscapes (Cordis et al., 2016), and Synapse (L.andman et al., 2015) datasets, across three synthetic corruption scenarios: individual basis augmentations (Basis Aug), compositions of photometric augmentations produced by sensitivity analysis in Adversarial Steering (AdvSteer) (Shen et al., 2021), and the synthetic augmentation benchmark ImageNet- C (IN CC) (Hendrycks & Dietterich, 2019). Our method consistently achieves improved performance synthetic corruption benchmarks while still maintaining or even improving clean evaluation accuracy.
weather type in Table 2. In total, the ACDC dataset has four different weather scenarios: Fog, Rain, Night, and Snow, where the largest relative boost over next-best method, IDBH (Li & Spratling, 2023), (6.20%) is in Snow scenar- ios. In three out of four weather categories, our method outperforms other methods, with the exception of Night scenarios. AugMix achieves higher aAcc but lower mloU than our method on Rain scenarios possibly due to class imbalances, such as the large number of pixels classified as "sky". While the total # of correct pixels is higher on AugMix, our method outperforms when averaged by class, on mloU. Night scenario visibility corruption stems from lack of lighting, as opposed to the other three, which may have more differences in object appearances and blurring effects. While our method does not perform worse in mIOU, we do perform worse aACC. This may suggest that the failure mode of our method in Night scenarios are due to smaller objects covering less pixel space. Performance both ACDC and the India Driving (IDD) (Varma et al., 2019) datasets across multiple methods can be found in Section C of the Appendix.
despite not having been trained on occlusion (RandomErase) augmentations.
4.2. Evaluation on Datasets
The results of previous experiments show the efficacy of our method in context of driving domains. this experiment, we demonstrate that our method also shows improvements across several datasets and visual computing domains com- pared to SOTA.
We evaluated our method on six semantic segmenta- tion datasets, ranging from generic everyday scenes to application-speccfic domains like driving or medicine. ADE20K (Zhou et al., 2019), VOC2012 (Everingham et al., 2012), POTSDAM (for Photogammetry & Sensing), Cityscapes (Cordts et al., 2016), Synapse (Landman et al., 2015), and A212Haze (Narayanan et al., 2023). In Table 3, we show mloU performance of our method versus the next- best augmentation technique, the SOTA baseline. We eval- uate on clean data and three different synthetic scenarios: individual transformations from the basis augmentations at uniform parameter intervals (Basis Aug), the combined perturbation benchmark from (Shen et al., 2021) (AdvS- teer), and ImageNet-C (IN-C) (Hendrycks & Dietterich, 2019) corruptions. On the synthetic benchmark ImageNet- C (Hendrycks & Dietterich, 2019), our model achieves im- proved scores, particularly in the robotics and medical domains. Our method performed worse primarily in the AdvSteer benchmark of Table 3, notably for Cityscapes and Synapse. This may be due to the sheer intensity of
Special case: co-occurence of windshield wipers and rainy weather. In the ACDC dataset, the rainy scenario eval- uation set contains co-occurences with windshield -wiper occlusion. This case is interesting in that occlusions are not included in any experiments except those of IDBH. In qualitative results, we observe that our method handles wind- shield wiper occlusions just as well, if not better, than IDBH. In Figure 2, we show an example of this, where our method shows comparatively less artifacts in the building and sky,
6

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Effect of Dataset Size mioU Performance In Adverse Weather
Effect of Dataset Size miou Performance In Domain Shift
miou
ours
ours
sugmix
augmix
autoaugment
autoaugment
trivialaugment
Uivielaugment
randaugment
randaugment
idbh
idbh
1330
1582
2800
1103
20.0
2000
4023
Semples
-Samples
(a) # Samples vs. ACDC mloU performance.
(b) # Samples IDD mloU performance.
Figure 3. Comparison of Ours vs. Other SOTA Methods: Ours (top. blue) outperforms all others in performance as the number of samples increases, while other methods plateau both (a) adverse weather data (ACDC) and domain shift (IDD).
ViT+SAM
Method
aAccf
mloUt
mAcct
Method
aAccf
mAcc
mloUt
Baseline
94.19
66.67
75.28
AugMix
93.99
66.07
73.60
Baseline
84.93
62.84
52,20
AutoAugment
93.84
64.99
72.09
AugMix
84.69
63.25
54.18
RandAug
92.60
59.37
66.03
AutoAugment
85.17
61.28
53.11
93.55
65.03
71.71
RandAug
85.16
59.33
51.95
VIPAug
92.98
63.67
70.86
TrivialAugment
84.87
59.92
50.58
IDBH
93.62
65.49
74.96
VIPAug
84.29
61.00
51.93
Ours
93.88
68.03
75.96
IDBH
85.14
62.82
54.35
Ours
85.37
65.18
54.84
Table S. Domain adaptation; fine-tuning small target do- main dataset. We show performance on the validation set the ACDC Snow dataset, after training for 20k iterations the ACDC Snow training set. All experiments are initialized with a Segformer-bo bo model pre-trained Cityscapes.
Table Performance ACDC when fine-tuning downstream segmentation with SAM. We show additional comparisons when initialized with SAM weights, similarly to results Table
benchmark corruption-the AdvSteer benchmark applies combination of intense perturbations (not the same as the augmentations used during training), resulting in extreme case from the original distribution. This may be related to degraded performance on Night scenarios in ACDC evalu- ation, as both scenarios heavily сorrupt visibility based on color. Examples of the AdySteer benchmark corruptions can be found Appendix Section K. We emphasize that our method not necessarily bound image segmentation- we find similar boosts in performance in preliminary experi- ments with classification (see Appendix Section 12).
task with Cityscapes. choose Cityscapes due to the avail- ability of real-world corrupted images (ACDC and IDD) to evaluate on. In experiments, formance on our method all three metrics. While the largest boost in robustness stem from robust foundation model fea- tures, our results suggest that our method can complement approaches centered around model architecture (such as Segformer). Additional results on downstream finetuning with D2 can be found in Table 8 of the Appendix.
observe the highest per-
4.4. Domain Adaptation With Small Datasets
evaluate the efficacy of our approach domain adap- tation settings, we finetune Segformer Cityscapes models on the training set of the ACDC snow data, then evaluate the validation set of the ACDC snow data. The main difference from previous experiments here is that the model has been trained on target domain data, rather than being evaluated in a zero-shot setting. These results can be found in Table 5. Interestingly, the baseline approach seems to work second-best for domain adaptation, behind our method.
4.3. Downstream Finetuning with Foundation Models
A popular choice for boosting feature robustness is fine- tuning downstream tasks from foundation models. In these experiments, we examine how our approach can comple- ment robustness provided by foundation models when fine- tuning on downstream tasks. We first initialize a distilled SAM (Kirillov et al., 2023) model on the ViT-Small (VIT-S) architecture, then fine-tune on the semantic segmentation

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Disteer
2.0
2.0
2.0
15
1.5
112
3.0
1.0
68
0.0
0.5
0.3
0.2
23
2.d
Backe
2.0
Cur Cordlatte Gesatee g/o
Bokee
Bakee
Naw
15
1.5
1.0
3.8
1.D
2.0
6.5
0.5
0/8
0.0
00 3.5
0.5
0.5
1
0.5
00
055
0.5 Derradette Eipeaaty fa
20 40
Purrabetee Eoweait 101
Perrathatie Eeoeadty od
Factubesa Edaaaty ta1
Dertataatio 22n0EE
Figure 4. Sensitivity curves (g values) for color channels, noise, and blur throughout training of Cityscapes. We visualize how the estimated cumulative sensitivity curve, Equation 5, changes for RGB, HSV, Gaussian blur, and Gaussian noise during augmented training. In this plot, the most recent curve is opaque, while others decrease in opacity in order of recency. The red X markers indicate the values which a values are selected (horizontal axes). Surprisingly, most curves remain largely stagnant throughout training, with the exception of Hue HSV (teal, center), which changes drastically as the model generalizes. This may suggest that Hue is a highly adversarial factor in model generalization. Ablation study results in Table 6 support this, where the model trained without photometric augmentations demonstrate a significant decrease in performance.
Clean
Basis Aug
AdvSteer
IN-C
ACDC
Method
aAcof
mloUf
aAcc?
mloUf
aAcet
mloU/
aAccf
mloUt
aAcet
mloUt
Baseline
95.610
75.130
92,042
65.319
62.040
21.995
79.437
38,362
78.49
37.54
OurS..g
95.780
75.500
93.405
68.877
71.070
27.997
83.032
44,385
78.13
43.69
Oursupp
95.740
75.210
92.544
69.002
64.907
22.437
80.817
40.876
75.74
37.97
Ours- Warmuu
95.830
75.430
94.458
71.891
69.138
28.472
84.438
45, 848
79.78
44.66
95.740
75.200
94.304
71.213
69.678
27.235
85.135
46.219
80.95
43.17
Ours
95.790
75.100
94.439
71.665
70.605
28.895
83.844
45.617
80.13
44.67
Table Ablation study results comparing different variants of our method. We compare: (1) baseline trained with no augmentations, (2) variant of our method that only augments with photometric augmentations (Ours-.g), (3) a variant of our method that only uses geometric augmentations (Ours-pp). (4) a variant our method trained without clean training warm-up, (5) a variant our method with uniform augmentation (OurSu ) of computed sensitivity analysis values and (6) our full method combining informed probability sampling, and adaptive sensitivity analysis, and augmentation types (Ours).
We suspect that the degradation in performance is due to overfitting: with smaller datasets and less challenging aug- mentations, the model may be more prone to overfitting than generalization. Our method may mitigate this by sampling evenly difficult augmentations with respect to the current model state.
data efficient than others. Interestingly, not all methods show increased robustness to adverse weather as the number of samples increases for training, indicating that in some cases, scaling data may not necessarily mean increased robustness.
4.6. Ablation Study
4.5. Data Efficiency
We examine several variants of our method to determine the impact of individual components in an ablation study: a baseline trained only with random cropping, a variant of our method using only geometric augmentations, a variant of our method using only photometric augmentations, a vari- ant of our method without clean training warm-up, and a variant of our method using uniform sampling instead of the Beta- eita-Binomial sampling described in Algorithm 1. Uniform sampling of augmentation parameters computed with sen- sitivity analysis decreases generalization to both synthetic and real-world corruption benchmarks by small margins. In addition, training without clean warm-up produces similar results to that with warm-up, suggesting that warm-up is optional. In our case, warming up with clean evaluation
We also analyze data efficiency of our method in comparison to other data augmentation methods by training various Seg- former models with varying training dataset sizes. For each augmentation method, we train five models with training dataset sizes of 1000, 2000, 3000, 4000, and 5000 sam- ples from the Cityscapes dataset. We plot the progression of mloU (Minimum Intersection over Union) performance (higher the better) on (a) adverse weather data (ACDC) and (b) the domain shift setting (IDD), as shown in Figure 3. Our method, in blue, shows consistent improvement on adverse weather and domain shift evaluation with increasing number of samples, and maintains best mloU performance across each # of samples slice, suggesting that our method is more
8

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
reduces the total number of sensitivity analysis updates, making warm-up with clean evaluation marginally less re- source expensive (~ 0.5 GPU hours total), Interestingly, while clean performance remains largely the same across all models, the largest decrease in performance on unseen corruption benchmarks comes from the lack of photometric augmentations.
Impact Statement
The goal of this work is to introduce an augmentation frame- work which enhances generalization corruptions. Our work is driven by problems in real-time systems such robotics. To the best of our knowledge, com- bating natural corruptions has only positive implications for the systems which it is a concern; for example, autonomous vehicles may be more robust to adverse weather or unex- pected scenarios.
nnturally-occurring
4.7. Color Channel Sensitivity Across Training
Sensitivity analysis can also be useful for interpretable anal- ysis on-the-fly during training, aside from being used in augmentations directly. To analyze how sensitivity changes with respect to different channel values, we plot each g curve computed from Equation 5 for RBG, HSV, Noise, and Blur channels across training for our Cityscapes ex- periments in Figure 4. Note that the curves in this figure are based on individual color channels and are separate from those used during training for generalization analysis purposes. Computed a values will center towards regions with higher sensitivity relative to the current model. From this visualization, we observe that Hue curves (teal, cen- ter) are most volatile during training, with most sensitive augmentation parameters falling towards a values close to 1.0 in the beginning of training. As the model generalizes, the Hue curve converges slowly towards a values centered around 0.5, similarly to other curves. This suggests that Hue may be a significant adversarial factor in model robustness compared to other channels.
References
Binkowski, M., Sutherland, D. J., Arbel, M., and Gret- ton, Demystifying mmd gans. arXiv preprint arXiv:1801.00 1en::000000100 2018.
Cordts, M. Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene under- standing. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Cubuk, E. D., Zoph, B., Mané, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113-123, 2019. doi: 10.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. Randaugment: Practical automated data augmentation with reduced search space. In Advances in Neural Information Pro- cessing Systems, pp. 18613-18624, 2020.
5. Discussion and Conclusion
In this paper, we present a method for musstivity-informme augmented training for semantic segmentation. Our work inspired by applications in real-time perception systems such as robotics, where natural corruptions may be abundant at inference time. Our method combines the information granularity of sensitivity analysis- basee methods and the scalability of data augmentation methods, which run on-the- fly during training. In our results, we show that our method achieves improved robustness on zero-shot real-world ad- verse weather and domain shift scenarios, in addition to improvements on synthetic benchmarks like ImageNet-C. Additionally, evaluation real world datasets show clear improvements over current SOTA methods for augmenta- tion. Our model can complements other approaches for model robustness such as architecture design and down- stream fine-tuning with foundation models.
Dong, Y., Fu, Q.-A,, Yang, X., Pang. T., Su, H., Xiao, Z., and Zhu, J. Benchmarking adversarial robustness on image classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenbomn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representutions, 2021. URL https:// openreview. net www.moorrrriorrr wow.mrii--x.ccbaaVTyy...
Everingham, M., Van Gool, L.,, Williams, C. K. I., Winn, J., and Zisserman, A. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascch oew.wttrwg/aaleeeeeeo/oceeoo/ooe
A Acknowledgement: This research is supported in part by the U.S. Army Research Labs Cooperative Agreement on "A "I and Autonomy for Multi-Agent Systems".
2012.
for Photogammetry, Potsdam:
S. and Sensing, R. labeling contest.
2d semantic
sem-label-potsdam.ayy

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Franchi, G., Belkhir, N., Ha, M. L.,, Hu, Y., Bursuc, A., Blanz, V., and Yao, A. Robust semantic segmentation with superpixel-mix, 2021. URL https: arxiv.
28070. URL https:://bbi.orr//////// v3814.28070.
aag/abs/2100 00968.
Li, L. and Spratling, M. W. Data augmentation alone can improve adversarial training. In The Eleventh In- lernational Conference on Learning Representations, 2023. URL hww.h::p////ec.m //pprr/ve//eeeeieww net/ forum? id=y4uc4N:TWag,
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Confer- ence on Learning Representations, 2019. URL https: //openreview www.//ee//et.... forum? id=Bygh9
Mintun, E., Kirillov, A., and Xie. S. On interaction between augmentations and corruptions in natural corruption robustness. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3571-3583 Curran Associates, Inc., 2021. URL https: httpsppss///oooeengg...eee cc wttpppreiiilleeeppeee////// 1 449788500888f477Fc6666111133333 pdf.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and pertur- bations. Proceedings of the International Conference on Learning Representations, 2019.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neu- ral networks. In International Conference on Learning Representations, 2017. http:: /pppnreeee.....eeeo net/forum? iidiiddiiie HHHHggTIXXx.
Muller, S. G. and Hutter, F. Trivialaugment: Tuning-fiee yet state-of-the-art data augmentation. In IEEE/CVE International Conference on Computer Vision (ICCV, pp. 754-762, 2021.
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2020.
Myronenko, A. and Hatamizadeh, A. Robust semantic seg- mentation of brain tumor regions from 3d mris. In Crimi, A. and Bakas, S. (eds.), Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 82-89, Cham, 2020. Springer International Publishing. ISBN 978- -38---330-46643-5
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: critical analysis of -istriiiittiittionn generalization. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 8340- 8349, October 2021.
Narayanan. P., Hu, X., Wu, Z., Thielke, M. D., Rogers, J. G., Harrison, A. V., D' Agostino, J. A., Brown, J. D., Quang, L. P., Uplinger, J. R., Kwon, H., and Wang, Z. A multi- purpose realistic haze benchmark with quantifiable haze levels and ground truth. IEEE Transactions on Image Processing, 32:3481 2023. doi: 10.1109/TIP.202.. 3245994.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pp. 404100142200 2023.
Olsson, V., Tranheden, W., Pinto, J., and Svensson, L. Class- mix: Segmentation-basel data augmentation for semi- supervised learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1369-1378, January 2021.
Landman, B., Xu, Z. Iglesias, E., Styner, M., Langerak, T. R., and Klein, A. Multi-atlas labeling beyond the cranial vault workshop and challenge. wwwp://wwww..oo./////y/s/seyaaaa 2015.
Oquab, M., Darcet, T. Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M.. Ballas, N., Galuba, W. Howes, R., Huang, P.-.,, Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Je- gou, H., Mairal, J., Labatut, P., Joulin, A., and Bo- janowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: / ooppreeveew. net forum? d=a 62FFt
Lee, I. Lee, W., and Myung, H. Domain generaliza- tion with vital phase augmentation. In Proceedings of the Thirty-Eigh telligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial In- telligence, AAAI224AAAT24/AAT doi: 11106//a/i.v33556
AAAI Conference Artificial In-
AAAI Press,
2024. ISBN 978- 1-88----7755-888
10

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Rusak, E., Schott, L.., Zimmermann, R. S., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W. A simple way to make neural networks robust against diverse image corruptions. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (eds.); Computer Vision - FCCV 2020, pp- 53-69, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58580-8
ments. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1743-1751, 2019. doi: 10. 10.1109/ACV.2211900000
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. Caltech-uesd birds-200-2011. Technical Report CNS- TR-2011-001 California Institute of Technology, 2011.
Saikia, T., Schmid, C., and Brox, T. Improving robustness against common corruptions with frequency biased mod- els. Proceedings of the IEEF/CVE International Con- ference on Computer Vision (ICCV), pp.
Xiangang Xu, H. 7. and Jia, J. Dynamic divide-and-conquer adversarial training for robust semantic segmentation. In ICCV, 2021.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, P. Segformer: Simple and efficient design for semantic segmentation with transform- ers. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 12077-12090. Curran Associates, Inc., 2021. URL https://proceedinng htw.roeeeednnns.nnri.......... cc cc/ppppe_fillee/ppprr///////// 61f1127751b6ee222244000055553000 Pappe. pdf.
October 2021.
Sakaridis, C., Dai, D., and Van Gool, L. ACDC: The ad- verse conditions dataset with correspondences for seman- tie driving scene understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2021.
Schneider, S., Rusak, E., Eck, L.,, Bringmann, O., Brendel, W. and Bethge, M. Improving robust- ness against common corruptions by covariate shift adaptation. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.), Advances in Neurul Information Processing Systems, vol- ume 33, pp. 11539-11551. Curran Associates, Inc., 2020. URL https:: https::/proceedingg. neurips. oc hc/paaperfflle/eeeeeeeeeeeeeseee fttees///aaaee///22///l.
Yi, C., YANG, S., Li, H., peng Tan, Y., and Kot, A. Benchmarking the robustness of spatial-temporal mod- els against corruptions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: / // wwwpneeview..eeeeeee for um i MOMMIIMI33
pdf.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference Computer Vision (ICCV), 2019.
Shen, Y.. Zheng, L.. Shu, M., Li, W., Goldstein, T., and Lin, M. Gradient-free adversarial training against image corruption for learning- beasenngbbased steer- ing. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 26250-26263. Curran Associates, Inc., 2021. URL https: https://proceeddii neurue. oc ppaper_fillespppeer//00////////
Zhang. X., Liu, C., Ou, N., Zeng, X., Zhuo Z.,
Duan, Y., Xiong, X., Yu, Y., Liu, Z., Liu, Y., and Ye, C. Carvemix: A simple data augmentation method for brain lesion segmentation. Neurolm- age, 271:120041, 2023. ISSN 1053-8119. doi:
oeeaaff33f0saaaccco3888222222222 Pape. pdf.
URL https::/www.scienneed.ee com/ scieenee/artiicll//////s33.11111
Shu, M., Shen, Y., Lin, M. C., and Goldstein, T. Adversarial differentiable data augmentation for autonomous systems. In International Conference on Robotics and Automation (ICRA), pp. 14069-14075, 2021.
Zhao, A., Balakrishnan, G., Durand, F., Guttag, J. V., and Dalca, A. V. Data augmentation using learned transfor- mations for one-shot medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1118-2826612 2016.
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Varma, G., Subramanian, A., Namboodiri, A., Chandraker, M., and Jawahar, C. Idd: A dataset for exploring prob- lems of autonomous navigation unconstrained environ-
Zheng, Y., Zhang, Z., Yan, S., and Zhang, M. Deep autoaug- mentation. In ICLR, 2022.

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-321, 2019.
12

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Appendix
A. Proof on Equal Spacing
1. 00 represents an augmentation intensity O, i.e. clean image.
2. MA approaches as models get better, but to be more precise, the function g(a) should be
DRID ( [
g(a) MM((0) .M((a)
λa.
DRID (Tam [eleam
3. Q is the set of points fai,.... oz) that maximizes the following:
min (A(a,,,;11) - + M(a; 0---1)
sitt
min isiS
((:)) g(ai-1).
4. From the definition of g(a), we note
(()) 0
g( Gmax (MA(0) MA(a1)) - 1 + Ao
In our implementation, we normalize M. so that g(a)) = 2.
Proof of equal-spacing:
We will prove that g' = Gmax/ L, i.. given the set of a; values that fulfills Q, the set of g(a;) are equally spaced along the y-axis of the function g(a). BWOC, assume g' Gmax/ L. Then, Vi, g(a,) - g(a;-1) 2 g' G L. Taking the sum
over yields a contradiction:
L Gmax = g((a) 900) Σ g(as)) - g(an-1) L Gmax
Gmax
Thus, g' S Gmax/ / L. If we assume g(a) is continuous over [0, ar..). then by the Intermediate Value Theorem, Vi, 3a, such that
g(a,) Guax i L
Then, g(a)) - g(05-1)= = Gmax/ T., so the maximum g' G / can be attained for specific set of points Q. In the paper, we assume g(a) is strictly monotonically increasing because M A(a) decreases and DKID Zelean) increases as increases. Based on this, we obtain formula
B. Sensitivity Analysis Pseudocode
Algorithm 2 Fast Sensitivity Analysis
Data: Number of levels L, Uncertainty threshold
Result: Perturbation Levels {a1...., 01.-1 g(a)
Equation 5 points + (00, 0), (ar, 2)} loop
é < PCHIP(points) for i 1...I do
30
Estimate(ć, 2i/L.) (w)) + Estimate upper and lower y-values of à at x è + PCHIP inspoitttiiisrttr/r/ &n + PCHIIP(piints inserttsinserttyy)))) ai <- Estimate(é, yi) a, + Estimate(C, yu) 6
+ (aki - //2
31
end
32
a²,e" <- Choose level with max 6 if e then Break loop:
points.insert(cer (a()))
enn;
13

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
C. Overall Performance on ACDC and IDD
Weather ACDC
Domain
Method
aAce?
mloU't
mAcot
BAce
mloU't
MAce?
Baseline
76.31
35.48
47.36
85.82
38.44
59.14
AugMix
79:57
40.90
52.74
86.52
40.50
62.43
AuloAugment
62.24
RandAug
78.46
39.07
52.32
85.54
38.99
59.82
TrivialAug
75.50
38.56
53,62
85.23
39.61
61.04
78.65
41.67
40.48
61.74
VIPNug
81.85
42.26
53.59
85.94
38.97
58.79
Ours
80.16
45.45
57.58
85.76
40.33
63.03
Table Evaluation results on Unseen Real World Driving Datasets. We conduct zero-shot evaluation of Cityscape models on both ACDC (Sakaridis et al., 2021) and IDD (Varma et al., 2019) datasets, which represent adverse weather and domain transfer India respectively. Our method achieves clear improvements compared to other methods which require chained, more computationally expensive augmentations external augmentation models in terms of generalization real world scenarios, with relative mloU improvement up 9.07% on ACDC compared to the next-best, IDBH.
D. Downstream Finetuning with Dino
ViT+DinoV2
Method
aAcc
mAcef
mloUf
Baseline
77.65
45.83
32.70
Augmix
79.99
51.63
41.38
Augment
81.18
55.93
43.65
RandAugment
80.42
54.02
43.25
TriminlAugment
82.56
54.27
43.58
IDBH
84.45
60.22
48.69
Ours
84.13
62.92
49.82
Table 8. Performance of Cityscapes models on unseen ACDC weather evaluation set across different augmentation methods, when fine-tuned from DinoV2 (Oquah et al., 2024) with ViT (Dosovitskiy al., 2021) backbone.
E. Downstream Finetuning with SAM
Fog
Rain
Night
Snow
Method
aAccf
mloUt
aAcet
mloUt
aAcc
mloU
aAcct
mloUf
Baseline
93.96
68.98
91.59
58.53
65.73
29.41
90.54
$6.27
AugMix
94.38
70.49
92.18
60.66
62.47
25.35
90.46
58.60
Augment
94.16
69.18
92.57
56.63
63.66
26.69
90.98
58.19
RandAug
93,92
67.67
91.70
55,30
65.19
24.00
90.47
$6.70
TrivialAug
93.71
66.45
91.06
53.09
60.57
20,25
90.56
$4.67
IDBH
94.12
70.61
92.06
55.71
65.25
28.22
90.98
59.67
VIPAug
93.92
67.54
91.06
55.49
63.00
25.01
89.88
$7.85
Ours
94.34
70.98
92.66
57.92
66.93
27.24
91.08
57.32
Table Evaluation on ACDC adverse weather performance with SAM downstream finetuning.
F. Qualitative Results Window Wiper Occlusion in ACDC
G. Detailed Experiment Hyperparameters
Let AA- [RandomCrop, Contrast, Equalize, Posterize, Rotate, Solarize, Shear X, Shear Y, Translate X, Translate Y, Color, Contrast, Brightness, Sharpness) be the standard set of augmentations used with AutoAugment.
14

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Method
Max Iters
LR
Optimizer
Augmentations
Batch Size
Backbone
Baseline
160,000
6e-05
AdamW
RandomCrop
SegFormer--O
Augmix
160,000
6c-05
AdamW
AA
SegFormer-h0
AutnAugment
160,000
6c-05
AdamW
AA
SegFormer-h0
RandAug
160,000
6c-05
AdamW
AA
SegFormer-bd b0
TrivialAug
160,000
6c-05
AdamW
AA
SegFormer-bo b0
IDBII
160,000
6c-05
AdamW
AA U (RandomFlip.
SegFormer-b0
RandomErusing]
Ours
160,000
6c-05
AdamW
AA
SegFormer-h0
To
1600:
T 9600;
Ours
160,000
6c-05
Adamm
AA
SegFormer-h0
W arrmap 6400
Table 10. Experiment hyperparameters for Table and Table 3. All experiments are trained under similar hyperparameter settings, with each evaluation conducted on the highest-performinng mloU checkpoint. In comparisons, prioritize official implementations released by authors and avoid re-implementations. Additionally, most comparisons use the same set of augmentations to ours, with the exception IDBH & Sprating,, 2023), whose original implementation includes RandomFlip RandomErasing, For all experiments, we use the SegFormer-be bo backbone (Xie et al., 2021), which is a recent state-of of-the-art segmentation--ppcialll architecture.
H. Qualitative Results on Synapse
liver gollbladder
liver right kidney
gallbladder
gallbladder
right kidney
nght
nont kidney
Dolneer e
Paporcdas
Pannrros
oorta
Ponereas aorta
aorta
aorta
left kidney
Deft_kddney
Siftey
kidney
(a) Ground truth.
(b) TrivialAugment Prediction.
(c) IDBH Prediction.
(d) Our Prediction.
Figure 6. Qualitative evaluation on multi-organ segmentation with motion blur corruption. We show predictions on a motion bluurrd sample from the Synapse (Landman al., 2015) dataset for TrivialAugment (b), IDBH (c), and Our method (d), against the ground truth (a). Our method able to segment right and left kidneys, liver, aorta accurately. In contrast, the TrivialAugment prediction is unable distinguish both kidneys.
Qualitative Results on Rainy Data
building
(a) AutoAugment Prediction.
(b) IDBHI Prediction.
(c) Our Prediction.
Figure 7. Qualitative comparison snowy urban driving sample between AutoAugment (Cubuk et al., 2020), IDBH (Li & Spratling, 2023), and Ours. In this example, cach method (AutoAugment, IDBH, Ours) is trained on clean Cityscapes data representing sunny weather, then evaluated adverse weather samples. Despite having rainy data the training set, our method is able segment the driving noticeably clearer than other methods. In particular, other methods consistently struggle to segment the vehicle confidently.
15

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
J. Special Case: Windshield Wiper Occlusion
(a) Ground Truth.
(b) Autonngment.
(c) IDBH.
(d) Ours.
(e) Ground Truth.
(I) AutoAugment.
(g) IDBH.
(h) Ours.
BEE
Ground Truth.
ii AutoAugment.
(k) IDBH.
Ours.
(m) Ground Truth.
(n) AutoAugment.
(o) IDBH.
(p) Ours.
Figure 8. More examples of special case on ACDC prediction: windshield wiper occlusion.
J.1. Details on Basis Augmentations
Previous work in robustification showed that learning with a set of "basis perturbations" (BP) significantly improved zero-shot evaluation against unseen corruptions (Shen et al., 2021) for image classification and regression tasks, such as vehicle steering prediction. The intuition behind basis perturbations that the composition of such perturbations spans much larger space of perturbations than may be observed in natural corruptions; observed zero-shot performance boosts on unseen corruptions subsequently might be attributed to learning model which is robust to basis perturbations. In our method, we extend this concept and introduce a more generalized and larger set of basis perturbations in sensitivity analysis to determine the most productive augmentation during training.
Let D Positive, Negative) describe the set of augmentations applied either a positive (lighter) direction or negative (darker) to either one channel an image or parameter of an affine transformation applied to an image.
Let P {R, G, B, S, V) describe the of channels in RGB and HSV color spaces which may be perturbed; in other words, these augmentations are photometric.
Then, let G {Shear X, ShearY, T (iaaaatexX,Traaattttee TranslateY, Rotate) denote affine, or geometric, transformations which are parameterized by a magnitude value.
Finally, let = (Noise, Blur} be the set of augmentations not applied along channel dimensions. Specifically, we use Gaussian Noise and Gaussian Blur.
16

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Thus, the set of all basis augmentations Ag used in robustification is Ag
To compute lighter darker channel augmentations of RGB HSV channels, we use linear scaling. Let the range of channel be mii m For lighter channel augmentations, we transform the channel values ve by an intensity factor like so:
d'c c'Umax + ax)
vc
Likewise, for darker channel augmentations, the transformation can be described like so:
rec O'Ummn + (1 a:) vC
The default values are Umin = and Umax = 255. For channel augmentations, we set the maximum channel values to be 180. For channel augmentations, we set the minimum channel values to be 10 to exclude completely dark images.
Afline transformations can be represented 3 matrix, which, when multiplied with a 2-dimensional image, produces geometrically distorted version of that image. Affine transformation matrices are typically structured in the form:
Shear x
TN Tg
M
Sheary
1
for shear and translation transformations. For rotations where the center of the image fixed as the origin point (0, 0), the transformation matrix defined as:
cost
sinO
0
Mrot
sinf
coso
0
0
0
1
To account for padded values in images after affine transformations, zoom in images to the largest rectangle such that padded pixels are cropped out.
All augmentations are parameterized by a magnitude value ranging from 0 to 1. A magnitude value corresponds to the most severe augmentation value. More details on exact parameter value ranges can be found in the appendix. Conversely, a magnitude value produces no changes the original image, and can be considered an identity function. We account for the symmetry of these augmentation transformations by considering both positive values and negative values separate augmentations. The fast adaptive sensitivity analysis algorithm introduced in the next section relies the property that increasing magnitude corresponds to increasing "distance" between images. Thus, augmentations cannot simply span the value ranges- -1 to 1, and we separate them instead to different augmentations (positive and negative).
We apply these augmentations on-the-fly in online learning rather than generating samples offline. Doing so greatly reduces the offline storage requirement by one order magnitude. Suppose L intensity levels are sampled for each basis augmentation. Then, offline generation of perturbed data requires up to L x (IP) + Gl) + 24L additional copies of the original clean dataset. With online generation, we avoid offline dataset generation entirely and only need the original clean dataset to be stored, similar to standard vamilla learning.
R
G
G
B
H
S
B
N
Figure 9. Visualization of cach photometric augmentation transformation bedroom image. Up indicates the "lighter", positive direction and J. indicates "darker". negative direction. "B" and "N" indicate noise, respectively.
17

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Shear
Shear
Shear's
Shear
Trans
Trans
Trans
Trans Y
Rotate
Rotate
Figure 10. Visualization of various geometrie augmentations applied to sample image of house. We use the following geometric transformations our sensitivity analysis scheme, which are also analegous to the set transformations used by other methods (Cubuk al., 2019; Zheng et al., 2022). Up arrows indicate augmentation in the positive, left, direction, while down arrows indicate augmentation in the negative, or right, direction.
Brightness
Brightuess
Color
Color
Contrast
Contrast
Sharpness
Sharpness
Posterize
Solarize
Figure Additional augmentation types used sensitivity analysis, which are used other methods such as AutoAugment. While these photometric tranformations are used in other methods, the transformations also overlap with the photometric transformations shown in Figure 9, namely HSV perturbations. However, we still conduct sensitivity analysis evaluation on these transformations for completion.
18

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
K. AdvSteer Benchmark Examples
Figure 12, AdvSteer benchmark examples.
19

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
L. Clean Performance on Different Backbones
PSPNct (Zhao 2017)
SegFormer
2021)
Method
aAcet
mAcef
mloUt
aAcet
mAcef
mlnUf
Bascline
63.770
48.695
35.715
86.825
57.280
48.365
Augmix
94.770
74.400
66.740
95.520
81,430
73.390
Augment
95.130
77.210
69.630
95.550
81.390
73.820
RandAugment
95.060
76,770
69.360
95.610
82.390
74.560
Augment
95.090
75.930
68.620
95.640
83.210
75.130
95.100
79.320
71.840
95.880
84.070
76.330
Table Comparison of clean evaluation performance across different augmentation methods on Cityscapes. evaluated our sensitivity-informee augmentation method against popular benchmarks on PSPNet and SegFormer. The baseline represents training with no augmentations.
M. Results on CUB Dataset for Classification
InceptionV3
Method
Clean
Basis Aug.
AdvSteer
IN-C
Baseline
41.647
15.965
3.679
20.501
Augmix
35.865
15.274
4.810
20.394
AutoAugment
16.793
7,219
2.575
8.158
Augment
33.914
13.338
4.229
17.586
RandAugment
36.624
15.466
4.821
19.345
Ours
47.670
18.122
5.276
21.822
Table 12. Performance CUB (Wah et al., 2011) dataset with InceptionV3 (Szegedy et al., 2016) backhone.
N. Fast Sensitivity Analysis Illustration
166
Evaluite ni candidate Us.
X
update curve
12
1.2
Condlctt candidate with min points. solve for la
0.8
0.8
and repeat
022
035
04
05
0.
012
04
04
055
0-1 02
03
05
Figure 133. Illustration of fast sensitivity analysis. Fach iteration of the fast sensitivity can be intuitively visualized. Since assume general monotonicity of the curve, we first initialize a candidate curve (a line the first iteration). We solve for the candidate perturbation levels â based the solution in Equation 6. In the next step (middle), evaluate the candidate level with the greatest uncertainty and adjust the candidate curve, the dotted red line, using PCHIP on the evaluated levels, which are guaranteed to be true points along the function from Equation 5. In the next step (right), we the previous two steps until the maximum uncertainty of any candidate level values falls below threshold 0.05.
and solve for candidate levels, repeating the process in the
20

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
O. Sensitivity Analysis Computed Curve Comparison
Perturh
Method
Pi
P2
P3
P4
Baseline
0.100
0.300
0.500
0.700
R-
Adaptive
0.149
0.253
0.399
0.604
Baseline
0.100
0.200
0,400
0.600
G+
Adaptive
0.103
0.204
0.395
0.619
Baseline
0.200
0.300
0.500
0.700
B-
Adaptive
0.146
0.328
0.551
0.788
Baseline
0.200
0.400
0.600
0.800
R.
Adaptive
0.225
0.503
0.625
0.803
Baseline
0.200
0.400
0.600
0.800
G4
Adaptive 0.256
0.447
0.607
0.812
Baseline
0.200
0.500
0.700
0.800
B.
Adaptive
0.231
0.450
0.594
0.730
Baseline
0.100
0.300
0.400
0.900
Ht
Adaptive
0.268
0.406
0.508
0.809
Baseline
0.200
0.500
0.600
0.800
S-
Adaptive
0.243
0.439
0.589
0.744
Baseline
0.200
0.400
0.600
0.700
V
Adaptive 0.193
0.360
0.517
0.680
Baseline
0.200
0.400
0.500
0.600
H1
Adaptive 0.279
0.433
0.548
0.699
Baseline
0.200
0.400
0.600
0.900
S,
Adaptive 0.199
0.344
0.562
0.847
Bascline
0.200
0.400
0.600
0.800
Vi
Adaptive
0.197
0.397
0.594
0.797
Baseline
blur
Adaptive
17
23
31
Baseline
10
noise
Adaptive
6.4
12.4
17.7
26.9
Table 13. Comparison of computed perturbation levels using baseline (Shen et al., 2021) sensitivity analysis method adaptive method. ps is for all RGB/HSV channels, 19 for blur, and 50 for noise. In previous work, each perturbation level is chosen from certain number sampled, discretized values. Additionally, these perturbed datasets generated offline in additional step before training. Our fast sensitivity analysis enables sensitivity analysis to be performed on the fly during training, and offers much more dynamic, accurate, and descriptive sensitivity curves.
21

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
O.I. KID FID Relative Error Comparison with Scaling Sample Sizes
Relative Error Across Sample Sizes
0.20
KID
FID
Relative Error
0.15
0.10
0.05
0.00
50
100
200
Number of Samples
Figure Relative error KID FID over several sample sizes. plot the relative emorr computed KID and FID values over several sample sizes, with the reference value being the computed value for each at 500 samples. From this, see that FID is significantly biased toward number samples used for evaluation. We can reduce the evaluation KID values in sensitivity analysis by notable fraction due to this property.
P. Train-time Evaluation on Perturbed Datasets
Red Eval
Green Eval
Blue Eval
Evo
Saturation Eval
Eval
Eval
150
Figure 15.. Evaluation perturbed test datasets over training iterations. We show the evaluation on each perturbed dataset during training our model and baseline for VOC2012 dataset.
22

---

Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation
Q. Adaptive Sensitivity Analysis with Different Number of Levels
Cumulative Seasilivily vs Perturbation Intensity for 5 Levels
Cumulative Sensitivity vs Perturbation Intensity for 10 Levels
2.00
2.00
Comalative Sensitivity gio)
1.75
Comuuativv Sensitivity glo)
1.75
1.50
1.50
Iyper
1,25
1.25
upeer
1.00
1.00
dater.C
daterr
0.75
0.75
0.50
1230
0.50
Spear
0.25
dater
0.8
0.0
0.2
0.4
0.6
0.8
1.0
Perturbation Intensity (o))
Perturbation Intensity (a)
Cumulative Sensitivity vs Perturbation Intensity for 15 Levels
Cumulative Sensitivity Perturbation Intensity for 20 Levels
Comulative Sensitivity g(o)
Cumulative Sensitivity g(o)
Jgger
lgver 2
1.00
1.00
decer
delor
0.75
1abox
0.75
0.50
0.50
Bater 21
0.25
0.25
0.00
123
0.00
0.6 Perturbation Intensity (o)
0.2
0.4
0.6
0.0
1.0
Porturbation Intensity
Figure 16. Visualization of cumulative sensitivity curve with varying number of levels L. We visualize the cumulative sensitivity curve Equation when computing for 5, 10, 15, 20 levels. find even when increase the number levels, curves remain approximately the same. Thus, we use 5 levels in our implementation to reduce compute for the sensitivity analysis step.
23


Reviews:
[{"review_id": "32hW2zNckk", "reviewer": "Reviewer_mfiq", "summary": "This paper proposes an adaptive, on-the-fly sensitivity analysis approach to design data augmentation for  increasing the robustness of the semantic segmentation models under naturally occurring corruptions. The proposed approach attempts to bridge the gap between choosing random augmentations like Trivial Aug and learned policies through RL. They do this by solving an optimization  problem on-the-fly. The problem is essentially posed as finding the right set of intensities /parameters of the chosen augmentations. The objective is set through the lens of the sensitivity analysis i.e, change in model accuracy with respect to change in intensities and this guides the augmentation. The change in intensities is captured through a Kernel Inception distance which measures the difference in Inception net features for the dataset due to one augmentation and the dataset with the other augmentation. By adaptively sampling from the intensity levels to which the model is most sensitive, the authors reduce overhead compared to prior sensitivity-analysis-based methods. They present extensive experiments on real-world driving datasets, as well as generic and domain-specific segmentation benchmarks  demonstrating notable improvements over several augmentation baselines. They also highlight the applicability  of their approach with foundation-model fine-tuning (e.g., DinoV2).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you! We appreciate your feedback, and address points below:\n\n- \"Could meta-learning offer a more direct optimization?\"\n\nYes.  NOTE: the difference is mostly wrt the choice between data augmentation vs meta-learning as the training approach, rather than an alternative for the sensitivity analysis (SA).\nIn MAML, we can interpret each parameterized augmentation type as a synthetic task. Then, during bilevel optimization, inner loop updates would optimize performance on specific augmentations (support) while the outer loop maintains overall performance on all aug’s (query).  Our SA modeling can still contribute additional info in this case when choosing the task data subsets; the way task data is selected or parameterized is largely open-ended in meta-learning.\n [1]  strengthens model robustness by adding a learned adversarial noise to query (outer loop) data.  The difference between this approach and our proposed technique is the paradigm in which sensitivity analysis is applied to add robustness to models (data augmentation sampling vs. noise sampling query attacks in meta-learning).  We may also add adversarial noises in our framework.\n\n- \"How to extend to less parameter-friendly corruptions?\"\n\nOne of our motivations is to utilize parameterized corruptions to improve generalization on natural corruptions that may not be easily parameterized. Prior art [1] has shown that many natural corruptions can be replicated with a composition of “basis augmentations”, from which this work is inspired by and generalized from.\nIn cases where we don’t have access to a parameter ‘alpha’, the problem becomes similar with AdvSteer [1], which samples sensitive augmentations from a fixed set of augmentation values (instead of a continuous range). However, this approach has increased computational complexity, since we need to test all values from a selected subset of augmentations.\n\n- \"How does it scale to domain-specific augmentations?\"\n\nIn InterAug’s case, the primary contribution appears to be a re-contextualization around the subjects in the image, s.t. spurious co-occurrences between subjects and background do not occur throughout training.   Our is a direct complement to the context area extraction. Instead of considering entire images in our SA computation, we can consider the context bounding box only. The two concepts applied together may produce context-specific sensitivities. As for other domains, the adaptation may be case-by-case. For example, lesion augmentation in medical applications involves synthetically increasing the diversity of lesion shapes, locations, intensities, and load distributions [2] – all of these can be considered as aug- types within our SA framework. \n\n- \"How might parameter-efficient approaches like LoRA mitigate overfitting to specific corruptions?\"\n\nTraining to increase robustness is often accompanied by degradation in clean accuracy, which may suggest either conflicting gradients or overfitting to corruptions. Using LoRA layers to mitigate this has been shown to work in AutoLoRA [3].   Since LoRA layers work very well when trained on small datasets, we may use our sensitivity analysis approach to select “uniformly difficult” augmentations for each class to generate the task dataset for LoRA training. Then, a routing approach similar to Polytropon [4] or MHR [5] can be used for inference on unseen data. This is an interesting extension of our  work and a valuable future direction.\n\n- \"A benchmark for efficient adaptation would be nice, especially with a small dataset.\"\n\nWe show results on fine-tuning for the ACDC Snow dataset in the bottom of the following spreadsheet: https://docs.google.com/spreadsheets/d/1KF0lY8iyv7Uo8K53EJmvfZqVqGVfKbuItcu-6iXXjNM/edit?usp=sharing \nand plan to include some more small dataset finetuning experiments in future revisions.\n\n- \"Relying on short adaptation intervals might be problematic given grokking is common.\"\n\nWe observe in practice that increasing the number of training iterations (thus increasing the number of iterations per interval, since the number of intervals is fixed) has very little effect on performance outcome. This may suggest that the current interval values may be sufficient for generalization to sensitivity curves within intervals. We can include analysis on this in future revisions.\nHow does this work perform relative to Segment Anything\nWe show downstream fine-tuning results and inference time/memory benchmarks on SegmentAnything in the same spreadsheet as above. We’ll also include these in the updated revision.\n\n- \"$a_{max} \\neq a_L$. $a_L$ is a level we solve for, but $a_{max}$ is the max intensity of the parameter range, which for our operations is 1.\"\n\n[1] https://proceedings.neurips.cc/paper/2020/file/cfee398643cbc3dc5eefc89334cacdc1-Paper.pdf \n\n[2] https://arxiv.org/abs/2308.09026\n\n[3] https://openreview.net/forum?id=09xFexjhqE \n\n[4] https://arxiv.org/abs/2202.13914 \n\n[5] https://arxiv.org/abs/2211.03831", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Not Applicable", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "It is related well with respect to the current literature on designing augmentation strategies for complex computer vision tasks that extend beyond classification", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The idea of focusing on where the model is most vulnerable (high-sensitivity intensities) is interesting and grounded.  Adapting this process for on-the-fly training is a good contribution. Furthermore, the paper’s evaluation is comprehensive and the results compelling both in terms of accuracy and efficiency \n\n\nWeaknesses and Questions\n\nThe approach adaptively selects corruptions, somewhat similar to meta-learning’s inner/outer loops (adaptation on a “task” and validation on a target). It would be helpful to discuss whether meta-learning techniques (e.g., using a gradient-based measure for how each corruption influences final performance) could offer a more direct optimization. Furthermore authors assume quite reasonably that they have an understanding of how the test environment behaves - a small discussion here could help in the rebuttal phase. \n\nNext, the authors examine a broad set of “basis” transformations (RGB, HSV, blur, etc.), there may be domain-specific or more complex corruptions  that aren’t captured. The paper could clarify how one might extend the method to less parameter-friendly corruptions.  \nAlso, while they consider generic augmentations, a lot of domain specific augmentations have been developed- for example this paper  (https://ieeexplore.ieee.org/document/10350672) designs a context-aware augmentation protocol for object detection, it is not clear how this approach will scale to those kind of augmentations. Also, it would be interesting to compare with  parameter-efficient or partial fine-tuning approaches (e.g., LoRA) that might mitigate overfitting to specific corruptions. The paper mentions partial or efficient adaptation in passing, but an actual baseline or experiment (especially for smaller datasets) would be better.\nFinally, given that deep networks can exhibit “grokking” or double-descent (e.g., https://arxiv.org/abs/2201.02177), the authors relying on short adaptation intervals for measuring sensitivity could be problematic. Could there be cases where the model’s sensitivity at a short training horizon is misleading at full convergence? I would like to hear authors’ thoughts on this. \nOfcourse, it was surprising that there was no discussion on segment anything which is now almost the defacto segmentation model. Can the authors comment on it too?", "other_comments_or_suggestions": "$\\alpha_{max}$ is not defined. Is it $\\alpha_{l}$?", "questions_for_authors": "Please see strengths and weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqs6UuW5X1", "reviewer": "Reviewer_SWwQ", "summary": "This paper introduces an adaptive, sensitivity-guided augmentation method to improve the robustness of image segmentation models against natural corruptions. The idea is to perform a lightweight, online sensitivity analysis during training to identify the most impactful perturbations. This approach aims to bridge the gap between the efficiency of random augmentation techniques and the effectiveness of policy-based augmentations guided by sensitivity analysis. The authors claim their sensitivity analysis runs significantly faster and requires less storage than previous methods, enabling practical online estimation during training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we are grateful to hear that you find our work impactful in real-world robotics applications and our analyses interesting for model sensitivity! We will be sure to add writing fixes in paper revision regarding Table 1 and increase visibility of results related to our claims, which many of were in the appendix.\n\nRegarding the question, “why is uniform augmentation of computed sensitivity analysis values (alpha) almost as good as beta-binomial sampling?”, we believe that this may be due to that the most optimal sampling in the basis augmentation spaces is already close to uniform. In future revisions, we will include an experiment with such a scenario to emphasize the advantage of not needing corruption gradients.", "claims_and_evidence": "- The paper claims 10x speed up but to be precise it's more like 9.3x according to Table 1, where it provides a runtime and storage comparison with AdvSteer (Shen et al., 2021). \n- The paper mentions in the introduction that the proposed approach is general and can be applied to other tasks, architectures, or domains, which are mainly shown in the appendix:\n\t- Different domains : medical domains in Appendix A.6\n\t- Different architectures: Appendix A.11\n\t- Different tasks: Classification, Appendix A.12", "methods_and_evaluation_criteria": "- The authors propose an adaptive sensitivity analysis method that iteratively approximates model sensitivity curves. They use Kernel Inception Distance (KID) to measure image degradation and define sensitivity as the ratio of change in model accuracy to change in KID. They optimize an objective function (Equation 4) to find optimal augmentation intensities. The method includes a training loop (Algorithm 1) that incorporates the sensitivity analysis.\n- The paper uses absolute pixel accuracy (aAcc), mean pixel accuracy (mAcc), and mean Intersection-over-Union (mIoU) to evaluate segmentation performance. They evaluate on real-world corrupted datasets (ACDC, IDD) and synthetic benchmarks (ImageNet-C, AdvSteer).\n- The use of KID for measuring image degradation is well-motivated, and the evaluation metrics are standard for segmentation tasks. The choice of datasets covers both real-world and synthetic corruptions.", "theoretical_claims": "- Not applicable as this is not a theory paper.", "experimental_designs_or_analyses": "- The paper contains several experiments to evaluate their method such as evaluation on real-world corruptions (ACDC), synthetic datasets (ADE20K, VOC2012, etc), ablation studies to analyze the contribution of different components of the proposed method.\n- The analysis of color channel sensitivity is an interesting addition that showcases the potential of sensitivity analysis for interpretability.", "relation_to_broader_scientific_literature": "- Addressing the problem of robustness against natural corruptions, which is a well-explored area in image classification, for semantic segmentation is an important real-world consideration such as self-driving cars etc. \n- Contrasting their adaptive sensitivity analysis with previous methods like AdvSteer (Shen et al., 2021), emphasizing the improvements in efficiency and practicality.\n- Building upon existing data augmentation techniques (e.g. AutoAugment, DeepAug) and highlighting their limitations.", "essential_references_not_discussed": "The related work section seems well-written.", "other_strengths_and_weaknesses": "- The proposed adaptive sensitivity-guided augmentation method is novel, and the 10x speedup boost compared to the previous sensitivity analysis is a strong point. \n- Improving the robustness of segmentation models is crucial for real-world applications, and the paper demonstrates significant improvements on challenging datasets.\n- The paper is well-written and easy to follow.", "other_comments_or_suggestions": "p8. sec5: \"Our model can complements\" -> \"Our model can complement\"", "questions_for_authors": "- What's your take on the fact that uniform augmentation of computed sensitivity analysis values (alpha) is almost as good as beta-binomial sampling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pYboN0Jjtk", "reviewer": "Reviewer_PSpB", "summary": "The paper addresses a practical challenge of enhancing model robustness to natural corruptions in semantic segmentation, a critical area for real-time perception applications. It proposes a novel, computationally efficient online adaptive sensitivity analysis approach (10x faster and 200x less storage than existing sensitivity analysis methods), facilitating practical deployment during training (lines 110-118).\n\nIt including real-world adverse conditions (Fog, Rain, Night, Snow) on the ACDC dataset (lines 275-288) and synthetic benchmarks such as ImageNet-C and AdvSteer across multiple datasets (6).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you!  We appreciate the suggestions and will improve clarity in the revision, adding more references and notation. We address points below: \n\n- “Efficiency improvement claims are missing benchmarks like inference runtime benchmarks and memory usage.”\n\nWe would like to clarify that our efficiency improvements are strictly compared to the previous works in sensitivity analysis like AdvSteer, as presented in Table 1. Previously, sensitivity analysis (SA) was NOT computationally feasible to compute during training; it required a huge amount of local storage and long computation times dependent on dataset sizes. Our work makes SA feasible for online training, and opens many possibilities of future work along this direction (see response above to mfiq). Our efficiency claims in the faster inference time or memory usage would not be a fair comparison against  other augmentation techniques. Thus, our significant efficiency claims are benchmarked purely for sensitivity analysis. \nIn terms of training complexity, our experiments re-compute SA only 4x during training, amounting to merely 38.4 more GPU minutes for one experiment. Dataset sizes do not influence the sensitivity analysis runtime due to the use of KID, whose data efficiency we verify against Frechet Inception Distance (FID) in Appendix Section A.15.  Otherwise, the total train time and memory of our adaptive SA method is comparable to randomized augmentation approaches.\n\n- “The results suggest that this approach is not SOTA, as shown in some results of Table 2 and 3.”\n\nWe respectfully disagree, and would like to clarify some interpretations of the results. In Table 2, our method occasionally performs worse on absolute accuracy (aAcc) metrics compared to other methods on rain and snow scenarios. As we mention in the corresponding Section 4.1 discussing the table results, higher numbers on aAcc but lower numbers on mIoU may be indicative of poor generalization to class imbalances, as aAcc measures the total # of correct pixels. In many of the Cityscapes data, there are disproportionately many pixels classified to “sky”. Our method achieves a higher mIoU than the other method (47.53 -> 49.36 compared to AugMix for Rain, and 45.35 -> 48.16 compared to IDBH for Snow). While more PIXELS are correctly classified, lower values on mIoU may suggest that underrepresented classes are poorly classified. We include BOTH metrics in table results to provide an interpretable metric (absolute pixel acc) as well as a class-balanced metric (mIoU) for a more complete picture comparison.\n\nAs for Table 3, while our method does not perform best on the AdvSteer benchmark compared to the next-best method, we note that it outperforms other methods in most other scenarios across multiple datasets. AdvSteer benchmark involves heavily altered synthetic data, as shown in the Appendix Section A.10; these alterations are not meant to reflect real-world corruptions, but should rather be interpreted as a synthetic limit test. In contrast, ImageNet-C benchmark reflects transformations meant to replicate real-world effects such as frost, snow, etc. We interpret the boost in performance of our results as contributing primarily to realistic corruptions, which aligns with our goals. Additionally, we include the results for AdvSteer for transparency.\n- “Could the authors clarify what distinct advantages their augmentation approach offers, particularly in practical deployment scenarios?”\n\nOur method estimates model sensitivity to various augmentation transformations and samples augmentations with uniform difficulty based on this sensitivity. Unlike randomized approaches (e.g., IDBH, TrivialAug, RandomAug, AugMix) that ignore model state or model-based methods (e.g., AutoAugment) requiring pre-trained policies, our approach is a middle ground by using a generic image classifier trained on ImageNet for KID computations. Although our method incurs additional computational cost compared to randomized approaches—adding ~38.4 GPU minutes to training (4 sensitivity evaluations at ~9.6 minutes each on an RTX A4000 GPU)—it scales efficiently across datasets since KID evaluation requires only a fixed number of samples.  Sensitivity analysis currently runs on a single GPU, further optimization and acceleration possible.  Also, our method affects only training time and has no impact on inference speed. Sensitivity analysis is also useful for interpretability of robustness, as shown in Figure 4.\n\nIn summary, our method 1) provides a middle ground between model-based and randomized augmentation, 2) adds a negligible amount of fixed overhead that is agnostic to dataset size due to use of KID, and 3) provides interpretability and explainability on failure cases for practical deployment.\n\nTo show inference comparison with competing methods, we include timing results on experiments with Segment Anything: \nhttps://docs.google.com/spreadsheets/d/1KF0lY8iyv7Uo8K53EJmvfZqVqGVfKbuItcu-6iXXjNM/edit?usp=sharing", "claims_and_evidence": "1) The authors claim significant efficiency improvements such as \"10x faster computation and 200x less storage\" compared to existing methods (abstract lines 010-014). However, these claims lack explicit evidence in the form of tables or figures within the main text. While Table 1 briefly touches upon this (compared with 1), crucial details like memory computation benchmarks or clear runtime comparisons during inference are missing, making these claims difficult to verify. for example. AutoAug (Cubuk et al., 2019), Basis Perturbations (Shen et al. 2021).\n\n2) The paper claims to achieve state-of-the-art performance; however, the reported results (Table 2, lines 275-288 and Table 3 lines 330-363) suggest otherwise. For example, according to Table 2, AugMix and IDBH demonstrate better performance under rainy or snowy weather conditions, highlighting the dependency on adaptive sensitivity. Specifically, the authors' augmentation approach is less than 1.6% compared to existing SOTA methods for rainy conditions. Even for foggy conditions, their improvement is only marginal (0.6%) over IDBH. Additionally, there is no comparison provided regarding computational savings in terms of time, speed, or memory cost. \n\n3) In Table 3, which compares results across a broader set of datasets (six different datasets), this limitation becomes more apparent. The authors' method achieves top performance only in basic augmentation scenarios, while in Clean, AdvSteer, and IN-C scenarios, TrivialAug and IDBH outperform the authors' method in 2/6, 4/6, and 5/6 datasets, respectively. Furthermore, in unseen scenarios, IDBH consistently leads performance metrics. This raises concerns about the actual benefits of including the proposed method. The authors should therefore clearly articulate the unique advantages or novel contributions of their augmentation approach.", "methods_and_evaluation_criteria": "The evaluation procedure lacks clarity and comprehensiveness, particularly concerning memory and runtime costs, as mentioned earlier. Furthermore, the experimental analyses provided (Figures 3 and Tables 2-4) are insufficient to justify claims such as \"10x faster\" and \"200x less storage,\"", "theoretical_claims": "1) The presentation and explanation of critical algorithmic details (Algorithm 1, lines 110-164) lack clarity. For instance, abbreviations such as \"pf\" and \"PDF\" appear without clear definitions or context within the main text. Terms like \"BetaBinom,\" \"Levels append,\" and \"Metrics append\" (lines 206, 179) are introduced without proper explanation or motivation. \n\n2) Additionally, equations (5) and (6) (lines 206-219) require clearer descriptions and justification to enhance reader understanding and reproducibility.", "experimental_designs_or_analyses": "1) Please consider to provide explicit computational and memory storage comparisons, including clear tables and figures, supporting the claims of \"10x faster\" and \"200x less storage\", not only to AdvSteer.\n\n2) Given the moderate to negligible improvement over existing methods like IDBH on certain benchmarks, could the authors clarify what distinct advantages their augmentation approach offers, particularly in practical deployment scenarios?\n\n3) Highly suggested to provide clearer explanations or intuitive descriptions for the choices of parameters and methods used in Algorithm 1, particularly \"BetaBinom,\" \"Levels append,\" and \"Metrics append\"?line 179, 206, Why Gmax=2?", "relation_to_broader_scientific_literature": "They author compaired Adverse Conditions Dataset with Correspondences (ACDC) (Sakaridis et al., 2021) with four weather scenarios: Fog, Rain, Night, Snow (Table 2). ADE20K (Zhou et al., 2019) etc ImageNet-C (IN-C) synthetic corruption benchmark (Hendrycks & Dietterich, 2019), AdvSteer synthetic augmentation benchmark (Shen et al., 2021).", "essential_references_not_discussed": "Several essential references in data augmentation and explainable AI are missing. Notably, key datasets or methods such as CVPR2020 Cityscapes, ICLR2022 Image-9, CVPR2024 XimageNet-12, BDD100K from Berkeley, Rain100L / Rain100H / Rain800 real rainy images for deraining and robustness studies, provide important contextual background, color channel and are highly relevant, have not been cited or discussed.", "other_strengths_and_weaknesses": "Please see the review feedback above mentioned.", "other_comments_or_suggestions": "Please see the review feedback above mentioned.", "questions_for_authors": "Please see the review feedback above mentioned.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2oh98L7PU", "reviewer": "Reviewer_kcJV", "summary": "This work proposes a sensitivity-guided method to improve model robustness against image corruptions. The sensitivity measure enables a selection of proper model-free augmentation policies. The experiments show that the method improves robustness of models on both real and synthetic datasets, compared to SOTA augmentation methods in image segmentation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!  We clarify some misunderstandings below.\n\n- \"Figure 1 not support equal spacing along function g, and there is no proof.\"\n\nBy equal spacing, we meant that given the set of α values that fulfills Q, the set of g(α_i) are at equal intervals along the y-axis of the function g. We have included a proof for equal spacing here: \nhttps://drive.google.com/file/d/1oOqCZXiSmLV4k4T5YyJqZ2NDmS91U2RE/view?usp=sharing \n\nWe will be sure to include this proof and clarification  in the final revision.\n\n- \"Authors explain that α1 < α2 < . . . < αL, without explaining why.\"\n\nThanks for pointing this out. The intuitive explanation behind solving for “adequate spacing” can be thought of as solving for “uniformly difficult augmentation levels” with respect to model sensitivity. For example, it could be that a model is robust to a wide range of intensity values for a particular perturbation, but robustness quickly degrades past a certain value. \n\n- \"Standard metrics like mAP should be used.\"\n\nWhile mAP is standard for object recognition and instance segmentation, it isn’t commonly used with semantic segmentation, which classifies by-pixel. \n\n- \"Sensitivity analysis only considers augmentation intensity, without the type.\"\n\nOur work considers only intra-augmentation sensitivity; currently, all augmentation types have an equal chance to be sampled at train-time, when models may be more sensitive to one type of augmentation over the other (e.g., geometric types and photometric types). Implementation wise, the change is simple: the augmentation distribution sampler can be modified to account for the absolute sensitivity of each augmentation all together.   Note:  we indeed considered such weighing  in practice, but this distribution may skew too heavily towards certain types of augmentations, resulting in unintended overfitting. \n\n- \"Experiments only use one fixed random seed. There should be multiple random seeds used.\"\n\nWe agree that multiple iterations of each experiment under different random seeds is important to validate that improvements in performance are not attributed to randomness. While we conduct all experiments under one fixed seed in this work, previous work (AdvSteer) has validated the consistency for sensitivity analysis results across multiple random seeds that our work has compared against. Additionally, we reduce the role of randomness in experiments by initializing ALL models to the same initialization weights. \n\n- \"Authors miss the augmentation operations that are applied to the frequency domain of images.\"\n\nThe mentioned works in frequency space augmentation primarily deal with classification tasks, while our work focuses on segmentation. Direct translation of these works to segmentation is nontrivial, as segmentation is likely much more dependent on high frequency details. This is apparent in that some works explore frequency-based domain adaptation specifically for this task (https://openreview.net/pdf?id=b7hmPlOqr8). However, this idea is very interesting and we believe it valuable to explore in future work! \nCurrently, the set of augmentation operations we use are consistent with other baselines we benchmark against. We choose to augment in the image space due to consistency with previous work, as well as interpretability/explainability and intuition w.r.t. sensitivity curves. Our framework can be directly applied in the frequency space, although the results in frequency space may not be a direct comparison anymore to the methods used in our current experiments. \n\n- \"The fine-tuning experiment results may be questionable since the baseline model may have already seen corrupted examples drawn from the same distribution as the evaluation set.\"\n\nWhile the foundation models we initialize weights with are based on much larger datasets, which may have seen adverse weather samples at some point in their training, we still observe improvements over baseline fine-tuning experiments when applying augmentation at fine-tuning. Considering that downstream fine-tuning does NOT involve adverse weather samples, we still find that improvement on generalization on natural corruptions to be valuable, since they were not involved in the fine-tuning process whatsoever.\nDid not disclose exact augmentation operations used, and whether they use the same set of operations in other methods for comparison \nIn Section 4 (Experiments), the “Experiment Setup” excerpt mentions that all methods use the same set of augmentation operations, with the exception of IDBH, which includes two additional augmentations.  As explained in the main text, hyperparameter tuning details can be found in Appendix Section A5, which describes all the augmentation operations.\n\n- \"Figure 4: unclear meaning of different lines in one subplot\"\n\nWe evaluate color channel sensitivity several times during training. This plot is meant to show how model sensitivity changes across color channels over the course of training.", "claims_and_evidence": "* LN191-192: The authors claimed that: The set of α values that fulfills Q are at equal intervals along the function g. However, there aren't any proof on why they should be equal spacing. And from Figure 1, it does not look like having equal spacing. \n\n* LN160-161: The authors claimed that: we seek to find a set of increasing, nontrivial augmentation intensities α1 < α2 < . . . < αL that maximize sensitivity. However, it comes abruptly that it is important to find out α1 - αL, without explaining why.\n\n* Same as the second point, the point of having 'adequate spacing' is not explained well (LN162-163 right)", "methods_and_evaluation_criteria": "* The metrics: standard metrics like mean average precision should be used. \n\n* The sensitivity analysis only considers the impact of augmentation intensity (strength), without considering the types of augmentation. \n\n* Confusing equation 4: it is unclear where the max-min is applied to. The authors claimed to maximize the minimum value, but it is still uncertain what is being optimized. The objective function should be better formulated.", "theoretical_claims": "N\\A", "experimental_designs_or_analyses": "* Experiments seem to use only one random seeds. There should be multiple random seeds used, and the results should provide the average and standard deviation. \n\n* Missing recent SOTAs: the authors miss the augmentation operations that are applied to the frequency domain of images, like AFA, VIPAug, HybridAug++, where the intensities of the operations also matter.  And the authors can also consider adding Fourier-basis functions as one of the augmentation options when carrying out sensitivity analysis. \n\n* Figure 4: unclear meaning of different lines in one subplot. The authors mentioned 'recency', but I cannot find the definition of recency in the paper. Also, the different lines in Lighter/Darker H have high variations. The authors only explained the changing tendency but not the high variations. \n\n* Concern regarding finetuning foundation model: the model was trained on numerous data, which means it might possibly have seen the corruption images with adverse weather effects. This makes the experiments less convincing, as: 1. the authors claimed that the proposed method is robust to unseen corruptions (LN061-062), 2. the authors did not disclose what exact augmentation operations are used, and also, whether they use the same set of operations in other SOTAs for comparison is unclear.", "relation_to_broader_scientific_literature": "The sensitivity-based augmentation technique is an interesting topic regarding efficient data augmentation. But the insights brought by this sensitivity analysis method are unclear,", "essential_references_not_discussed": "More recent SOTA augmentation techniques: \n\n* Hendrycks, et al., PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures, 2022.\n* Yucel, et al., HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness, 2023.\n* Wang, et al. Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification, 2024. \n* Lee, et al.,  Domain Generalization with Vital Phase Augmentation, 2024.", "other_strengths_and_weaknesses": "The contribution points are not strong, especially the third point which depicts mostly what has been done. There should be more emphasis on the benefits and insights brought by the proposed method.", "other_comments_or_suggestions": "* LN145-146: strange line space \n\n* Table 5: the subscript for Ours∼g and Ours∼p seems to be reversed\n\n* LN1128: 'Figure ??'\n\n* Table 1: too far from where it is referred", "questions_for_authors": "* The results of AutoAugment on ACDC (Fig. 3) are surprisingly low. Are the augmentation policy the default one, or computed on ACDC and IDD separately?\n\n* Is it a model-free augmentation policy or model-agnostic method? Since the sensitivity analysis is highly related to a model, it is hard to call it model-free.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received two weak accepts, an accept and a weak reject. Some of the important points raised by the reviewers prior to rebuttal stage were:

- scaling to domain-specific augmentations
- overstatement in case of reporting efficiency 
- methods lacks SOTA performance in all results
- possible scenario dependent performance
- concerns regarding fine-tuning foundation models

All reviewers acknowledged the rebuttal and several engaged in post-rebuttal discussion. Almost all of the major concerns were adequately addressed in the rebuttal. Some points such as non SOTA in a few cases and possibly scenario-dependent performance remained after post-rebuttal stage, but during discussion with reviewers, the consensus was that they are not of a major concern and should not lead to rejection. Also, reviewers liked the idea and found it sufficiently interesting. To this end, the decision is to recommend the acceptance of the paper.
Decision: Accept (poster)