Active Learning with Controllable Augmentation Induced Acquisition
Authors: Jianan Yang
Haobo Wang
Sai Wu
Gang Chen
Junbo Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: active learning
data augmentation
strength

Abstract: The mission of active learning is to iteratively identify the most informative data samples to annotate, and therefore to attain decent performance with much fewer samples. Despite the promise, the acquisition of informative unlabeled samples can be unreliable --- particularly during early cycles --- owning to limited data samples and sparse supervision. To tackle this, the data augmentation techniques seem straightforward yet promising to easily extend the exploration of the input space. In this work, we thoroughly study the coupling of data augmentation and active learning whereby we propose Controllable Augmentation ManiPulator for Active Learning. In contrast to the few prior work that touched on this line, CAMPAL emphasizes a tighten and better-controlled integration of data augmentation into the active learning framework, as in three folds: (i)-carefully designed data augmentation policies applied separately on labeled and unlabeled data pool in every cycle; (ii)-controlled and quantifiably optimizable augmentation strengths; (iii)-full but flexible coverage for most (if not all) active learning schemes. Through extensive empirical experiments, we bring the performance of active learning methods to a new level: an absolute performance boost of 16.99% on CIFAR-10 and 12.25% on SVHN with 1,000 annotated samples. Complementary to the empirical results, we further provide theoretical analysis and justification of CAMPAL.

Under review as a conference paper at ICLR 2023
ACTIVE LEARNING WITH CONTROLLABLE AUGMENTA- TION INDUCED ACQUISITION
Anonymous authors Paper under double-blind review
ABSTRACT
The mission of active learning iteratively identify the most informative data samples to annotate, and therefore to attain decent performance with much fewer samples. Despite the promise, the acquisition of informative unlabeled samples can be unreliable particularly during early cycles owning to limited data samples and sparse supervision. To tackle this, the data augmentation techniques seem straightforward yet promising to easily extend the exploration of the input space. In this work, we thoroughly study the coupling of data augmentation and active learning whereby we propose Controllable Augmentation ManiPulator for Active Learning. In contrast to the few prior work that touched on this line, CAMPAL emphasizes a tighten and better-controlled integration of data augmentation into the active learning framework, as in three folds: (i)-carefully designed data augmenta- tion policies applied separately on labeled and unlabeled data pool in every cycle; (ii--controlld and quantifiably optimizable augmentation strengths; (iii)-full but flexible coverage for most (if not all) active learning schemes. Through extensive empirical experiments, we bring the performance of active learning methods to a new level: an absolute performance boost of 16.99% on CIFAR-10 10 and 12.25 on SVHN with I, 1,000 annotated samples. Complementary to the empirical results, we further provide theoretical analysis and justification of CAMPAL.
INTRODUCTION
The acquisition of labeled data serves foundation for the remarkable successes of deep supervised learning over the last decade, which also incurs great monetary and time costs. Active learning (AI.)
is a pivotal learning paradigm that puts the data acquisition process into the loop of learning, locating
the most informative and valuable data samples for annotation (Settles 2009 Zhang et al. 2020 Kim
al. 202 la Wu et al. 202T). With much-lowered sample complexity but comparable performance compared to its supervised counterpart, active learning is widely used in real-world applications and ML productions (Bhatacharjee al. 2017] Feng et al.) [209 Hussein et al.) 2016). In spite of
its meritorious practicality, active learning often suffers from unreliable data acquisition, especially and
from the early stages. Notably, the models obtained around the early stages are generally undeveloped due to the insufficient data curated and sparse supervision signal being consumed. The subsequent cycle of the data query based on the model produced from the current cycle.
While this problem can probably be mitigated after adequate cycles are conducted, we argue that the problems the early stages of AL cannot be overlooked. Indeed, few works have resorted to data augmentation techniques to generate additional data examples for data distribution enrichment, e.g. GAN-based (Tran et al., 2019) and STN-based (Kim et al., (02Tb) methods. In this work, we attempt to take further step in investigating the role of data augmentation for AL.
To begin with, we provide a straightforward quantitative observation in Figure I The setup of these results is rather simple: we directly apply vanilla DA operations, such as flipping and rotation, to data samples and linearly increase the augmentation strengths. We may conclude from these scores follows. First, the simple augmentation (loosely) integrated into AL has led to surprisingly enhanced results, albeit their complicated designs. Secondly and perhaps more important, we have counterfactual observation that the same augmentation policy facilitated different data pools manifests notably different impacts. As shown in Figure when gradually stacking the augmentation operations, the labeled and unlabeled data pools achieves the best performance at different levels of

---

Under review as a conference paper at ICLR 2023
Augmented
Views
AACCiUnlabeled)
+2.32%
+3.03%
+3.60%
LACC(Labeled) +0.00%
+4.81
+8.36%
+6.47%
+6.03%
AACC(Both)
+4.00%
+4.87%
+10.49%
+10.67
+7.38%
Figure 1: A visualization for data augmentation their corresponding performance change as we stack aug- mentations over images when integrating them into active learning cycles. We test 3 cases where augmentations are applied to 1) unlabeled samples only; 2) labeled samples only; 3) Both. Details of experimental setups be found Appendix B.2
augmentation strengths. In hindsight, we hereby post our reasoning. To fully and tightly incorporate DA into AL schemes, the augmentation ought to serve different objectives on the labeled and unlabeled pools. In particular, the labeled pool favors label-preserving augmentation in order to obtain a strong and reliable classifier. By contrast, the unlabeled pool may require relatively more aggressive augmentation so as to maximally gauge the unexplored distribution. The phenomenon in preliminarily validates this reasoning. Noted, this counterfactual observation has not been studied or investigated by prior works (Tran et al. 2019 Gao et al. 2020 Kim et al. 202
Figure I
Motivated by it, we propose Controllable Augmentation Muniool for Active Learning. Core to method is purposely designed form of better controlled and tightened the integration of data augmentation into active learning. By proposing CAMPAL, we aim to fill this integration gap and unlock the full potential of data augmentation methods integrated into active learning schemes. In particular, CAMPAL integrates several mechanisms into the whole AL framework:
CAMPAL constructs separate augmentation flows distinctly on labeled and unlabeled data pools towards their own objectives;
CAMPAL.composes sfrength optimization procedure for the applied augmentation policies;
CAMPAL complies with most common active learning schemes, with carefully designed acquisition functions for both score- and representation-base based methods.
Besides the theoretical justification of CAMPAL offered in Section 4 we extensively conduct wide experiments and analyses on our approach. The empirical results of CAMPAL are stunning: a 16.99 % absolute improvement at .000-sample cycle and 13.34% lead with 2,000 samples on CIFAR-10 10, compared with previously best methods. Arguably, we may postulate that these significantly enhanced results may have the chance to greatly extend the boundary of active learning research.
METHODOLOGY
In this section, we deseribe CAMPAL detail. CAMPAL chiefly composed of two components. one hand, CAMPAL formulates decoupled optimization workflow to locate feasible augmentations being applied to labeled/unlabeled data pools with distinct optimization objectives. This optimization difference is eventually manifested by their augmentation strength (Section 2.2). On the other hand, augmentations with modified acquisition functions (Section 2.3). to be adaptable with most (if not all) active learning schemes. Hence we may posit that CAMPAL forms a much more tightened integration of DA and AI, due to not only its controllable mechanism both data pools but also its full adaptability for all common active learning schemes. The framework for CAMPAL is summarized in Figure
CAMPAL aggregates the information provided by properly-controlly
2.1 SETUP AND DEFINITIONS
Active Learning. The problem of active learning (AL) is defined with the following setup. Consider D C. [& as the underlying dataset consisting labeled data pool D, and an unlabeled data pool Dr. with |Dvr > [Dz]. Based on fully-trained classifier fa that assigns a label each data point, a data acquisition function heeg fo) De - R calculates the score for each data instance. We also use P(yla:; : fe) denote the probabilistic label distribution of given by fe. Then AL selects the most informative sample batch and updates the labeled set accordingly. the remainder this paper, omit parameter in hang when the reliance acquisitions classifiers clear.
2

---

Under review as a conference paper at ICLR 2023
Fahaured Acquistiin
Angmeatation Generater
Sampies
PL
T(pel
Ammotation Roques
Unlabeded
Strength Optimizer
Tixal
COO
Oracle
Angmentation
Trait
Eahanced
Classifier
Updinte
Figure 2: An active learning cycle for the CAMPAL framework. We optimize strengths su, St for the unlabeled/labeled pool separately, then generate the corresponding strength-guided augmented views. We train enhanced classifier over augmented labeled samples and induce enhanced acquisition with augmented unlabeled samples. Finally, the acquisition selects informative samples labeled by an oracle.
Data Augmentation. We denote single data augmentation (DA) operator by T(x) that transforms data point to another view via translation, rotation, other augmentation operations. Then we denote the set consisting of these augmentations by T. practice, several studies also provide extended augmentation operators consisting of multiple operators (Hendrycks et al. 2019 Xu et al. 2022). The number of operators in the composition is named the strength of T, then T denotes the augmentation set with strength 8. Intuitively, 8 also quantifies how far an augmentation drifts images away from their original counterparts. Given a data point I, we also use T(x) to denote all its augmented views. With the notations given above, we continue our controllable augmentation-induus acquisition in the following sections.
2.2 CONTROLLABLE AUGMENTATIONS ACTIVE LEARNING
As shown in Figure I_ data augmentation plays different roles in different data pools. On labeled data, it targets improve the model prediction performance such that requires the generated virtual examples to be label invariant. In contrast, unlabeled examples, it enlarges the exposed data distribution the pursuit of better acquisition distribution. In this section, we propose a principled framework that searches for feasible DA configurations for different data pools in their habits. It is worth noting that we adopt a dynamic control on the strength of augmentations across different cycles, making them adaptable to changes in AL the cycle proceeds. It can be empirically verified that such dynamic control is better than a fixed augmentation strategy (Section 3.33). Through appropriate strength control, expect to increase the quality of augmentations for AL.
natural
Strength for Unlabeled Data. The primary goal for augmenting unlabeled data is to offer precise informativeness evaluation with an enriched distribution induced accordingly, thus inducing more reliable acquisition. A problem with this is the invalid information introduced by potential unde- sirable augmentations. In detail, weak augmentations contain trivial augmentations that contribute little to the distribution enrichment, while drastic augmentations introduce excessive distribution drifts that mislead the acquisition. We resolve this problem by proposing proper strength that maximizes the overall informativeness of the augmented unlabeled pool. Specifically, maximize the least information augmentations can provide, ensuring that an optimized strength offers reliable informativeness. which can be formulated follows:
Su arg max Σ
min min((H(@v)) ăo E (ZU), fa(āv) = (xv))
(I)
wy EDu
where HH derive a proper g By adopting max-min optimization procedure, we can eliminate the potential negative impact brought by the corruption from aggressively augmented samples with min min(H(@v) ((@v))} and maximize the overall informativeness of the augmented unlabeled pool with arg max.
be an arbitrary informativeness metric, and we adopt entropy here is sufficient
Strength for Labeled Data. By involving augmentations in model training, we aim at obtaining a dependable model from limited labeled data and further enhancing the acquisition process. Different from augmentation for unlabeled data that maximize overall informativeness, augmentations for labeled data prone to training stability and convergence. To give out proper control over labeled
3

---

Under review as a conference paper at ICLR 2023
augmentations while avoiding extra training costs, introduce virtual loss term Ly and search the proper strength st for labeled samples by minimizing it:
Σ En.e.. Ly(m1,,),
81 arg min
[Dz]
where (,(r, [)(r) (PP(s) i; fo) 1 Toingle
A2
[(#),
IT (a:)
2ET (x)
where ((x), (xx) denotes the normal loss term and the augmented loss respectively, X1, A2 denotes fixed weights. For ainll-imaaee augmentations singtee we integrate the augmented information into the model by making them produce similar outputs, which the dissimilarity quantified with densen-Shannon (JS) divergence term, For Image Mixing T we just follow the setup of Mixup.
With the strengths Su. $t given above, we locate augmentations Tu that effectively enlarge the distribution, and the augmentations Ti that help deduce dependable classifiers. The combination of the two enables to enhance acquisitions by making classifiers and informativeness evaluations in the AL framework work collaboratively and efficiently. We will show how augmentations for unlabeled samples (UA) and labeled samples (LA) contribute the acquisition in Section 3.3
2.3 CONTROLLABIE AUGMENTATION
ACQUISITION FOR ACTIVE LEARNING
With the properly-controlled augmentations in Section 2.2 we proceed by providing a fast and efficient approach to integrate the functionalities of acquisitions and augmentations, i.e. controllable augmentation-ine induced acquisition. A key challenge for inducing the augmented acquisition hazy arises from the complicated forms for hioase which denotes basic acquisitions and varies across different studies. In this section, we highlight two types of acquisitions, i.e., score-based acquisition and representation-bait the corresponding augmented acquisition forms. Notably, CAMPAL can adopt various kinds of acquisitions and enhance them, see Section 3 Since training a classifier fo with augmentations is straightforward, focus formulating augmented acquisition with augmented unlabeled data.
acquisition. We treat these two types hhose differently and describe
Integrating Augmentations into Score-based Acquisitions. Score-based acquisition calculates an information score for each data point and selects samples with the highest score, like Max Entropy (Settles) 2009). We enhance them by aggregating the information provided by augmentations, which given by real value scores. Specifically, for methods that calculate acquisition score hease (a) for each sample z, we calculate an information score hbase (ō) for every augmented counterpart T(x) and aggregate them into one score. We propose several variants of heeg. including:
1. heeg (x) mingeT.te (a) hbose (ž) reduces potential redundant information with minimum acqui- sition score within the augmented batch;
heeg E EEET.()) Mbase (z) sums up all the informativeness provided by augmentations; 3. haeg (x) = Ez (x)) sim T (a))e(()) weights the informativeness of é by its similarity to its non-augmented counterpart, thus introducing the inter-sample information.
Integrating Augmentations into Representation-based Acquisitions. For representation-basee based acquisitions, hbase provides a feature vector embedded into a representation space and performs sampling according this space, like Core-set (Sener et al. 2018). Notice that pppresentation-basee methods rely on a distance function to measure the correlation between instances, we generalize the distance functions between individual samples to point-set distance functions between augmented sample batches. By adopting set distance functions, we enhance the acquisition process by taking the correlation across augmentations over different samples into consideration. To this end, we focus on well-defined set distance functions and propose the corresponding variants follows:
1. Standard distance: d(x, =) min | =
2. Chamfer distance: d(x, 2) = Exer, (x) millseT.(a) || =1 + ΣxeT, mingeT.(O)) ||==1|3
considers pairwise similarities for the augmented views from two samples;
3. Pompeiu-Hausdorff
d (z, Tx(2)) max d (T,()), highlights the maximal potential difference between two samples.
Hausdorff distance: d(æ,2) 2) =
4

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of the averaged test accuracy on benchmark datasets and different AI. strategies. Since CAMPAI has multiple versions, we choose the one with the hest performance and denote it with CAMPAL* The best performance in each category is indicated in boldface. Ny. denotes the number of labeled samples.
Dataset
Method
Nr. = 500
Ny. 1, 000
Ny. = 1, 500
N₁. = 2, 000
Random
52.42++2110
6.938+1.91
6..55=1.30
71.43
Entropy
55.86=1.66
66.42=2.49
73.08=2.84
75.40=2..4
BADGE
56.19+1.97
67.30+2.19
76.35++0.57
80.03++1.68
SVHN
BGADL
40.18=00.4
1..58=130
1..56+1.34
69.73
CAL
16.98:1.07
66.22:1092
72.09+1.83
75.22:42.11
LADA
16.61 66.56+1.21
7.66
75.84+1.12
CAMPAL*
61.34+4.26
78.81
0.42
85.66
Random
38.5442.28
49.77=3.08
58.61
21.49:+2.06
Entropy
39.80=1.60
55.43=1.71
60.76+22.64
1..95
BADGE
44.18:2.09
55.97
62.40
67.03
CIFAR-10
BGADL
37.54
4..57
51.81+1.000
56.73:L0.75
CAL
1.60
54.24+2.30
59,83=2.66
64.24
LADA
41.87+2.33
56.37=2.24
62.76-1..99
66.26
CAMPAL
2.02
73.36
77.87 +0661
00.3
Dor of Labeled Samsies
Namber of Labelee Samppes
DSC
Mamber of Libeled Samples
(a) SVHN
(b) CIFAR-10
(c) CIFAR-100
Figure 3: Test accuracy on number of labeled samples over different datasets.
Controllable DA-Driven Active Learning Cycles. With those nugmentation-induce acquisitions, we complete the active learning cycle within CAMPAL. First, we generate the labeled augmentations Ti with properly controlled strength 8t, then produce an augmented classifier fa trained over them. This makes up for the insufficient labeled information and further brings a reliable model. Second, we generate the unlabeled augmentations with an optimized strength sa and induce the enhanced acquisition hacy with Tu and fa. Notably, CAMPAL offers dynamic strength control on augmen- tations across cycles, which also leads controllable acquisition adapting itself the changing data pools. This augmentation-indum guarantees the positive impact of augmentations, which finally helps produce better querying results. As result, these two steps jointly ensure the quality of data to label at the end of the active learning cycle, largely boosting the performance. Our experiments in Section 3 show their separate effects as well as the combined impacts in detail. The pseudo-code of our algorithm is provided in Appendix C
acquisition step provides precise information evaluation and
3 EXPERIMENTS
3.1 BASELINES AND DATASETS
We instantiated our proposed CAMPAL with several existing strategics, including 1) Entropy, 2) Least Confidence (LC), 3) Margin, 4) Core-set (Sener et al. 2018), and 5) BADGE (Ash et al. 2020).
modes that integrate augmentations into an enhanced acquisition, including 1) MIN, 2) SUM, 3) DENSITY for Entropy, LC, Margin, and 1)
We also implement several aggmentation-aggeegaall
STANDARD, 2) CHAMFER, 3) HAUSDORFF for Core-set, BADGE, as shown Section and Table 2 In this section, specify the instantiated augmentation-aqquiiitoon with basic strategy hhase its subscript and the augmentation-ggregation mode its superscript, e.g. CAMPAI MIN
CAMPAL-Eatropy-
5

---

Under review as a conference paper at ICLR 2023
Table Performance CAMPAL with different hbese and aggregation modes. The experiment is conducted over CIFAR-10 with 2,000 labeled samples.
Method
Aggregation Modc
Type of haase
bhase
MIN
SUM
DENSITY
Entropy
76.90-10.76
75.87:10.32
78.89-10.74
Score
LC
7..82
72.74:11.31
76.76-10.54
Margin
78.70=0.58
71.33+00.66
79.16=10.48
Type of hoase
hbase
STANDARD
CHAMFER
HAUSDORFF
Core-set BADGE
78.20=0.28
79.67:0.68 80.37+0.86
78,4910.51 79.84+0.24
Representation
79.71
Table 3: Comparison CAMPAL with its non-augmented counterpart with different AI. strategies. indicates the performance boost brought by CAMPAL.
Dataset
Method
Entropy
LC
Margin
Coreset
BADGE
baseline
81.3310.86
81.15=1.16
80.71=1.11
83.36:10.82
82.89:10.95
Fashion
+CAMPAL
85.89+0.29
84.63++.31
84.82+0062
84.36++0.48
86,24+1.044
+4.56110.1
13.4811.75
+4.11=1.32
+1.00=10.95
+3.35=11411
baseline
75.40+22.43
76.39=1.330
76.32++1.77
77.81+0.93
80.03++1.66 1.68
SVHN
+CAMPAL
85.36:10.45
84.34-0.62
84.90-10.57
84.35110.81
85.66.10.99
+9.96=#2.47
1.49
+8.58
+6.54
+5.63
baseline
6..95
66,97:11877
66.76:11.77
66.90:10.93
67 67.03:10.62
CIFAR-10
+CAMPAL
78,89:0.74
06.62:00.2
79.16::0.48
79.6710.68
80,37
A
+12.94+1.5 1.55
+..95+1.97
+12.40+1883
+12.77+1.15
+13.34+1.00
baseline
45.18:10.13
45.70:10.18
45.64:10.25
46.52:1.21
47.75
CIFAR-100
+CAMPAL A
48.76+00.0
49.24+0.70
49.63+00.55
46.80+0.27
49.46=0065
+3.58.10.33
40.54
+3,99
+0.28.10.34
+1.71
We also denote the version with the best performance for CAMPAL, i.c. CAMPALBADGE as CAMPAL* as shown in Table We repeat every experiment times.
CHAMPER
In this work, we compare our method to 1) Random, 2) Coreset, 3) BADGE, 4) Max Entropy. 5) Least Confidence 6) Margin. We also compare our method with other active learning strategies with data augmentations, including 1) BGADL (Tran et al.) 2019), 2) CAL (Gao et al. 2020), and 3) LADA (Kim et al. 2021b in Table For a fair comparison, CAL does not use original semi-supervised setting but uses a supervised procedure. Since LADA has multiple versions, we choose the one with the best performance for comparison in Table We further prove the efficacy of CAMPAL by comparing its performance with the corresponding baseline versions in Table 3
3.2 MAIN EMPIRICAL RESULTS
CAMPAL achieves SOTA results. As shown in Table T and Figure 3 CAMPAL significantly outperforms their rivals on many datasets and data scales. Specifically, on the CIFAR- 10 dataset, we improve upon the best baseline by 8.08%. 16.99%, 15.11%. 13.34%, where the labeled has 500, 1000, 1500, 2000 instances respectively. Moreover, CAMPAL exhibits the most significant performance boost with a moderately small Nc. which is approximately around 1,000 for CIFAR-I0 and SVHN. Besides, we can see that different versions of CAMPAL consistently achieve superior results on CIFAR- 10, as shown in Table 2 As shown in Table 3 in all combinations of baselines and datasets, CAMPAL variations exhibit the best performance. Notably, CAMPAL also brings consistent performance boost with strong scalability.
In addition, it is worth noting that previous works (Tran et al. 2019 Kim et al. 2021a) are typically
evaluated with a large number labeled samples (e.g., 10% 40% labeled samples for CIFAR- 10). We also challenge this by querying fewer samples over benchmark datasets, shown in Table [ When Nz = 500 or 2,000 on CIFAR-10, recent augmentation-ben based AL strategies fail to outperform other simple baselines like BADGE. Notably, BGADL performs the worst, because of the inadequate training with insufficient instances in the current active learning setting. Since CAL is originally designed for semi-supervised setting, fails to outperform simple baselines like BADGE under

---

Under review as a conference paper at ICLR 2023
Table 4:
Test accuracy of CAMPAL when UA LA are individually applied over CIFAR-I0 with 2,000 labeled samples. The results produced over 5 different AL strategies.
Components
Entropy
LC
Margin
Core-set
BADGE
UA
LA
1..9
66.97.11.77
66.76.11.77
66.90.10.93
67.03
1.8
69.59-12..5
71.8611.116
68.83.11.29
71.24
0.99
05.9
77.73+10.44
77.73=0.200
78.89=0.22
76.70
76.62++0.62
78.7010.588
78.20=0028
79.71=0.51
1:4
1:8
10
150
2.52
0:30
274
6.93
6.70
774
740
6.54
9:7
5-14
614
12
27
9104
745
17
2:20
1-6
642
6:33
7:55
651
K16
5:37
(a) Entropy
(b) Least-Confidence
(c) Margin
Figure 5: A heatmap visualization performance boost brought by augmentations different strengths, when attached to the labeled pool and unlabeled pool. The experiments are performed over CIFAR- 10 with 2,000 labeled samples and are conducted over Entropy, LC, and Margin.
our supervised setting. LADA outperforms other baselines on CIFAR-10 but fails on SVHN since maximal-entroopy augmentations can easily change the semantics of the digit data. In contrast, proposed CAMPAL remains competitive, indicating its superiority.
The learnt augmentation strength differs for unlabeled/labeled data. In Figure 4 we visualize the dynamics of the learnt strength su: of across active learning cycles. In particular, we conduct MIN the experiment 5 times on CIFAR-10 with CAMPAL and STANDATD and figure out the average optimal strength CAMPAL CAMPAL-BAGE BADGE
Srredaaan Batiral
value. We can observe that the s", is generally larger comparison with of across the AL cycles. This verifies postulations that labeled requires moderate augmentation for label preserving. In contrast, unlabeled data prefers relatively stronger augmentations enrich the data distribution such that a wider range of informative regions can be explored. We conclude that AL is better enhanced by DA with a combinatorial scheme of weak and strong augmentations applied to labeled and unlabeled data in our framework, which also corroborates our theoretical findings in Section
Oyd:
Figure 4: The learned strength on CIFAR-IO, 10, with CAMPAL in- stantiated with Entropy/BADGE.
3.3 EMPIRICAL ANALYSIS
In this section, we present our ablation results to show the effectiveness of our framework. We exemplify the superiority of your CAMPAL by using scored-based acquisition with MIN as the aggregation; see Appendix for more ablation experiments.
Impact of Unlabeled/Labbled Augmentations. Here, we compare the performance boost of nugmentation-inducee acquisitions based on different AL strategies and the results are reported in Table 4 We can see that without augmented labeled information, the enhanced acquisition gives out consistent performance boost over several strategies, and the maximal boost is presented by Margin 10%. The enhanced training process also plays an important role promoting the performance of the existing strategies by 8.35% 11.86%. A combination of these two components also shows consistently best performance compared to other ablation versions, indicating that they work well with each other and unleashes different types of information. We conclude that both the augmented unlabeled information and the labeled ones help resolve the problem of unreliable judgment in AL strategies.
7

---

Under review as a conference paper at ICLR 2023
Compare with fixed augmentation strengths. Since we emphasize the importance of strength control over St, &u in Section 2.2] we will provide more details here. brief, augmentations with various strengths contribute to the performance but can be inefficient when strengths not chosen appropriately. To further look at the impact of augmentation sets with different strengths, we fix the value for 8t; Su and see how they decide the final performance. Specifically, we test different combinations of st and 8u in the range [0, 4], with other settings following the main empirical studies, The relative performance boost compared to their non-augmented counterparts is shown in Figure 5 MIN Margin
Without proper strength control, the performance boost can decrease. For instance, CAMPAL CAMPALMarggu with $3 3. Su = 1 leads to a 4.32% performance drop compared to the optimal one, when the worst MIN case in CAMPAL Entropy causes a 3.28% drop similarly. In addition, we can also see trend similar to Section 3.2 that the classifier fo prefers weakly labeled augmentations when stronger unlabeled augmentations induce stronger acquisitions, even without a dynamic strength control.
4 THEORETICAL ANALYSIS
In this section, we theoretically analyze why weak and strong augmentations being strategically applied labeled and unlabeled data exhibit the best performance when combining AL with DA. Following the previous sections, we use fo to denote the model fully trained over augmented labeled samples. When an unlabeled sample lies within the augmented region for a particular labeled sample, we can propagate the labeled information to the corresponding unlabeled samples. Formally, with feature map fg femb derived from fo we define a covering relation between augmented labeled batches and unlabeled samples as follows:
Definition 1. Given a collection of augmentations TT we say that an image x is covered by 24 with respect to the augmentation set T, if the feature embedding of x lies within the convex hull of the augmented views of z;: famb (x) E conv (feub (T(zi))). We denote the covering relation by x 4T;-
Without loss of generality, assume there are labeled samples.x1 æ1...., .......... LL. together with the unlabeled samples covered by its augmentations, constituting L components. For each component Ci(i 1,..., L). let P. be a probability that a data point sampled from the underlying data distribution covered by C₁2 To make the analysis tractable, assume the properly controlled augmentations for labeled samples, eliminating the potential overlaps across different components:
Assumption With moderately weak augmentations for labeled samples, C, 's do not overlap with each other. i.e. vi + j. P(C, n C + 0) 0.
With Assumption 2 the error for So can estimated by how these components cover the data space. To further illustrate this, we provide comparison between different augmentations in Figure 6 The following proposition characterizes the relationship between the error and the components.
Proposition 3. Let E denote the probability that the fo cannot infer the correct label of a test example. Then& E is upper bounded by
E E E. P.(1-P.)"
(1-E'
where denotes the number of samples that lie within the labeled components.
In Eq. B). the first term denotes the risk brought by ill-defined augmentations, while the second term denotes sub-sample empirical risk. With Eq. (3). we continue reduce the error much as possible by acquiring informative samples. By adding newly queried sample I1+1, the error reduction is estimated as follows:
AE PL+1) Σ P(( P((1-P.)* P.) (1- (1-(11 (1-1-p))) -PL+1 (1- - PL+1) (4)
where Am is the number of samples newly covered after labeling #1+1-
We take step further by illustrating two terms in Eq. The first term denotes the performance boost brought by better coverage with newly-annotated samples. Specifically, the samples that drift farthest from the existing components better cover the under-explored data space, indicating larger Am in turn the performance boost. This is also consistent with the max max-min optimization objective for unlabeled samples described in Eq. (T). with the intuition provided in Figure (d). The second
8

---

Under review as a conference paper at ICLR 2023
term characterizes the potential error induced from augmentations on unlabeled samples, i..., too strong augmentation excessively increases the value of PL+1, leading to its increase. Therefore, is important to locate moderately strong augmentations for unlabeled data in AL.
Theorem 4. With properly selected augmentation sets and sufficient large L, the maximal value for error reduction AE(Am. PL+1) with newly-annotated samples can be estimated as follows:
AE PL+1) E @
(5)
where U denotes the number of unlabeled samples, with K m. Llog( 4 +U) 1--1)
L-U
From the theorem, we can see that properly selected samples and augmentations give out signif- icant error reduction. Specifically, m/ denotes the average number of samples covered by each component, which indicates better coverage induced from properly controlled components when being larger. Revisiting our theoretical proof, we further explain that DA indeed serves different goals in AL.. On the one hand, the augmentations on labeled data guarantee that Assumption _ holds, indicating that we need a dependable model and weak augmentations. On the other hand, this theorem emphasizes the importance of acquiring newly-labeled samples guided by moderately strong augmentation, ensuring better coverage while also avoiding potential misleading information. With all the discussions above, auggentatioon-aqquiiiin augmentations, where better augmentations result in more dependable classifiers for AL and larger error reduction across AL cycles. This echoes our discussion of the benefit of appropriately controlled data augmentations for AL. A more detailed analysis is given in Appendix A
integration effectively relies on the quality of
5 RELATED WORKS
Data Augmentation is a technique that improves the generalization ability of models by increasing the number of images and their variants in a dataset (Xu et al. (002). The most commonly used augmentation techniques include geometric transformations (Shorten & Khoshgoftaar 2019). random
erasing (Devries& & Taylor 2017 Zhong et al., 2020) and generative adversarial networks (Zhu et al.
2018: Bowles et al. 2018). Another type of augmentation is image mixing (Zhang et al. 2018 Yun et al. 2019), which blends multiple images and their corresponding labels. Instead of designing new types of augmentations, recent studies also collect a group of augmentations and optimize their strength (a.k. strength) (Cubuk et al. (2020), which quantifies how far an augmented image drift from its original counterpart. By optimizing this strength, several studies attain state-of-the- performance over several benchmarks (Zheng al. 2022 Yang et al. 2022).
Active Learning is a machine learning paradigm in which a learning algorithm actively selects the data it wants to learn from the unlabeled data sources (Settles) 2009, Ren al.) (02T). The crucial part of active learning in most existing strategies is exactly the data acquisition process, which targets selecting the most informative examples. Current studies mostly focus on specifie parts of samples and can be roughly categorized as follows: (a) Encertainty-basel methods that prefer the hardest
samples (Choi al., 2021 Mai et al. 2022) or the ones the current silly-trained model uncertain about (Kirsch et al. 2019: Wang et al. 2022); (b) Representation--asee methods searching for the
samples that are the most representative of the underlying data distribution (Sener et al. 2018 Ash et al., 2020;; Kim & Shin 2022). To date, the unreliable informativeness evaluation with very few samples remains a critical issue for Active Learning.
CONCLUSIONS
In this work, we propose novel active learning framework CAMPAL. Based on the observation that the impacts of augmentations applied to the disparate data pools differ due to their different goals, CAMPAL conducts appropriate controls on data augmentation integrated into active learning. We empirically find CAMPAL attains state-of-thh-an performance with significant performance boost, especially with fewer labeled samples. Our theoretical analysis further guarantees this difference and claims the reliance of AL on the quality of introduced augmentations. In the future, we hope to generalize CAMPAL more tasks and investigate the impact of DA over AL in more detail.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL URL httpttttt.//peeeeeeeee.ee http://ewe...eeeoommmeaerr.....a
Sreyasee Das Bhattacharjee, Ashit Talukder, and Bala Venkatram Balantrapu. Active learning based news veracity detection with feature weighting and deep-shallow fusion. In 2017 IEEE International Conference on Big Data (Big Data), pp. 556-565. IEEE, 2017.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger N. Gunn, Alexander Hammers, David Alexander Dickie, Maria del C. Valdés Hernández, Joanna M. Wardlaw, and Daniel Rueckert. GAN augmentation: Augmenting training data using gencrative adversarial networks. CoRR, 11111810.10666 2018.
Razvan Caramalau, Binod Bhattarai, and Tac-Kyun Kim. Sequential graph convolutional network for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2021.
Jongwon Choi, Kwang Moo Yi, Jihoon Kim, Jinho Choo, Byoungjip Kim, Jinyeop Chang, Youngjune Gwon, and Hyung Jin Chang. Vab-al: Incorporating class imbalance and difficulty with variational bayes for active learning. In Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6749-6758, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks wttp://////aa/....o//////
with cutout. CoRR, hts//708.04530 2017. URL
Di Feng, Xiao Wei, Lars Rosenbaum, Atsuto Maki, and Klaus Dietmayer. Deep active learning for efficient training of a lidar 3d object detector. In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 667-674. IEEE, 2019.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö Ank, Larry Davis, and Tomas Pfister. Consistency- based semi-supervised active learning: Towards minimizing labeling cost. In European Conference on Computer Vision, pp. 510-526. Springer, 2020.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1912v22 2019.
Ahmed Hussein, Mohamed Medhat Gaber, and Eyad Flyan. Deep active learning for autonomous navigation. In International Conference on Engineering Applications of Neural Networks, pp- 3-17. Springer, 2016.
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8166-8175, 2021a.
Yeachan Kim and Bonggun Shin. In defense of core-set: density-aware core-set selection for active learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 804-812, 2022.
Yoon- SeonY Kim, Kyungwoo Song. JoonHo Jang, and I1-Chul Moon. Lada: Look-ahead data acquisition via augmentation for deep active learning. Advances in Neural Information Processing Systems, 34:22919-22930, 2021b.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32:7026-7037, 2019.
10

---

Under review as a conference paper at ICLR 2023
Xiaoyi Mai, Salman Avestimehr, Antonio Ortega, and Mahdi Soltanolkotabi. On the effectiveness of active learning by uncertainty sampling in classification of high-dimg gigh-dimensional gaussian mixture data. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4238-4242. IEEE, 2022.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
Silvio Savarese Sener al. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver. BC, Canada, April 30 May 3. 2018, Conference Track Proceedings. OpenReview.nntt 2018. URL https: / //operrvvi ew net. fort//frum? iidinidd=lla aTuk- -u--uw
Burr Settles, Active learning literature survey. 2009.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5972-5981, 2019.
Toan Tran, Thanh-Toan Do, Ian Reid, and Gustavo Carneiro. Bayesian generative active deep learning. In International Conference on Machine Learning, pp. 6295-6304. PMLR, 2019.
Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Cheng- Zhong Xu, and Min Xu. Boosting active learning via improving test performance. Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp.
2022.
Tsung-Har Han Wu, Yueh-Cheng Liu, Yu-Kai Huang, Hsin- Ying Lee, Hung-Ting Su, Ping- Inn-Chia Huang, and Winston H Hsu. Redal: Region-based and diversity- active learning for point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15519915519, 2021.
Mingle Xu, Sook Yoon, Alvaro Fuentes, and Dong Sun Park. A comprehensive survey of image augmentation techniques for deep learning. arXiv preprint arXiv:2205.01491, 2022.
Zihan Yang, Richard O Sinnott, James Bailey, and Qiuhong Ke. A survey of automated data augmenta- tion algorithms for deep learning-base based image classication tasks. arXiv preprint arXiv:2206.06:: 06544, 2022.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 6022 6031. IEEE, 2019. doi: 10. 109/TCCVV2019.00611
URL https: /tops://dos //oii www.ooggg//.. 1w1.1////...21....00
Beichen Zhang, Liang Li, Shijie Yang, Shuhui Wang, Zheng-Jun Zha, and Qingming Huang. State- relabeling adversarial active learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 8756-8765, 2020.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.nes, 2018. URL
net. wwtp://eeetoootdodeeeeeeo.m.
Rb
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep autoaugment. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRe- view.net, 2022. htl.httts:////peeee
net wtttt//frrmmiiimmm -ts-:==-7t----------0- 53J9ZARF

---

Under review as a conference paper at ICLR 2023
Zhun Zhong, Liang Zheng, Guoliang Kang. Shaozi Li, and Yi Yang. Random erasing data
augmentation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020,
The Thirty-Secome Innovative Applications of Artificial Intelligence Conference, IAAI 2020,
The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020,
New York NY, USA, February 7-12, 2020, pp. 13001-13008. AAAI Press, 2020. URL
https: //tp://///aa...///////////////// http////s//nnxxxx//// index.nen.
Xinyue Zhu, Yifan Liu, Jiahong Li, Tao Wan, and Zengchang Qin. Emotion classification with data augmentation using generative adversarial networks. In Pacific-Asia conference on knowledge discovery and data mining, pp. 349-360. Springer, 2018.
12

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 6: The coverage on the data space presented by augmentations, where colored circles are labeled samples, white circles are unlaheled samples and the colored shade denotes the region covered by corresponding augmentations. double circle denotes the unlaheled sample he annotated. The figures ahove show: a) Proper augmentation for laheled samples; b) Drastic augmentation laheled samples; c) Suh-optimal unlaheled sample with the corresponding augmentation; d) A proper unlabeled sample with the corresponding augmentation.
A THEORETICAL ANALYSIS
This section provides complete derivation for the analysis given in Section I An intuition for this is given in Figure 6 Before the actual acquisition process, we must ensure convergence for the underlying classifier. Specifically, with proper augmentations over labeled data and the approximate loss term in equation
can deduce the upper bound for Pr(A) and guarantee the convergence
for training, shown as follows:
Theorem 1. Under the setting for CAMPAL, Let & denote the probability that the classifier cannot infer the label of newly given samples drawn from the underlying data space, with L labeled samples given D and augmentation set T. Then & upper bounded by E as follows:
S - P.(-- -
(6)
With properly selected augmentation set T and sufficient large L, & can be estimated by O with OLL/e) samples covered by labeled components,
m & 0(s) E
0
Proof. With proper control over augmentations, we assume that each component does not overlaps with at most one other component in Proposition 3 which can be controlled with appropriate augmentations, and generalizable multiple components. Let be the sampled example, the probability of not covered only in one of Cs's
& = P3i CCNC)) + P (x'ix is uncovered)
Σ P.)
Σ 11
With sufficiently large L, we can also have component set that covers the entire dataset, leading to Σ, P, 1. Now remains find the maximum value of Σ/= P.(1 P.) to bound the error term, with the following optimization objective:
min P((--P))" s.t. P.
With the KKT condition, we attain its maximum value when all P, is set to i.e.e Eg (1 1m With O(L/e) and sufficiently large L, we have
3 x exp (-T7 exp (-o(3) < (()e).
13

---

Under review as a conference paper at ICLR 2023
With the conditions in theorem reduction on upper bound E:
1 remains to consider the approximate boost provided by the
(Am, PL+1) = EDL U fIL+1), T3) -.
E P. - P.) (1- (1) PL+11 (1 (1- PL+1) P mm)
where Am is the number of samples newly covered after labeling 11+1
Theorem 2. With the conditions given in Theorem [ the maximal value for error bound reduction
(Am,m PL41) with newly-annottte samples can be estimated as follows:
(E PL+1) v (1- Ke-
(1- keemm/))..
(8)
where U denotes the number of unlabeled samples, with
K m log(I U+) (log 1)
L+U
Proof. Under this setting, Pt+1 appears to be proportional to Am, when no unnecessary overlap appears across components (guaranteed by Theorem [ Therefore, we can estimate PL+1 s Am/ (L+ where U denotes the number of unlabeled samples. With those conditions, we estimate the relative eтror reduction as follows:
E P1(1 pm) 1 (-(1-P.) P.) PL+1 (1 (1- (1-(1-LLt)) - m+mm
E P,(1- - (1)--))
(1-E P.
Am
Am +
(1-
1- 11(11-
Am
m++ Am
2
1
exp
L+U
Since m is large with sufficient labeled samples, we can further estimate this term as:
(1-
Am exp( (xsp(-T,1 I-+U
Then the maximum value for this is attained when Am reaches
1
m
log (-log 1 (1-1) ))
Am"
log
log (1 - ))
T
T++U)
L
m
log
1
+ loog
+ (log (L U) log L)
L
L L+U
Then
AE c
1
1
(-7 m
2
+
L ++U
log(1 - +
Am"
exp
m (log( (L +U) log
m71
exp
L+U
B ADDITIONAL EXPERIMENTAL SETUPS AND RESULTS
B. IMPLEMENTATION DETAILS
We conduct experiments on four benchmark datasets: FashionMNIST, SVHN, CIFAR-10, and CIFAR-100. We will construct random initial dataset with 100 instances for FashionMNIST,
14

---

Under review as a conference paper at ICLR 2023
Table 5: The list of all the augmentations used in the experiments. The letter x. denotes given images. a( b) denotes a continuous uniform distribution at interval (a, b). when B(a, b) denotes a beta distribution with parameters and b.
Augmentation AutoContrast(rr)
Parameters
Description
Maximizing the (normalize) image contrast
Brightness(z.e))
U ~ (111, 1.18): an enhancing factor
Enhancing the brightness of a given image
Color(r,,))
(11, 1.18): an adjustment factor
Adjust the color balance of given image
Contrast(z,0)
2/ [1, 1.18):
Enhancing the contrast of given image
CutOut(r,,))
< (4.099 0.1 CutOut ratio
Cut out a part of image and fill with black
CutOutAbs(22.0)
(0.09, 0.11): CutOut ratio
Cut part of image and fill with gray
Equalize(ar)
Equalize the image histogram
Identity(z)
Return the image itself
Invert(1r)
Invert all pixel values
Postcrizc(z,0)
e~ ((66660))) Posterizing degree
Postcrizing the image
Rotate(r,.))
-~~ (,2,4(2,,00) 30): Rotation degree
Rotating the image
Sharpness(z..))
((1, Sharpen degree
Sharpen the image
ShearX(z,v)
8 (0.115 O. 18): Affinity degree
Affine transformation in x-axis
ShcarY(2z.)
~~ (2(111 0.18); Affinity degree
Affine transtormation in y-axis
Solarize(1,00)
~~ (996, 128): Solarization degree
Solarizing the image
SolarizeAAd (z,2)
~~ ((00 60); Solarization degree
Solarizing the image and add back
TranslateX(z,00)
24(0.1, 0.15): translation ratio
Translating the image in x-axis Translating the image in y-axis Mix up the two given images
Translate ((z,))
V (.10,, translation ratio
MixUp(ur.rr.. A)
A: the mixing ratio
SVHN, and CIFAR-10, and 1,000 instances for CIFAR-100. Then we acquire 100 instances for FashionMNIST, SVHN, and CIFAR-10, and 500 instances for CIFAR-100 at each cycle. We repeat the cycle 20 times. Then we generate 10 single-image augmentations and mix-up augmentations for each sample. We normalize the images with the channel mean and standard deviation all the datasets. For CIFAR-I0 and CIFAR-100, we apply standard augmentation after conducting augmentations in the pipeline. We adopt ResNet-18 as the architecture and train the model for 300 epochs with an SGD optimizer of learning rate 0.01, momentum 0.9, and weight decay 5e-4. For the virtual loss term in equation? 2
we also set A1 X2 1.
B.2 IMPLEMENTATION DETAILS FOR THE SIMPLE APPLICATION OF DA FOR AL IN FIGURE
We integrate DA into AL. with fixed augmentations T as follows. This experiment is also conducted on dataset CIFAR- 10 with a ResNet-18 18 architecture. The basic acquisition here is Max Entropy. First, we augment the labeled pool with T. and train the classifier fo accordingly. Then we augment the unlabeled pool with T and performs acquisitions directly on the augmented unlabeled pool. Other settings are the same as the main empirical experiments.
B.3 AUGMENTATIONS INCLUDED
The details of the 19 augmentations in the (CAMPAL) with their parameters are shown in Table ()) In brief, the augmentations we use can be categorized into single-image augmentations and image-mixing. Formally, we provide an augmentation functional set that covers (i)-singular input augmentation means such as rotation for low-level image processing. The corresponding functional set is denoted by Tsingle = (w; })} where w points to an instantiated augmentation function. The sample x is taken as an input to w together with varying augmentation hyper- parameters X, such as the angle in the image rotation function. Similarly, we also construct a combinatorial augmentation functional set, Tmiz {7(2, x²; })). where the augmentation function 7 takes two input samples and x' together with apeee--rrmmter. With slight abuse of notations, we uniformly use À to refer to augmentation-rellen
In the implementation of CAMPAL, we simply adopt Mix for combinatorial augmentation. As we can see, upon fixed input, both singular and combinatorial augmentation functional sets be arbitrarily expanded, by varying À in continuous scalar space.
tiypper-prrameeee
15

---

Under review as a conference paper at ICLR 2023
Table 6: Test accuracy of CAMPAL and augmentation-induued acquisition with learned RandAug- ment.
Method
Fashion
SVHN
CIFAR-10
CIFAR-100
Ent w. RA
16.15
82.8411.12
76.83110.2
46.70:10.34
CAMPAL DENSITY
-attoop 'Entropy
16.17-0.58
83.49+00.96
78.89+0.774
48.76++0.30
Table 7: Comparison of the averaged accuracy when each type of augmentation is separately integrated into CAMPAL. We ran each experiment on CIFAR-10 10 with 2,000 samples annotated at the last cycle, and repeat them 5 times. .Vz denotes the number of labeled samples.
Augmentation
Nr. 500
Nr 1, 000
N₁ 1, 500
Nr. 2, 000
None
39.80.11600
55.43:1..71
60.76:12.64
65.95 1.36
AutoContrast
48.59:11.12
63.35:0.30
72.77+1011
76.36-0.28
Brightness
45.63:10.26
60.1612.25
69.50=0.22
74.25=0.16
Color
50.8413.50
6..04=1.75
72.2440.25
76.64=0.46
Contrast
49.77=3.911
56.5221.32
68.95-1..61
74.47=0.63
CutOut
47.91+2.66
62.29+3.69
71.37+1.19
76.62-0.34
CutOutAbs
53.59+0.30
63.9610.57
67.94+0.19
71.94-0.18
Equalize
19.38
13.1811.86
69.86110.78
74.25-0.27
Invert
51.73:10.29
63.65:0.12
71.89+10.33
75.98-0.28
Posterize
19.02=1.16
64.24+1.30
72.25+0.90
75.62-0.64
Rotate
44.60+1.39
56.4710.21
63.08
67.60-0.45
Sharpness
47.3142.35
62.40+1.15
70.98+1.226
74.38-2.46
ShearX
45.1944..9
58.22+0.99
6..96+0.55
72.16-1.20
ShearY
44.99
62.13+1.98
70.98+00.55
76.75-0.96
Solarize
18.5
63.58
70.1410.73
73.94-0.02
SolarizeAdd
52.48.22.06
64.9510.34
69.1740.20
71.91=0.49
Translate
11.41
54.97
64.88
70.03=0.68
Translate
55.12+1.12
68.2340.53
73.44+0.02
76.98=0.48
B.4 ADDITIONAL RESULTS COMPARED TO RANDAUGMENT.
Since CAMPAL locate feasible augmentations guided by their strength, we also compare CAMPAL with RandAugment (Cubuk et al. 2020). To show the effectiveness of separate control on unla- beled/labeled data CAMPAL, we trained RandAugment on the labeled data within each AL cycle, then applied the optimized augmentation to both the labeled pool and unlabeled pool. As shown in T CAMPAL shows better performance than the RandAugment, indicating the superiority of the separate control. It should be noted that RandAugment is originally designed for training over full labeled data, but is obliged to be conducted over the labeled pool with limited samples under the AL setting. Therefore, directly adopting RandAugment AL is infeasible, since it can be heavily biased towards limited labeled data, contributing little to the distribution enrichment on unlabeled data.
B.5 ABLATION STUDIES OVER TYPES AUGMENTATIONS
The impact of each single-image augmentation operator on CAMPAL. To further dive into the impact of the contribution of augmentations, we also provide the results when each augmentation separately applied to CAMPAL with different strengths, shown in Table 7 on CIFAR-10 10 with CAMPAL
DENSITY We can see the impact of different types of single-image augmentations varies. An interesting observation is that different augmentation operator does not contribute equally at the different AL cycles. For example, Sharpness performs better than Rotate when Nr = 500, but underperforms Rotate when Nr. 2000. It reveals a sophisticated mechanism of the benefit of these augmentation operators on AL. However, the profound theory behind why data augmentation works have not been fully revealed to date, making it difficult to principally pick up the best optimal augmentation type. Hence, we naively adopt a simple strategy that uniformly selects and stacks these operators enjoy their mixed benefits to AI..
16

---

Under review as a conference paper at ICLR 2023
Table 8: Test accuracy of CAMPAL when integrated with different combinations of single-image augmentations and the MixUp.
Augmentations
Entropy
L.
Margin
Core-set
BADGE
Single
MixUp
7..87+0.32
76.04+10.41
77.61+0.91
76.62-+0.50
77.78+0.45
/
75.41
7..89+0.64
77.76+0.48
75.21+0.32
7..69+0.62
76.90110.76
76.8210.62
78.70
78.20:10.28
79.71
Table 9: Test accuracy of CAMPAL when integrated with different combinations of single-image augmentations and the MixUp.
Coefficients
Entropy
LC
Margin
Core-set
BADGE
A1
Ag
1.0
7.32
23.26
70.15-2.19
76.62:10.50
07.78:
1.0
0.5
75.62=0.76
75.8
77.20=0.52
79.23=0.51
77.07
0
1.0
75.41=0.25
72.9310.65
67.69=0.72
73.99=0.03
75.69=0.62
0.5
1.0
75.95 0.62
7..55+0062
76.25=0.60
7..21+0.32
0..8
1.0
1.0
7.79
76.82+0.62
78.70=0.58
78.2040.28
79.71
Effect of single-image augmentations and mix-up. To prove the efficacy of including both single- image augmentations image-mixing into one query batch, we further explore the effect these two kinds of augmentations separately. To verify this, conduct experiments over two variants of CAMPAL that only use one type of augmentations, i.e. single-image augmentations and MixUp. The tests performed by the ResNet- 18 model with 4% (2000) data from CIFAR-10. For fairness, when only one kind augmentation is used, we generate 15 augmented samples of this type. In Table 8 consistent performance boost when using both kinds of augmentations over Entropy (A 1.03), LC (A 0.78), Margin (A 0.94), Coreset (A 1.58), and BADGE (A 1.93). conclusion, an integration of both single-image augmentations and image-mixing better unleashes the potential information of each sample than they separately do.
Effect of X1, X2 in the virtual loss term To optimize my i.c the strength for augmentations per- formed over labeled samples, we use Å₁ A2 to trade off the impact of single-image augmentations and image mixing. We dive deeper into this scheme by applying different combinations λ₁ Az shown MIN STANDARD
in Table 9 Specifically, the experiment is conducted on the following versions: 1) CAMPAL CAMPALEnrogy' Entropy: MIN MIN
2) CAMPAL
MIN, 3) CAMPAL Margin' 4) CAMPALSTANE STANDARD, 5) CAMPAL BADGE
Coreset
B.6 FURTHER EXTENSION: AUGMENTATIONS vs. UNSUPERVISED TRAINING
Recall that several studies tried involve unlabeled samples in training auxiliary networks assist querying(Sini) et al. 2019) Zhang et al. 2020 Kim et al.) 202 la Caramalau et al.) 2021), which inevitably brings high computational costs. We claim that data augmentations are sufficient to enforce the acquisition process without much extra cost over unsupervised training. To verify this, we compare the running time and performance of augmentation- based strategies and those utilizing extra unsupervised architectures, shown in Table TO We can see that augmentation-bon based methods with the best performance consistently outperform other strategies when becoming computationally efficient. Since active learning usually faces the problem of heavy computational cost in acquisitions, data augmentation may serve as an effective tool for both boosting the speed and performance at once. More importantly, this thought restricts the training process merely over labeled data, thus reducing the need for numerous unlabeled data in AL and making AL paradigms more applicable. We also adopt augmentations for labeled samples for methods with unsupervised representations.
17

---

Method Under review as a conference paper at ICLR 2023
Table 10: Comparison of the averaged test accuracy and the run-time of single AL. cycle over CIFAR-10. The run-time is calculated the ratio to Random Sampling. Bold indicates the best performance of different data scales within each category.
Nr 500
N₁. 1,000
N7. 1, 500
N₁ 2, 000
Time
Random
38.5412.22
49.77=3.08
58.61
61.4942.06
Entropy
39.8011.60
55.43=1.71
60.76112.44
65.95+1.36
1.03
L.C
38.50++1.10
53.83++2.71
59.74+2.12
66.97+1187
1.01
Margin
40.03:12.49
54.22:12.47
62.6111.91
66.7611.77
1.07
Core-sel
43.42442.09
53.54+-2.74
62.00++1444
66.90++0.93
1.33
BADGE
44.18+22.09
55.97+11557
62.40+2.155
67.03++0.62
1.28
TA-VAAL
61.72110447
66.67:0.92
70.53:10.50
74.4110.70
5.82
Unsupervised Representation
SRAAL
60.53+0.89
67.08=0028
71.02+0.66
75.05++0.15
6.04
CoreGCN
56.0311.73
59.81
65.19
69.61+2.33
2.73
Entropy
62.7811133
69.3411.35
7.8811135
76.90:10.76
5.13
LC
61.89+00.0
69.06=1100
73.49+0.92
76.82+00.2
5.08
CAMPAL-based Augmentation
Margin
65,4611.63
72.77:0.555
75.96
78.70:10.58
5.10
Core-set
62.59+0.89
71.55+0.29
7..69+0062
78.20++0.28
5.42
BADGE
66.40=1.001
73.48=0.42
77.38+0.53
79.71+0.51
5.54
B.7 ADDTIONAL RESULTS FOR ABLATION STUDIES OVER STRENGTHS
Heatmap visualization for Core-set and BADGE. We also provide different combinations of strengths on CAMPAL STANDARD and CAMPAL STANDARD which have similar phenomena as BBDGEE
CAMPALBADGE
Coreset
described in Section 3.3 and shown Figure 7 Specifically, augmentations over unlabeled samples tend to produce better results as strength increases and produce the best results mostly with strength indicating that drastic augmentations help induce a stronger acquisition. In contrast, augmentations over labeled samples produce the best results mostly with strength of 2, fitting with the conclusion Section 3.2 that weak augmentations for labeled samples are better at boosting the classifier. These phenomena show the difference between the impacts of augmentations over training and acquisition in active learning. and further guarantee the importance of a combinatorial scheme of weak and strong augmentations being strategically applied to labeled and unlabeled data.
10
0.00
-1.11
-1
0.60
1.02
0.00
3.40
5.24
3:43
4.1
10
8.86
9.84
9:91
384
10 10:24
10:05
11 11.44
11.87
12:53
12.06
8.82
10.34
10.06
10.21
10. 38
88
12
12:18
12 26
12.68
0
972
10
11
12
44
2
12
8.09
0
Sy
Su
(a) Core-sct
(b) BADGE
Figure 7: heatmap visualization of performance boost brought by augmentations of different strengths, when attached the labeled pool and unlabeled pool. The experiments performed over CIFAR-10 10 with 2,000 labeled samples and are conducted over Core-set and BADGE.
Strength Change Across Different Cycles. Since show the strength CIFAR-10 change MIN across AL cycles for CAMPAL Entropy and CAMPAL STANDARD AAMPALBADGE
in the main content, also show other versions strength change on CIFAR-10 in Figure 8 We can also see that larger m,, 's in comparison with m4's. We each experiment with 100 cycles and repeat them times. In more
18

---

Under review as a conference paper at ICLR 2023
detail, the difference between Mu, my is not clear in the early stages of active learning, since the model lacks the ability to uncover information from images.
C
PSEUDO CODE
We summarize the pseudo-code our CAMPAL within one active learning cycle Algorithm
Algorithm 1:
An active learning cycle for CAMPAL.
Require: Labeled data pool Lz, Unlabeled data pool Dy, Model fo-
4 arg ming [P IxED, L ((f((x))))
my = arg m [oll LXLED, Lf(at. m.). where L, is shown in equation
Generate augmentation set T with strength mi;
arg ming (D()) Lzer(mi (ff(z)).);
my arg max, Exy EDg min(H((o)) [Tu E TT(((aU) fo(āv) fo(ov));
Generate an augmentation set (mmm) with strength mau
Deduce the enhanced acquisition hacg with (mm)) and shown in Section 2.3
Select optimal sample batch Q according to harg'
DU 4- Dv Q: Dr. < D,, U
19

---

Under review as a conference paper at ICLR 2023
Att of AM dh M Magh Wh You M.M
MM I
M4 MA M
opena spon
Soungh
Optina Sranch
PAL
14 Mo MM
/ M
Cycle
Syde
oyce
(a) CAMPAL
MIN Entropy
(b) CAMPAL SUM Entropy
(c) CAMPAL DENISTY Entropy
Mou h Nam Io Miro
Wor M Mww Mor Ma M
Maa. pa Mo M
Optina Smrrggh
Osdia Saueggh
Optindl Shergth
Cysle
Gyde
CPC
(d) CAMPAL MIN
(c) CAMPAL SUM SUM -LC
(f) CAMPAL DENISTY
NW
M MoM
Opena Strrrgn
Sounghh
Ortina Srunch
Cyde
CAMPAT MIN Margin
(h) CAMPAL SUM Margio
CAMPAL DENISTY Margin
Optiral Steegh
Opiind Stregg
Optindl
Crdee
Cyde
oce
CAMPAL STANDARD Coreset
(k) CAMPALCHAR CHAMFER
(I CAMPAL HAUSDORFF CorescC
Aag M0
opt Strong
Spimml/Srnnp Strnngh
Opemal Strnnds
11 w/ ww
WRM M MMN
MM
Cycle (m) CAMPAL STANDARD DAMPALDADGE
Oyde
Oxe (0) CAMPAL HAUSDORFF CAMPALEADGE BADGE
(n) CAMPAL CHAMPER
Figure The average optimal strength m,, m. across different AL cycles on CIFAR-10 with different instantiated versions for CAMPAL.
20


Reviews:
[{"review_id": "k2d5zzoHyc", "reviewer": "Reviewer_ev91", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a heuristic pipeline to improve existing active learning methods using controllable augmentation.", "summary_of_the_review": "Overall, I think the paper lacks technical rigor (see Weakness for details) --- unfortunately, I cannot recommend acceptance. ", "strength_and_weaknesses": "Strength:\n1. The proposed method applies to a wide range of augmentation methods and acquisition methods.\n2. The authors clearly explain the heuristics in the proposed algorithm. \n\nWeakness:\n1. My primary concern is about the comparison against semi-supervised learning. Given the same quota of labeled samples, active learning should outperform semi-supervised learning --- while active learning (AL) can actively select the training subset, semi-supervised learning (SSL) passively takes the training subset. However, I find the proposed algorithm significantly underperforms the MixMatch, the benchmarking method in SSL. For example, given 2000 samples, CutMix achieves an error rate of 7.03% for CIFAR-10 and 3.04% for SVHN (See Appendix B); in contrast, the proposed algorithm only gets 19.63% for CIFAR-10 and 14.34% for SVHN (See Table 1). It raises a natural question of why people would like to use AL instead of SSL. \n2. The technical descriptions in Section 2.2 are somewhat hand-waving. The authors did not describe the math definitions of different types of augmentations. Also, it is unclear how to solve the optimization problems in (1) and (2). Whether these problems are solved in closed form or iteratively? Are they solved repeatedly once the labeled set is expanded or once at the beginning of the algorithm? Minor: I guess the denominator in (2) actually means the cardinality of the augmentations.\n3. The theoretical explanations are disconnected from the actual algorithm. The paper does not justify why the proposed algorithm meets the assumptions/conditions in theory. For example, why the augmentations used in the paper is \"moderately weak\" (Assumption 2)? What does it mean by \"properly selected\" in Theorem 4? Without rigorous development, the theory will not shed light on the algorithm.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j_fnQQZNLV", "reviewer": "Reviewer_mJxF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new active learning approach by taking into consideration augmentations during the Active Learning acquisition phase. The paper argues that the existing active learning baselines do not effectively utilize unlabeled samples, particularly during the early stages of acquisition. To fix this, the authors propose to use augmentations both for training over the labeled samples as well as for acquisition over the unlabeled samples. Importantly, the way augmentations are used differs for the labeled and the unlabeled samples. For labeled samples, we want to have label-preserving augmentations that ensure consistency in the model's output across augmentations. However, for unlabeled samples we want the augmentations to be informative enough to enrich the distribution of the unlabeled data, making it easier for the active learning strategy to acquire the best samples for labeling, in other words, the augmentations that provide the maximum least information across the unlabeled samples. The paper thus optimizes for the strength of the augmentations (or the number of augmentations) separately over the labeled samples and the unlabeled samples using the above-mentioned criteria. Once the augmentation strengths are chosen, the authors propose different strategies in which the augmentation over the unlabeled samples can be used for the acquisition, both for score-based and representation-based acquisition functions. Via extensive study, the authors show that their proposed approach performs better than the different Active Learning baselines over different datasets and across different labeled data sizes. In addition the paper also proposes a theoretical justification behind their approach.", "summary_of_the_review": "To summarize, while I agree that the paper's idea is good and novel I have my concerns with the experimental setup. The paper would do great if the proposed approach could be applied over semi-supervised approaches also. I also have some concerns over the comparison with a fixed augmentation strategy, RandAugment. \n\nI would be willing to update my ratings if my above concerns are addressed.", "strength_and_weaknesses": "Strengths - \n\n1. I like the motivation mentioned in the paper about separating the role of augmentations over the labeled and unlabeld set. \n2. The ablation study does show that indeed the strengths of augmentation over unlabeled set is higher than labeled set, which justifies the reasoning of the paper. \n3. I also like that the paper shows the comparison against having a fixed augmentation strength, where it is shown that certain combination of fixed augmentation strength can be non-optimal and hence the proposed approach of optimizing for the augmentation strength is better. \n\nWeakness - \n\n1. My main weakness of the paper is that since it is also considering augmentations over the unlabeled set, a fairer comparison would be the semi-supervised learning approaches such as [1] which achieve much superior results with much fewer samples. The existing Active Learning approaches do not play with the unlabeled samples generally and thus their setup is much simpler and unfair to compare with other semi-supervised learning setups. In contrast, the proposed approach here explicitly optimizes the augmentation strength over unlabeled samples, making their setup complex. If indeed an ML practitioner wants to use unlabeled data for their model, they instead can rely on using semi-supervised learning instead of the proposed approach here which involves multiple steps. The paper's worth would be improved much more if the authors show that their method can be added on top of the semi-supervised approaches such as [1] too.\n\n2. While I like the comparison done in the appendix with RandAugment, it seems that the proposed approach is not very better than RandAugment (only 1-2%). Further, the authors optimize for RandAugment over labeled samples. What happens if we instead use the RandAugment policy that's been optimized over larger datasets such as ImageNet and then use the same policy across all the datasets mentioned in the paper?\n\n3. How do the authors select their best performing CAMPL, as there are different search axes, which active learning to use and also which aggregation mode to use during including the augmentations in the Active learning approach. Do the authors have a separate validation set and then select the best variant by its performance over the validation set? I could not see any reference to the validation set in the paper. In particular when comparing against RandAugment, is the validation set same for learning the best RandAugment policy and for selecting the best CAMPL approach?\n\n4. Since the proposed approach involves multiple steps, I believe the time taken for this would be much larger than other active learning approaches, however, I could not find a mention of it. Can authors also compare the time taken for their approach against the baseline Active learning approaches?\n\nReferences - \n[1]FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. Sohn et al.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cLTSuURLif", "reviewer": "Reviewer_VeMh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method, CAMPAL, for better enhancing active learning with data augmentations. To achieve this, they provide a strength controlling mechanism for augmentations integrated into active learning, as well as a newly developed augmentation-acquisition procedure. The author also provides strong empirical results on several benchmarks, together with theoretical guarantees. The paper brings an overlooked research interest for active learning, fully investigating the role of data augmentations when integrating them into active learning.", "summary_of_the_review": "The paper proposes a new augmentation-acquisition framework for active learning, CAMPAL, to better enhance active learning with data augmentations. The proposed method is quite effective on several benchmarks, especially with scarce labeled data at the first few active learning cycles. The data augmentation is important and effective in active learning, but very few works have been devoted to this problem. Therefore, this paper is great to fully investigating the impacts of data augmentations on active learning.", "strength_and_weaknesses": "The paper proposes a new augmentation-acquisition framework for active learning, namely CAMPAL, to better enhance active learning with data augmentations. The author empirically and theoretically verified the effectiveness of CAMPAL. The point that the augmentation flows impacts AL differently on labeled and unlabeled data pool is also interesting.\nThe strengths are:\n1.\tThe proposed method is novel and well-motivated, and is clearly written in this paper.\n2.\tThis paper fully investigates an overlooked research problem in active learning, and illustrates its importance with significant performance boosts.\n3.\tThe authors also provide a theoretical guarantee of the proposed method.\nThe weaknesses are:\n1.\tThis emphasizes the efficacy of CAMPAL when labeled data are scarce. How about the efficacy of CAMPAL in a normal setting, i.e. with 10%~40% labeled samples from the dataset?\n2.\tHow do you define the image similarity $\\rm{sim}(x,\\tilde{x})$ in the score-based acquisition in Section 2.3? Please elaborate on it.\n3.\tAs shown in Figure 3 and Table 3, the performance boost on CIFAR-100 brought by CAMPAL is not as significant as that on SVHN, and CIFAR-10. It would be better to give out an explanation for this phenomenon.\n4.\tIn equation (2) in Section 2.2, the term $\\frac{\\lambda_{2}}{\\mathcal{T}^{(s)}(x)}$ is confusing, maybe it is $\\frac{\\lambda_2}{|\\mathcal{T}_{\\rm mix}^{(s)}(x)|}$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject