Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a heuristic pipeline to improve existing active learning methods using controllable augmentation.
strength_and_weaknesses: Strength:
1. The proposed method applies to a wide range of augmentation methods and acquisition methods.
2. The authors clearly explain the heuristics in the proposed algorithm. 

Weakness:
1. My primary concern is about the comparison against semi-supervised learning. Given the same quota of labeled samples, active learning should outperform semi-supervised learning --- while active learning (AL) can actively select the training subset, semi-supervised learning (SSL) passively takes the training subset. However, I find the proposed algorithm significantly underperforms the MixMatch, the benchmarking method in SSL. For example, given 2000 samples, CutMix achieves an error rate of 7.03% for CIFAR-10 and 3.04% for SVHN (See Appendix B); in contrast, the proposed algorithm only gets 19.63% for CIFAR-10 and 14.34% for SVHN (See Table 1). It raises a natural question of why people would like to use AL instead of SSL. 
2. The technical descriptions in Section 2.2 are somewhat hand-waving. The authors did not describe the math definitions of different types of augmentations. Also, it is unclear how to solve the optimization problems in (1) and (2). Whether these problems are solved in closed form or iteratively? Are they solved repeatedly once the labeled set is expanded or once at the beginning of the algorithm? Minor: I guess the denominator in (2) actually means the cardinality of the augmentations.
3. The theoretical explanations are disconnected from the actual algorithm. The paper does not justify why the proposed algorithm meets the assumptions/conditions in theory. For example, why the augmentations used in the paper is "moderately weak" (Assumption 2)? What does it mean by "properly selected" in Theorem 4? Without rigorous development, the theory will not shed light on the algorithm.
clarity,_quality,_novelty_and_reproducibility: I find the paper writing is fairly hand-waving --- numerous rigorous definitions and technical details. But, most importantly, the theoretical explanation seems disconnected from the proposed algorithm.
summary_of_the_review: Overall, I think the paper lacks technical rigor (see Weakness for details) --- unfortunately, I cannot recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method, CAMPAL, for better enhancing active learning with data augmentations. To achieve this, they provide a strength controlling mechanism for augmentations integrated into active learning, as well as a newly developed augmentation-acquisition procedure. The author also provides strong empirical results on several benchmarks, together with theoretical guarantees. The paper brings an overlooked research interest for active learning, fully investigating the role of data augmentations when integrating them into active learning.
strength_and_weaknesses: The paper proposes a new augmentation-acquisition framework for active learning, namely CAMPAL, to better enhance active learning with data augmentations. The author empirically and theoretically verified the effectiveness of CAMPAL. The point that the augmentation flows impacts AL differently on labeled and unlabeled data pool is also interesting.
The strengths are:
1.	The proposed method is novel and well-motivated, and is clearly written in this paper.
2.	This paper fully investigates an overlooked research problem in active learning, and illustrates its importance with significant performance boosts.
3.	The authors also provide a theoretical guarantee of the proposed method.
The weaknesses are:
1.	This emphasizes the efficacy of CAMPAL when labeled data are scarce. How about the efficacy of CAMPAL in a normal setting, i.e. with 10%~40% labeled samples from the dataset?
2.	How do you define the image similarity $\rm{sim}(x,\tilde{x})$ in the score-based acquisition in Section 2.3? Please elaborate on it.
3.	As shown in Figure 3 and Table 3, the performance boost on CIFAR-100 brought by CAMPAL is not as significant as that on SVHN, and CIFAR-10. It would be better to give out an explanation for this phenomenon.
4.	In equation (2) in Section 2.2, the term $\frac{\lambda_{2}}{\mathcal{T}^{(s)}(x)}$ is confusing, maybe it is $\frac{\lambda_2}{|\mathcal{T}_{\rm mix}^{(s)}(x)|}$.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly rewritten with high quality. The novelty is also guaranteed with its newly-developed augmentation flows constructed distinctly on labeled and unlabeled data towards their own objectives. The reproducibility is also guaranteed with the code given.
summary_of_the_review: The paper proposes a new augmentation-acquisition framework for active learning, CAMPAL, to better enhance active learning with data augmentations. The proposed method is quite effective on several benchmarks, especially with scarce labeled data at the first few active learning cycles. The data augmentation is important and effective in active learning, but very few works have been devoted to this problem. Therefore, this paper is great to fully investigating the impacts of data augmentations on active learning.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new active learning approach by taking into consideration augmentations during the Active Learning acquisition phase. The paper argues that the existing active learning baselines do not effectively utilize unlabeled samples, particularly during the early stages of acquisition. To fix this, the authors propose to use augmentations both for training over the labeled samples as well as for acquisition over the unlabeled samples. Importantly, the way augmentations are used differs for the labeled and the unlabeled samples. For labeled samples, we want to have label-preserving augmentations that ensure consistency in the model's output across augmentations. However, for unlabeled samples we want the augmentations to be informative enough to enrich the distribution of the unlabeled data, making it easier for the active learning strategy to acquire the best samples for labeling, in other words, the augmentations that provide the maximum least information across the unlabeled samples. The paper thus optimizes for the strength of the augmentations (or the number of augmentations) separately over the labeled samples and the unlabeled samples using the above-mentioned criteria. Once the augmentation strengths are chosen, the authors propose different strategies in which the augmentation over the unlabeled samples can be used for the acquisition, both for score-based and representation-based acquisition functions. Via extensive study, the authors show that their proposed approach performs better than the different Active Learning baselines over different datasets and across different labeled data sizes. In addition the paper also proposes a theoretical justification behind their approach.
strength_and_weaknesses: Strengths - 

1. I like the motivation mentioned in the paper about separating the role of augmentations over the labeled and unlabeld set. 
2. The ablation study does show that indeed the strengths of augmentation over unlabeled set is higher than labeled set, which justifies the reasoning of the paper. 
3. I also like that the paper shows the comparison against having a fixed augmentation strength, where it is shown that certain combination of fixed augmentation strength can be non-optimal and hence the proposed approach of optimizing for the augmentation strength is better. 

Weakness - 

1. My main weakness of the paper is that since it is also considering augmentations over the unlabeled set, a fairer comparison would be the semi-supervised learning approaches such as [1] which achieve much superior results with much fewer samples. The existing Active Learning approaches do not play with the unlabeled samples generally and thus their setup is much simpler and unfair to compare with other semi-supervised learning setups. In contrast, the proposed approach here explicitly optimizes the augmentation strength over unlabeled samples, making their setup complex. If indeed an ML practitioner wants to use unlabeled data for their model, they instead can rely on using semi-supervised learning instead of the proposed approach here which involves multiple steps. The paper's worth would be improved much more if the authors show that their method can be added on top of the semi-supervised approaches such as [1] too.

2. While I like the comparison done in the appendix with RandAugment, it seems that the proposed approach is not very better than RandAugment (only 1-2%). Further, the authors optimize for RandAugment over labeled samples. What happens if we instead use the RandAugment policy that's been optimized over larger datasets such as ImageNet and then use the same policy across all the datasets mentioned in the paper?

3. How do the authors select their best performing CAMPL, as there are different search axes, which active learning to use and also which aggregation mode to use during including the augmentations in the Active learning approach. Do the authors have a separate validation set and then select the best variant by its performance over the validation set? I could not see any reference to the validation set in the paper. In particular when comparing against RandAugment, is the validation set same for learning the best RandAugment policy and for selecting the best CAMPL approach?

4. Since the proposed approach involves multiple steps, I believe the time taken for this would be much larger than other active learning approaches, however, I could not find a mention of it. Can authors also compare the time taken for their approach against the baseline Active learning approaches?

References - 
[1]FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. Sohn et al.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easy to follow. The idea is also novel.
summary_of_the_review: To summarize, while I agree that the paper's idea is good and novel I have my concerns with the experimental setup. The paper would do great if the proposed approach could be applied over semi-supervised approaches also. I also have some concerns over the comparison with a fixed augmentation strategy, RandAugment. 

I would be willing to update my ratings if my above concerns are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper suggests a new active learning (AL) pipeline named CAMPAL by introducing controllable data augmentation in the acquisition (for unlabeled data) and training (for labeled data) phase. For controllable data augmentation, the author optimizes the strength of the augmentations (i.e., the number of applied augmentations) over the labeled and unlabeled samples, respectively, with the proposed objective. Two reviewers are negative, and one reviewer is positive. The main concern of negative reviewers is that CAMPAL underperforms the semi-supervised learning (SSL) method (i.e., MixMatch) even though CAMPAL uses additional annotation query and optimization over the unlabeled dataset. The author argues that CAMPAL has a different setup and suitable applications compared to SSL approaches. However, considering additional annotation cost and optimization over an unlabeled dataset in CAMPAL, AC thinks that in many scenarios, ML practitioners given an unlabeled dataset would prefer SSL approaches, which show better performance than CAMPAL. As the authors mentioned that AC and SSL are two separate (or even somewhat irrelevant) learning paradigms, one can combine AC and SSL for even better performance, which AC thinks the authors should do to avoid the criticism of the inferior performance of AC to SSL. Reviewer mJxF indeed suggested to apply CAMPAL on top of SSL, but the authors did not provide the corresponding results. Overall, AC tends to recommend rejection. 
justification_for_why_not_higher_score: It is unclear why one should use the proposed active learning method instead of using existing semi-supervised ones.
Decision: Reject