Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
Authors: Simret A Gebreegziabher, Kuangshi Ai, Zheng Zhang, Elena Glassman, Toby Jia-Jun Li


Reviewer paper summaries: The paper uses variation theory to tackle the cold-start problem of active learning. The proposed approach uses a large generative LLM to first create a dataset of counterfactual examples, after carefully selected filtering), and then uses it to fine-tune a large generative LLM. Finally, the fine-tuned counterfactual generator can be used to create examples that will be selected for training a separate model via active learning. The results in three datasets with two models show that the proposed approach outperforms the baselines in the first active learning iterations.
Summary of strengths: - The paper is clearly written and has a concrete motivation for the proposing to use variation theory as a way to address the cold start problem in active learning.
- The description of the proposed approach is thorough and experimentally rigorous.
- The results show that the proposed approach outperforms the baselines.
Summary of weaknesses: - The paper misses an entire sub-section of related work on active learning, which is the main contribution of the paper. The authors should add a subsection 2.3 and discuss related approaches to active learning with LLMs from the literature (see some missing references below). It is important to position the paper with the current state-of-the-art in order to better evaluate its impact and contribution.
- The choice of baselines is poor. Their description is not clear (see below in Questions). There are also several baselines missing of considerably similar approaches. For instance ALPS from Yuan et al., 2020, which is referred throughout the paper is not considered as a baseline even thought it is an AL algorithm proposed specifically yo address the cold-start issue.
- Taking a step back, I am trying to understand what is the actual value of such an expensive approach. The proposed method uses several LLM calls to create counterfactual data, an LLM discriminator to filter the candidate generated examples, fine-tunes GPT-3.5 (an 100-billion parameter model) to be used as a counterfactual generator. I am not convinced that is it worth it (especially regarding the lack of simpler baselines as well). The authors should do some analysis on the computation required for their approach compared to the baselines. This way we could have a holistic view of the trade-off between cost and performance improvement.
Comments, suggestions and typos: Comments & Questions
- It is not what "user querying" (line 5) refers to in the abstract.
- Limitations should be a different section. Conclusion section, typically the last section of the paper that summarizes the paper and proposes future research directions is missing. Instead Future work is mixed with the limitations section (which does not count on the total page limit).
- What is the process of fine-tuning the generator GPT-3.5? How many examples are used? What is the cost?
- What word vectors are used for the clustering-based baseline (line 453)?
- How are model logits used exactly as uncertainty estimation for the uncertainty-based baseline for BERT (line 462)?
- How are the counterfactuals created in the baseline without variation theory (line 464)? This is not clear at all and is not trivial.
- What does "high intra-coder reliability, but low inter-coder reliability" mean exactly (line 476)? How is it estimated/considered?
- While reading the paper there is a confusion as to the main goal of this work. Is it to improve the cold-start problem of AL or to create a robust counterfactual generator? I thought is was the former but the first results (4.4.1) are about the latter. This is confusing.
- It is not clear what fine-tuning with 10-170 examples means (lines 499-508). Do the authors actually fine-tune GPT-3.5 with 10 examples?? Or they did in-context learning? What is the process? Crucial details are missing from the paper.

Missing References
- Active Prompting with Chain-of-Thought for Large Language Models, ACL 2024
- Active Learning Principles for In-Context Learning with Large Language Models, Findings of EMNLP 2023
- On the Limitations of Simulating Active Learning, Findings of ACL 2023
- A Survey of Active Learning for Natural Language Processing, EMNLP 2022
- On the Importance of Effectively Adapting Pretrained Language Models for Active Learning, ACL 2022
- Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates, EACL 2021
- Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering, ACL 2021
- Active Learning by Acquiring Contrastive Examples, EMNLP 2021
- Active Learning for BERT: An Empirical Study, EMNLP 2020
Overall assessment scores: 2.5