Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is in the field of Active learning, which has proven to be useful for minimizing labeling costs by selecting the most informative samples from unlabeled datasets, which are then to be labeled by a human in the loop. The goal is to achieve high accuracy of the trained model, while minimising the number of points to be labeled by the human in the loop. However, as the authors say, many existing state of the art active learning methods are trained on datasets which are not very realistic, and they do not perform well in many common real-life situations such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. Given the size of these datasets, cleaning them manually is not realistic. The authors propose an active learning framework which they call SIMILAR, which uses submodular information measures as acquisition functions. They argue that SIMILAR performs well on the realistic scenarios mentioned previously, and empirically demonstrate that SIMILAR outperforms previous state of the art methods in these scenarios. 

main_review: I think this paper is well written, addresses important practical problems and has interesting results. Some comments:

1) It seems like out of the three cases of imbalance/rare classes, out-of-distribution data in the unlabeled set, and redundancy, the framework can only deal with solving one of these issues per round/batch. Because it can handle only one situation at a time, the framework may not perform well for datasets facing multiple of these issues.

2) The performance of the algorithm depends crucially on and is very sensitive to the choice of the similarity function. The choice of similarity function needs to be better justified, and I suspect finding an ‘optimal’ similarity function will be a very hard problem to solve. A bad choice of similarity function could lead to bad performance, and maybe even lead to discrimination or unfairness. 

3) It seems that one would often need to know the unlabeled dataset pretty well before using this framework. For example, in the task of ‘creating  a (small) held-out set R which contains data points from rare classes’ before running the algorithm, which may not be an easy task. 

4) It seems like a user of this framework would need to know beforehand what issues the unlabeled dataset faces (for eg., rare classes, OOD, redundancy), which may not always be a reasonable assumption. 

In addition, a more detailed explanation of choices made in designing the framework and why they lead to good performance would be nice.

Edit (after author response): Thanks for the detailed comments. After going through them, I have changed my score accordingly.  

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new active learning framework SIMILAR based on submodular information measures (SIM). The framework aims to solve the problem of active learning (AL) raised in realistic scenarios including rare classes, redundancy and out of distribution. 
main_review: ### Strengths:
The proposed AL framework, SIMILAR can solve multiple realistic scenarios of active learning problem with a unified acquisition function form.     
Computation of the acquisition function is reasonable.    
The experiments on real-world high dimensional datasets demonstrate the effectiveness of the proposed framework.     

### Weaknesses:
The paper needs a more detailed description of the active learning problem setting for the readers, for example this paper focuses on the "pool-based AL with batch queries...". 

Although SIMILAR can applied on multiple scenarios, it seems that the framework still lacks flexibility for more general and realistic scenarios. 
* For example, in the rare classes scenario, the acquisition function is defined as the SMI between $A$ and $R$, focusing only on the rare classes, so the SIMILAR is only applicable to the case that the rare classes are much more important than the regular classes, there is no parameter to balance the tradeoff between them. 
* Another example is that in the redundancy scenario, the labeled set $L$ is set as the conditioning set, while in the OOD scenario, the labeled in-distribution set $I$ is set as the query set, so SIMILAR cannot be applied on scenarios with both Redundancy and OOD. 


limitations_and_societal_impact: The authors have adequately addressed the limitations and societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a novel active learning framework (called SIMILAR) that leverages Submodular Information Measures to deal with both "mainstream" active learning scenarios and situations in which there are rare classes, redundancy, or out-of-domain classes.
main_review: The authors propose a novel approach to an important problem: active learning in the real world. The paper is well written & reasonably easy to follow, and it makes a strong case for the proposed approach. The contribution appears to be original, and the authors make a compelling case for the use of Submodular Information Measures to deal with rare classes, data redundancy, and out-of-domain classes. 

However, the paper suffers from two main weaknesses:
- while it claims to be motivated by applying active learning in the real world, the datasets used to illustrate its ability to handle in real world scenarios (i.e., rare classes, data redundancy, and out-of-domain classes) are synthetically derived from existing domains that do NOT exhibit these properties. Even though the experimental results are encouraging, the paper would greatly benefit by showcasing at least one organic, real-world domain that has these properties
-  part of the challenge in dealing with rare classes & out of domain (OOD) classes (and, in part, data redundancies) is to have an algorithm that can automatically detect the rare or OOD classes, such as [Muslea et al, 2006] below. Requiring the user to discover and provide examples from rare or OOD classes represents a challenge in itself. While this is easily done in synthetic domains, it if far from obvious that this assumption  will hold in the real-world, especially if it is to be done a reasonable cost/effort. This is another reason while the paper would benefit from a real-world domain. 

Muslea, I., Minton, S., & Knoblock, C. A. (2006). Active learning with multiple views. Journal of Artificial Intelligence Research, 27, 203-233.

OTHER COMMENTS:
- line 247: please elaborate & quantify "more scalable." In this reviewer's opinion, the scalability issue deserves a section of its own, rather than being hidden under a paragraph called "Partition Trick," especially given the scalability claim in the abstract (line 9)
- line 343: does it matter which classes are chose as out of distribution? it probably does depend on how similar an OOD class is wrt an In-Distribution one 
limitations_and_societal_impact: none
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a general framework for batch-based deep active learning
that (1) leverages submodularity of the acquisition function to select a
batch in almost linear time, (2) defines a modular acquisition function
that can support cases like selecting instances different from known OOD
samples and favoring instances similar to known rare samples, (3) supports
a variety of submodular functions defined over a gradient-based kernel.
main_review: The paper flows very well, all concepts are presented clearly.  The
contribution is clearly on-topic, simple and elegant.  I appreciate having a single formulation
that covers several challenging use-cases.


The related work does a good job at positioning the paper method against the SotA.  The only missing
paper that I was expecting to see cited is:

  Shui et al. "Deep Active Learning: Unified and Principled Method for Query and Training".

as they give the (to the best of my knowledge) best theoretical justification / foundation for batch-based deep AL to
date.  Not a huge deal.

The design of the experiments is reasonable.  The results show clear improvements over the competitors in most
settings.

Generally speaking, I enjoyed reading this paper.  I only have a few remarks / questions:

1. The proposed method is not really justified from a learning theoretical standpoint, in the sense that the specific choice of submodular function is not derived from first principles / clear desiderata.  I realize that SIMILAR is meant as a framework, and therefore it can fit a variety of different submodular functions without requiring modifications, but it is also true that different submodular acquisition functions may have different properties.  This is exactly what the empirical evaluation shows.  Not a huge deal.

2. One downside of the experiments is that the data sets are manipulated to introduce the desired conditions, e.g., rare classes.  It would have been more convincing to use data sets in which OOD data and rare classes occur naturally.  It is also not ideal that all experiments are done on image data only.

3. My only real complaint is that the use-cases are not entirely realistic.  For AL under OOD, OOD data is often not available - this is precisely the problem - and cannot be used to guide the batch selection strategy.  A more thorough discussion of the practical applicability of the proposed method would have been appreciated.

(As a side note: I see that in many papers BADGE is considered SotA - however in practice it tends to underperform compared to simpler methods.  The experiments in the BADGE paper are not particularly convincing either.  I wonder why people keep comparing to it.  Of course, I don't want to single this paper out;  plenty of papers out there do the same.)
limitations_and_societal_impact: A better description of the practical applicability of SIMILAR would have been appreciated.

The broader impact does a good job at covering potential negative effects.
review_title: null

Meta-review: The work proposes a novel active learning framework based on submodular information gain maeasures. The framework is well motivated and explained, and then empirically evaluated.

This submission was well appreciated by the reviewers. The authors are encouraged to take the reviewer's comments and their responses into account when preparing their final version for publication.
Decision: Accept (Poster)