Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed the "purity-informativeness dilemma" problem and a meta-learning algorithm - MQ-Net to adaptively balance the purity and informativeness scores for better sample selection. The experimental results verify the reasonability of the concerned problem and the proposed solution. 
strengths_and_weaknesses: # Strengths 

1. The "purity-informativeness dilemma" problem is essential, and the analysis is convincing. 
2. The design and implementation of MQ-Net is reasonable. 
3. The active learning architecture is clean and clear, with significantly better performance.  

# Weakness 

1. The paper does not discuss the benefit of training the MQ-Net in each round.
2. The performance comparison between MQ-Net and other simple alternatives is not discussed. Those alternatives may include  heuristic rules like $\frac{1}{(P(x) - 1) ^2 + (I(x) - 1)^2}$, logistic regressions, and ranking SVM.  
3. The architecture of MQ-Net is not clearly stated. The activation functions are not reported, and the layer number is only reported in the appendix. 

questions: As said in the weakness part:

1. Is it necessary to train the MQ-Net in each round? What if we train it in the first round (or first several rounds) and fix it in the remaining round?

2. Figure 4 indicates that the function learned by the MQ-Net is fairly simple. How do the MQ-Net's performances differ from those simple alternatives?
 
limitations: The authors mentioned one limitation: they only consider one purity score and one informativeness score as input. They left the multi-score input as future work. 

However, the multi-score version is intuitive by adding more input dimensions to MQ-Net. Since the authors had already computed many scores in  Section 5.4, the multi-score version is very convenient to implement. Furthermore, when more scores are used, there would be a score selection problem in MQ-Net. The solution to that problem will increase the quality of the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduced the open-set noise problem that might exist in active learning, illustrated the purity-informativeness dilemma and proposed a meta-query (MQ) net as a plugin in normal active learning processes. This paper is well-written before experimental part and easy to follow.
strengths_and_weaknesses: Strengths: The idea of adding a mq net into normal active learning is interesting, it just like LL4AL, which add an extra plugin/network to estimate whether the unlabeled data set belong to ood data or not. The design is both effective and flexible. Additionally, the L(SQ) inherently contains multi-objective optimization (pareto optimization) design ($P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$ for finding pareto fronts). It is interesting.  

Weaknesses: there are still several problems left in this paper. 
- Firstly is the definition of the open-set noise problem. I think it is just ID/OOD problems, which is properly defined as class mismatch in CCAL and OOD data scenarios in SIMILAR, there is no need to create a new concept and called it open-set noise problem. This concept is wider, for instance, some instances are really belongs to the ID data distribution but contains noise in x, it is not OOD data but it still contains noise. But the author only conduct experiments follows class mismatch settings in the experimental part. 
- Secondly is the motivation of the purity-informativeness dilemma,  1) the author provide an example in Figure 1. This example is not convincing enough since LL4AL and CCAL are both non-typical methods, LL4AL is jointly trained with a LossNet and CCAL use SimCLR/CSI for extracting features, they are not comparable. Additionally, the example only show the first 10 rounds and show low-noise (10% and 20 % ood rate) situations; 2) Why can't we maintain purity all the time and at the same time acquire high informativeness? In addition, if there is a method to achieve the ideal/ Optimal effect, the proportion of OOD sample in unlabeled data pool will naturally be higher and higher, and more attention should be attached to purity. 
- Thirdly is the experiments, there is no error bar of the conduct experiments. The experimental results on CCAL and SIMILAR are indeed very strange, on low noise situations (10% and 20% OOD data rate), are even worse than typical uncertainty-based sampling strategy (e.g., CONF), I have contacted the authors of CCAL and SIMILAR to asked them if their models would perform worse than typical uncertainty-based measures like (CONF, ENTROPY), the author of SIMILAR paper said "If there is low-noise then it should only be less challenging and the performance should at least be consistent and better than MARGIN.". CCAL author show me their new experiments on low-noise data scenarios, also better than typical uncertainty-based measures. Since the author didn't provide the code (only provide the pdf version of appendix), I cannot check the implementation. Is it fair comparison?

questions: Most of the questions are listed in previous part.

Questions:

1. The definition of open-set noise (see previous part).

2. The motivation/example of purity-informativeness dilemma contains a contradiction (see pervious part). In my view, in ideal case, it is purity rather than informativeness should be emphasized in latter AL processes.

3. In line 63-64, the author said "The input to the meta-model, which includes the target and OOD labels, is obtained for free from each AL round’s query set by leveraging the multi-round property of AL." This is an advantage, learned from the queried ID and OOD samples. But learning is for updating the META model to better output $\Phi(<P(x), I(x)>;w)$, instead of using it to get better $P(x)$ and $I(x)$, it feels like it is just to train a classifier, but in deep learning tasks, the feature representation and classifier are jointly trained. 

4. In line 95, why choose classifier-dependent approach to get a meta-model, what is the motivation? Did the author compared with other density- and self-supervised  based methods?

5. In line 277-278, why use CSI and LL for calculating the purity and informativeness scores? Especially LL, some researches show that LL is not stable due to the joint training with LossNet in some tasks. 

6. Is mq-net joint trained with basic classifier (e.g., ResNet18) or not like LL?

7. Equation 3, the situation is both $P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$, this is apparently to find pareto fronts if one regard it as pareto-optimization problem. Could the author provide some discussions of the situation the number of pareto front set (the data samples that satisfies both $P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$) less than batch size (labeling budget b in main paper) in active learning process?

8. Is the ResNet18 pre-trained?

9. Did the author conduct repeat trials per experiment? (no error bar problem, mentioned in previous part)

10. The experimental results of the CCAL, SIMILAR and standard AL approaches (mentioned in previous part)

11. In line 284-285, the author already defined the cost of querying ood data samples, why it is not an evaluation metric in later experimental result analysis? It is important if the author could present the cost of querying ood data samples, the less cost means that the labeling budget is less wasted.

Hope the authors could provide convincing responses in rebuttal, I will increase my score if the author could persuade me.

Small typo: Line 261, "OOD" not "ODD".






limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Active learning typically involves querying and labeling points according to their informativeness, e.g. some notion of uncertainty or diversity. However, when there is out-of-distribution (OOD) data in the unlabeled dataset, such criteria may result in querying these OOD points, which wastes the labeling budget. Recent work has focused on improving the purity of queried points by filtering out all the OOD data and selecting informative in-distribution (IN) points. This paper challenges the notion that filtering OOD data needs to be done to the same extent at each round of active learning. Instead, it may be true that at later rounds, informative but impure OOD data points could be useful to query. There is hence a purity-informativeness dilemma that presents a dynamic tradeoff in the querying process. The paper introduces a meta-model, Meta-Query-Net (MQ-Net) that learns a score that is a function of both purity and informativeness scores updated after each round of AL using the labeled self-validation set already created from previous rounds. They also enforce a skyline constraint to encode ordering of the scores and implement it via nonnegative constraints. Empirically, MQ-Net outperforms both AL that doesn’t account for OOD data, and AL that is OOD aware but static throughout rounds. They also show how MQ-Net gradually prioritizes informativeness more than purity in later rounds, and perform ablation studies with varying purity/informativeness scores, removing the skyline constraint, and replacing the self-validation set with a random validation set.
strengths_and_weaknesses: **Strengths**

_Significance_: The presence of OOD data can indeed confound standard querying approaches in AL and is an important problem in real datasets. It is unclear how to prioritize potential signal in such data dynamically, which this paper takes a principled step towards.
 
_Originality_: the idea of using a self-validation set to dynamically meta-score points is novel to me and can inspire new ways of adaptively combining different metrics on unlabeled data.

_Quality_: thorough experiments and ablations convince me of MQ-Net’s utility. 

**Weaknesses**

_Quality_: it seems like MQ-Net would be more computationally expensive than previous approaches since a model is trained at each round. It would be interesting to see how much longer MQ-Net takes. 

_Presentation_: as someone who does not publish in this area, I had a few comments about clarity (in questions below).


questions: **Q1.** Given that MQ-Net requires training a model at each round, how long does MQ-Net take to run versus other baselines like CCAL and SIMILAR? 


**Q2.** A running example of what the OOD data and informative versus non-informative data looks like would be helpful. For instance, Figure 1a explains the purity-informative dilemma, but it is not clear what the dataset and task are.


**Q3.** Equation 1 is defined as the optimal query set approach, but is not mentioned otherwise in the paper. It is not clear what the purpose of presenting the upper limit of a query set is. Also, the cost constraint in equation 1 is used in MQ-Net but is not mentioned in that section.

**Q4.** The intuitive interpretation of $L(S_Q)$ was not clear. The main idea from previous sections was that we want a loss that emphasizes informativeness more in later rounds. How does this loss function do that? In particular, it would be good to just have some further discussion about the optimal ranking learned by $L(S_Q)$, case by case. E.g., OOD examples are always scored lower than IN examples, but amongst OOD examples they are roughly ranked by their dot product of purity and information.

**Q5.** Typo: lines 52-53, “The HP-focused approach improves….than the HP-focused one at earlier AL rounds”





limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper discusses the purity-informativeness dilemma for open-set active learning, where excessively increasing purity can lead to loss of informativenss. As mitigation they propose a new meta-heuristic, which learns to monotonically combine a purity and an informativeness metric, called MQ-Net. MQ-Net is trained on the acquired query sets alternatingly to utilizing it for query selection.
strengths_and_weaknesses: Strengths
* the paper addresses a practically relevant problem
* the proposed heuristic intuitively makes sense, and is compatible with different choices of purity or informativeness measures
* there is small overhead of the heuristic and it seems to be easy to implement

Weaknesses:
* in contrast to the answers in the checklist, no code is provided. It is also unclear how hyperparameters of the MQ-Net were chosen, and there is no report about the resource usage.
* without reporting standard deviations, it is hard to assess the significance of the empirical results - since the authors already report running the experiments five times with different random seeds it is unclear why they did decide to do so; in addition there is also no comparison against the performance of a random acquisition baseline - having it would justify the necessecity of using an active learning strategy at all
* some parts of the paper could be improved in readability, e.g., in the problem statement, $X_{IN}$ is initially defined as set of in-distribution samples (without their label), but later one, $T_{IN}$ coming from the same distribution $D_{IN}$ contains pairs of (input, target).
questions: Major questions:
* could you additionally provide the standard deviation across the five different runs?
* what computing infrastructure was used? In particular, I wonder about the resource requirements for experiments conducted on the ImageNet dataset
* how do you do the z-score normalization, i.e., over which parts do you compute mean and standard deviation?


Minor questions:
* Figure 4: it appears that all red OOD samples received a high purity score - shouldn't it be vice-versa?
* in Theorem 4.1., the constraint that the activation function needs to be monotonically non-decreasing is not mentioned (it is however in the appendix). This is an important requirement, and while most commonly used activation functions are monotonically non-decreasing, some, e.g., Swish, are not
* in the problem statement, you introduce different costs for annotation of OOD and IN examples. To me it is not clear what would motivate this. In the experiments, you then set same costs for OOD and IN labeling, as in related work.

Minor comments:
* L52-53: One of the "HP" should be something else
* L80: directions -> direction
* L89: k-MEANS -> k-MEANS++
* L115: more higher -> higher
* L120: delimma -> dilemma
* L187: a -> the
* L324-325: this could make a good plot, e.g., x=AL round, y=noise ratio, color=slope
* L363: legit -> legitimate / valid
* Figure 4 is very small and thus hard to read
limitations: Yes, although the discussion is only present in the appendix.
review_title: null

Meta-review: metareview: The reviewer agree that the paper studies an important problem, and they supported acceptance for highlighting the trade-off between purity and informativeness, which is both novel and significant. 

Among the weaknesses that have been raised, there is
- simpler alternatives to using a their MQ-Net would be a plus (Rev s9UD) 
- motivating example is a bit handwavy and no clear theoretical motivation (Rev joSH)
- overall, the improvement with respect to baselines is small [answered in the updated appendix with stdev]
- Theorem 4.1 is fairly straightforward, and monotonic networks have been studied for long 

While the reviewers agree that the strength outweight the weaknesses, the authors are invited to take these remarks into account when preparing the final version of their manuscript.
confidence: Less certain
Decision: Accept