Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper focuses on the active learning (AL) paradigm to reduce the label complexity compared to typical deep learning methods and proposes to capitalize on the semi-supervised learning (SSL) context instead of the typical supervised learning (SL) one, to be combined with AL. To that end, the authors advocate a neural pre-conditioning (NPC) algorithm  that relies on neural tangent kernels (NTKs) leveraging the minimum eigenvalue of the Gram matrix induced by NTKs. Empirical results demonstrate that merging NPC with some well-motivated SSL approaches can lead to improved prediction performance utilizing only few labeled data.

strengths_and_weaknesses: STRENGTHS 

- The proposed framework seems to work well in practice. 
- The literature review is good.

WEAKNESSES

- The writing and presentation of the paper admit further improvements. There are many parts of the paper that are hard to follow (see the Questions section below for some examples).
- The novelty of the paper is somewhat limited. The SSL methods used are well known and the use of the Gram matrix (and its properties) in NTKs has been explored in the literature. In addition, besides the empirical performance, the contributions of the advocated approach need to be highlighted more clearly. 

questions: - In line 25 in the sentence “Given images X and a labeling budget B, what’s the maximum performance that can be achieved?”. Where does the performance refer to? Does it refer to a classification task (such as object categorization) or regression (or both)? 
- In lines 29-33, the sentence “SL fails to extricate…newly-acquired labels“ is not clear. Does it just mean that SL does not leverage $\mathcal{Z}_U$ during training?
-  In lines 55-56, in the sentence “Because of the difference..”, where does the “difference” refer to?
-  In line 129, in the sentence “In AL, it is impossible to know acquire labels while ensuring” what is the main verb (know or acquire)? Is it maybe to know acquired labels?
-  In lines  156-158, the sentence “We show after presenting our algorithm in Sec. 3.3 that our selection criterion does indeed select based on the gradient’s direction which captures uncertainty information, which…” is incomplete and unclear. What is selected? 
-  I believe that in Eq. 1 there is a typo. Instead of $\mathcal{Y}^* \subset \mathcal{X}_U$ is it maybe $\mathcal{Y}^* \subset \mathcal{Y}_U$? 
-  In lines 34-35, it is mentioned that “Performance gains as a SL algorithm incrementally acquires labels remain minor relative to passive learning (PL).” This is not true in general. There exist AL-SL based  methods with well-documented merits compared to passive approaches; see e.g [1], [2]. 
-  In line 35, it is mentioned that “This may be attributed to the large numbers of labels required to attain meaningful  accuracy for SL…”. However, this does not always hold; see e.g [1], [2].
-  In lines 38-39, it is mentioned that “While in principle more data should always be better…”. However, this does not always hold especially when the data are noisy (or redundant).
 
Some examples of more minor issues are listed below

- In line 5, instead of “Although unlabeled data is readily…”, it should be “Although unlabeled data are readily…”.
- In line 27, instead of “have been evaluated by measuring downstream supervised…” it should be “have been evaluated by measuring the downstream supervised”. 
- In line 28, instead of “but we argue downstream semi-supervised learning (SSL) performance...” it should be “but we argue that the downstream semi-supervised learning (SSL) performance...”.
-  In line 69, the verb “have” appears twice. 
-  In line 114, instead of “computes” it should be “compute”.

 I strongly suggest that the authors go over the paper carefully and correct the aforementioned and other similar grammatical/syntax errors. 

[1] Kapoor, Ashish, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. "Active learning with gaussian processes for object categorization." In 2007 IEEE 11th international conference on computer vision, pp. 1-8. IEEE, 2007.

[2] Pasolli, Edoardo, and Farid Melgani. "Gaussian process regression within an active learning scheme." In 2011 IEEE International Geoscience and Remote Sensing Symposium, pp. 3574-3577. IEEE, 2011.

limitations: The presentation of the limitations is good in general. Some additional concerns of mine are listed above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission proposes the neural pre-conditioning (NPC) algorithm to leverage the unlabelled data using semi-supervised learning and acquire the valuable data that could be labelled carefully active learning strategy. Furthermore, this work mentioned that traditional active learning methods are typically evaluated with the supervised learning method, which cannot reflect how much the trained model gains the performance from newly acquired labels. Instead, they argue that semi-supervised learning can be a benchmark method to evaluate the Active learning downstream task. The authors state how the NPC algorithm uses the direction of gradients to select the uncertainty samples and prove that the batch-mode operation can increase the diversity of the data selection. They extend the experiments on two datasets with different query sizes and the number of labelled data, comparing the NPC against the other four state-of-the-art active learning methods, which shows the NPC achieves exceptionally high accuracy. In the end, they concluded this work and discussed the limitations and future study directions. 
strengths_and_weaknesses: Strengths
1. The results have significant improvement compared to the previous work, achieving state-of-the-art results. 
2. Quality: This work reads as a complete paper that has complete sections and clear formatting. There are extended comparative experiments to evaluate the proposed method's effectiveness. 

Weaknesses
1. The notations and some presentations are unclear. For example, in eq.1, χ*, Y*and Y_U; in Table 1. What is the Q_0; in 277th row, -u. 
2. Clarity: The presentation is not clear in some parts, like in the introduction part, you mentioned that simply replacing the SL with SSL would introduce a new problem because the class imbalanced issue can deteriorate the SSL performance significantly. Although here you added the reference, I think that supplying more details that explain why this issue induces the SSL performance so significant is necessary since the imbalance issue would also degrade the SL. 
3. Clarity: For the algorithm section, I consider giving more proof in detail is better. The analysis of the two main contributions on the algorithm, gradients direction and diversity, is not very clear. 
4. Originality: this work builds upon prior solutions to improve the performance of active learning using semi-supervised learning in the downstream tasks. However, the novelty is limited. 


questions: 1. What’s the performance of showing in Fig.1? 
2. In the implementation part (the 288th  row), how did you randomly sample one image per class? OR that is the preprocess of selecting the labeled training data? 
3. The augment of “SL fails to extricate performance gains from newly acquired labels from the influence of newly acquired samples. In contrast, SSL trains a model on Z and the only difference in datasets used for training is the label, and performance improvements between rounds s and s + 1 reveal gains from newly acquired labels” is the key point in this paper. Intuitively, the data and label act simultaneously to improve the model's performance. Why did you mention that semi-supervised learning is better for evaluating the effect of the newly acquired labels? Could you please give more explanation?

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Semi-supervised learning and activa learning are two representative learning paradigms to learn with incomplete supervised data. The difference lies in that active learning relies on human knowledge. This paper argues that the enhancements of SSL performance could be used to evaluate active learning methods to measure their label efficiency. This is an interesting idea. The proposed method is based on a neural tangent kernel analysis, which evaluates unlabeled data based on how they would contribute to updating the training set's inclusion. This is technically sound. The experiments also demonstrate the effectiveness of the proposal.
strengths_and_weaknesses: Strengths:

1) The paper gives a new perspective on semi-supervised learning and active learning, and shows that the enhancements in SSL performance could be used to evaluate active learning methods. This is a new idea and could inspire subsequent researchers.

2) The proposed method is technically sound.

Weakness

1. The experiments could be further improved. For example, conduct experiments in more settings, such as imbalanced semi-supervised learning and open-set semi-supervised learning. Moreover, it would be better to report performance on more related methods.
questions: 1) Can the proposal be applied to more realistic SSL scenarios? For example, recently there have been many studies about open-set SSL and class-imbalanced SSL. Can the proposal be applied with these methods?
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes a batch mode active learning method for active learning. The active sampling criterion mainly leverages the gradient of the currently trained model with respect to both labeled and unlabeled data to sample the informative data batch for humans to label. A theorem is presented to show that if the proposed sampling strategy is ideally implemented, the model loss decay rate is subject to linear.
strengths_and_weaknesses: Strength:
The methodology section is well organized; the proposed method is clearly described and explained with the underlying motivation, making it easy to read and understand.
Weakness:
1. The proposed method can be seen as a variation of the gradient embedding-based active learning while the main difference is the gradient of the model output, rather than the loss function, is used to construct the active sampling function. However, I am not convinced that the uncertainty can be fully captured without considering the potential labels an unlabeled data instance could have during the sampling. The paper mentioned in multiple places that the gradient of f(x) contains the uncertainty of x. But the paper does not spare any effort to prove/demonstrate that. In fact, I believe the label-instance pair is the menial unit for informative measurement in active learning. Two identical data points with different labels(provided by a human annotator) could have significantly different empacts on the model training. But the proposed sampling method treats them as the same. I do not believe the proposed method can perform uncertainty sampling as claimed in the paper.

2. The batch mode sampling is impractical to me because eq3 suggests that the optimal batch needs to be searched within a solution space of combination(N, Q) where N is the pool size and Q is the batch size. For a practical active learning problem, N can easily exceed 10000, and Q is typically set from 50~500, making combination(N, Q) unreachable. The proposed batch solution in algorithm 1 only randomly tests m batches out of combination(N, Q)  solutions. It is almost impossible to have the optimal batch in that m test batches. No need to mention that each batch testing has O((T+Q)^3) complexity given the T is the current labeled data. 

3. The bound of the decay rate proposed in theorem1 is based on the optimal batches. I don’t think it will hold for most of the time since the actual batch mode sampling could not guarantee finding the optimal batch. 

questions: I am interested in how the label information omits in the loss decay bound in theorem1. Can you explain how the inequality in appendix line 16 is derived? I think that is where the label information disappeared in the loss decay.
limitations: None
review_title: null

Meta-review: metareview: This paper proposes an NTK-based active learning method that, when combined with SSL methods, performs well on a number of tasks.  There are several passages in the paper that reviewers feel need to be improved for clarity.  The authors respond to these issues, and I think the responses can be transferred to the camera ready.   

confidence: Certain
Decision: Accept