Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work introduces a new active learning technique, that aims to use the model uncertainty over the pool of unlabeled data, after the model is retrained on a new additional point on the pool (so not a myopic strategy - but they call this a "look ahead strategy"). To make the strategy computationally feasible, approximations based on the neural tangent kernel are used. 
strengths_and_weaknesses: 1) The work introduces a technique based on an idealized criteria, which is then approximated to make it computationally feasible. However, the approximations are changed in such a way, that the idealized criteria is not being approximated anymore. This weakens the story, as we cannot explain what the active learning strategy is doing, and why it works. The performance boost seems now to come from somewhere else, but the reasoning is unclear why this is the case. Various heuristics are implemented but there is not a careful ablation study to verify where the performance gains are coming from. 

2) It is claimed the proposed look ahead strategy MLMOC improves matters. However, a detailed comparison with other already published lookahead strategies is missing, such as the original EMOC strategy. It is claimed that EMOC didn't work well, but no experiments are given to back up this claim. Similarly, several ad-hoc changes are made (line 208-210), which are claimed to "empirically not hurt accuracy" but this is not shown in the paper. Why are these results withheld? 

3) Some other claims that are made; which are not backed up by evidence: 
3a) "We also suspect that, when ... is already reasonably accurate, we can “trust” the most likely label more than we can trust accurate estimation of probabilities for low-probability losses, especially when training networks with..." (line 210).  Can we not check this?
3b) "These classes of models, however, tend to not work as well as modern neural networks, meaning these approaches are outperformed by simpler acquisition functions on stronger models." Can we check this with a Gaussian Process classification baseline with a GP acquisition strategy? E.g. compare with [5] or [6]? (line 38)
3c) "Sequentially adding true labels of new data gives performance substantially better than more common batch setups" [line 58] Source? I think actually, the batch strategies can sometimes outperform sequential strategies, especially for myopic strategies that do not take diversity into account, while a batch adaptation usually does check this. 

4) Since the approximation allows (I think) various variants of the EMOC criteria, I was expecting to also see a comparative study on this aspect, for example GP-impact, EMOC, and some of the other baselines in from [5,6]. 

5) The experimental setup is not carefully described and not using the standards of the active learning field (e.g. check Settles). Check q9 for more detail. 

6) Strength: the approximations introduced to make look-ahead strategies feasible using the Neural Tangent Kernel, which I think is an important contribution, since it can be used to design various new active learning strategies. 

questions: 7) In Figure 1b; how can it be that GP (Random) is performing better than GP? This must be wrong... It seems like the baseline was modified - were these modifications sensible? Would it be possible to run the experiments with the original GP for the classification tasks? I actually wonder how that would compare... It is 

8) In the figures, there is a shaded region indicating variability, but it is not explained how this is calculated or what this is? How many runs were used? Is this the standard error or standard deviation? Are the differences between the methods significant?  Since it is claimed that "existing look-ahead strategies by large margins," I would expect to see some significant p-values from a paired t-test comparing methods pairwise (as a posthoc test after a Friedman test). 

9) The query scheme, I don't understand... First of all, the figures mention "Cycles" - why not just indicate the amount of training data available? It makes it unclear. Second, the cycles seem only defined up to 4 or 5. What happens after that? This is not specified in the "query scheme" (line 280). On the y-axis sometimes is given "\Delta Accuracy"; what is that and how is that defined? 

10) The analysis and derivation of equations 6-8 seems to share many similarities with the derrivation of the EMOC strategy itself. What are the similarities? Was your work inspired by their work? Please indicate the similarities and give proper credit. 


limitations: Please see above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To reduce the computational cost during active learning process, this paper proposed an interesting active learning algorithm that utilizes look-ahead strategies by large margins and approximate neural network using neural tangent kernel. So that the look-ahead strategies that can be easily adopted in classical ml tasks like EER, EMOC, could also be implemented in deep learning tasks. The authors proved that the outputs of very wide NNs can be approximated using a linearized network wit neural tangent kernels. Compared with naive computation of look-ahead strategies, this approximation is of 100 times faster and has better performance.

strengths_and_weaknesses: Strengths: this paper is well-written with abundant proofs/experiments to theoretically/empirically demonstrate the effectiveness of their proposed models of look-ahead strategies with large margin.

Weaknesses: using NTK for approximation itself is not new in active learning, [1] use NTK approximation that construct a suitable neural embedding that determines the feature space in active learning with model selection. [2] use active learning to speed-up up the convergence for SSL deep learning algorithms, is also inspired by NTK, when the eigenvalues of the NTK are large, the convergence rate is faster. 

[1] Wang Z, Awasthi P, Dann C, et al. Neural active learning with performance guarantees[J]. Advances in Neural Information Processing Systems, 2021, 34: 7510-7521.

[2] Kong S T, Jeon S, Lee J, et al. Relieving the Plateau: Active Semi-Supervised Learning for a Better Landscape[J]. arXiv preprint arXiv:2104.03525, 2021.
questions: I have several questions about this paper:

1. Figure 2 with $\Delta acc$ is really hard to read, can author just show accuracy-cycle(labeling cost) curves?

2. For implementation of LL4AL, it is very strange that LL4AL perform so bad on SVHN, CIFAR10 and CIFAR100 datasets, although the author mentioned that this is because the unstably training during AL training processes, but according to [3], the LL4AL (named LPL in [3]) perform well on SVHN, CIFAR10 and CIFAR100 datasets with ResNet18 as basic learner. I wonder did the author correctly re-implemented the LL4AL with proper way, i.e., the original version <https://github.com/seominseok0429/Learning-Loss-for-Active-Learning-Pytorch> or the re-implemented version <https://github.com/Mephisto405/Learning-Loss-for-Active-Learning>.

**I will give score 4 first and I hope the author could answer the question 2 properly. I will increase my score if I can be persuaded.** 

[3] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.


limitations: The author addressed the limitations but I didn't find the discussions about potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a new method for approximating active learning acquisition strategies. Specifically, it uses the neural tangent kernel to approximate the result of retraining, namely it linearly approximates the network and mimics the new output of adding a new sample. Then it compares the network output without training on the sample (x) and the linearly approximated network with training on (x), and accordingly selects (x). 
strengths_and_weaknesses: Pros: 
Using the neural tangent kernel to approximate the result of retraining greatly reduces the training cost and improves the efficiency of active learning, compared with the previous one. 

The authors provide some experimental results to support its effectiveness. 


Cons:
Using the neural tangent kernel method to approximate a network is actually very hard. This is also mentioned by the authors. Although the authors use a local NTK which may improve the accuracy, I still think there is a big gap between NTK network and the vanilla one, since the width is not huge in practice. So how do the authors guarantee the quality of this local NTK? For example, do the authors have some theory to guarantee or have compared vanilla network output and this new NTK network output?

Efficiency: in algorithm 1, for each epoch, one needs to pass the whole unlabelled dataset to select proper samples. For each sample selection, one further needs to use eqn (12) to compute the NTK output. But when the lablled dataset is large, e.g. imagenet 1K, NTK kernel Theta would be very large. So computing its inverse may be expensive. In this way, it seems it is hard to scale this method to the large-scale dataset.

This labeled data plus unlabelled data is actually a semi-supervised setting. So it is necessary to compare with some classical semi-supervised methods, e.g. fixmatch, paws, etc. It is very valuable to see 1 ) whether there is performance improvement or drop compared with classical semi-supervised methods; 2) whether the proposed active learning can greatly improve the training efficiency of classical semi-supervised methods. For performance, it seems the results on CIFAR10 reported in this work is much lower than the vanilla fixmatch method.

questions: Please see the above questions. 
limitations:  the authors did not well discuss the limitations, and did not discuss potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an active learning method for neural networks that leverages neural tangent kernels to approximate the network. The proposed approach leverages lookahead to predict the expected NN output change when it is retrained with a specific input pair. The paper further proposes an acquisition function, Most Likely Model Output Change (MLMOC). The authors compare the proposed approach with state-of-the-art methods on three benchmark datasets for classification tasks and show that it either outperforms existing methods or performs equally well.
strengths_and_weaknesses: **Originality:** Related work seems to be adequately cited and the differences between the paper and previous work are explained.

**Quality:** The claims are somewhat well supported. For example, in line 210 it is stated that "empirically does not hurt accuracy" however I did not see a comparison between the acquisition functions.

**Clarity:** The paper makes a good job at introducing the active learning setting however it then starts getting confusing, particularly the introduction to the lookahead setting lacks details. For instance, in Section 3, the contribution and insights of the paper are too diluted with existing work. Specifically, having subsections would be useful to distinguish the different steps more clearly: 1) sequential training of the network (theorem 3.1); 2) approximating the retrained network. I felt that the part on "NTK approximations of retrained networks" after remark 3.2 was diluting your conclusions with those of related work. Here perhaps you could start by discussing the limitations/conclusions of related work, explaining how you build on them. 

**Significance:** It is likely that other researchers will use the ideas presented in the paper or build on them.
questions: **a)** Why is your acquisition function more suitable for a linearised network?

**b)** In algorithm 1 (you should add line numbers) you mention an oracle, but I don't see any mention to it anywhere else. Where does the oracle come from?

**c)** You mention lookahead but it is not clear to me if it is lookahead of: a) 1 point/label; b) n points/labels; c) 1 batch; d) n batches?

**d)** What is the meaning of T in algorithm 1?

**e)** Are there non-NN active learning baselines? How would your method fare against them?

**f)** Why do you only use MLMOC in Figure 1.b)? Why not test also with different acquisition functions? Also, in this experiment, why do you modify the models to "use an infinite NTK" (line 305-206) when you know that infinite NTK perform worse? You should aim to beat the strongest baselines, not the weakest.

**Comments:**
- Line 104: typo "the" also show => "they" also show
limitations: Authors address some limitations of their work (time complexity). Is this method only applicable to classification tasks? How does it generalize to regression tasks?
review_title: null

Meta-review: metareview: The paper presents an active learning method which selects samples from an unsupervised pool to be labeled, based on an estimate of the output changes if the selected samples were labeled (thus the name look-ahead). The estimated behavior of the networks after incorporating the selected samples is computed based on the neural tangent kernel approximation to wide neural networks. The authors provided theoretical results for the infinite width case, and also provided practical implementations with local linearizations (with efficient linear algebra operations, which is more straightforward). The method is compared with alternatives to show faster convergence to high quality final model. The authors shall take into account discussions to verify the approximation quality of linearization, and make more rigorous statements to avoid confusions. 
confidence: Less certain
Decision: Accept