Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a coreset selection framework, called RETRIEVE, that aims to improve the efficiency and robustness of semi-supervised learning (SSL).
The main insight that underlies RETRIEVE is that effective selection of a subset of unlabeled data - instead of using the entire unlabeled data - and training on such a carefully selected subset can enable current SSL algorithms to converge faster and significantly reduce the computational cost of the learning process.
The study empirically demonstrates that RETRIEVE can reduce the training time and improve the performance for out-of-distribution or imbalanced data for various SSL algorithms.

main_review: GENERAL COMMENTS

This study presents the potential benefits of employing coreset selection schemes in the context of semi-supervised learning.
Evaluation results based on a number of benchmarks show that the proposed coreset selection scheme, RETRIEVE, may potentially improve the performance of various SSL algorithms - especially, for out-of-distribution (OOD) data and imbalanced datasets.
However, the evaluation results are somewhat limited and further evaluations of RETRIEVE, as well as comparisons with other coreset selection schemes, would be needed to demonstrate the benefits of RETRIEVE in a more convincing manner.
There are also a number of ambiguities in the paper that need to be clarified.


DETAILED COMMENTS

1. Figure 1 is not very informative in its current form.
The authors may want to consider removing it and instead provide a better explanation in the main text or try revising the figure to make it more informative.

2. "Accuracy degradation" is not clearly defined and its use is a bit confusing.
For example, when the accuracy is improved (despite the speedup), it is shown as "negative" accuracy degradation.
It would be better to show the "accuracy" or "accuracy improvement" rather than "accuracy degradation" to make it easier to understand.

3. In the caption for Figure 2, does "Efficient SSL" refer to "Traditional SSL"?
In other places of the paper, the authors typically refer to "Traditional SSL" and "Robust SSL", and it would be better to use consistent terminology throughout the paper.

4. In lines 77-79, what does it mean that RETRIEVE "minimizes the performance on the labeled dataset"?
Please rephrase to make the meaning clear.

5. It is unclear what is meant by "Equation (5) can be solved efficiently using greedy algorithms [30, 31] with approximation guarantees."
What are the mentioned "approximation guarantees"?
This requires further elaboration.

6. In the performance evaluation results, only Craig was tested and compared with the proposed coreset section scheme.
To demonstrate the advantages of RETRIEVE, performance should be evaluated for other combinations of recent coreset selection algorithms (e.g., Bayesian coreset as well as other coreset selection schemes discussed in lines 129-137) with SSL.

7. It is mentioned (lines 287-289) that "for the Robust SSL scenario, we do not warm start with the full unlabeled set for training" and that this is "because training on an entire unlabeled set containing biases which instead hurts the model performance"
Please provide further insights underlying this statement.
Also, this sentence appears to be broken and needs to be rephrased.

8. Please clearly explain how the speed-up and the accuracy degradation were evaluated.
Furthermore, explain what the data points shown in the figures represent.

9. Despite the importance in the proposed greedy algorithm, connections with weak-submodularity and its implications are not discussed in detail (e.g., lines 366-367)
Please elaborate.

10. Is there a way to balance the trade-off between speed and accuracy when integrating RETRIEVE into SSL algorithms?


----------------------------

NOTE:

The original rating has been updated after reviewing the authors' point-by-point response to the above comments.



limitations_and_societal_impact: The authors discuss limitations of the current work and also discuss any potential negative societal impact in the supplementary material.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors propose a framework to select a subset of unlabeled data for semisupervised learning using a greedy selection approach.  The authors show the effectiveness of their method by extensive experimentaion on real world data sets. They show that their method gives significant speedups in traditional semi supervised learning tasks with very less drop in accuracy. For Robust SSL where the unlabeled data has out of distribution points or class imbalance , their method actually outperforms the state of the art with a significant speedup.
main_review: The paper proposes a framework to select a subset of unlabeled data for semisupervised learning. The subset selection and the learning task are done together by solving a bilevel discrete-continuous optimization problem. The authors show that the task of subset selection can be done using a greedy approach as the function is weakly submodular. They demonstrate the effectiveness of the framework with extensive experimentation

Significance:
The task of labeling data is very costly. Hence semisupervised learning is important . However the state of art algorithms for SSL are expensive and for Robust SSL even more so. Hence the problems addressed in the paper are well motivated.

Originality:
From the theoretical point of view, the paper is not that novel. The main theoretical contribution, I believe, is the framework itself and the proposal of the subset selection problem along with learning as a bilevel optimization and proving it weakly submodular. The proof of theorem builds up on existing techniques. Also traditionally the word "coreset" is used when there are some provable guarantees(in terms of sample size, approximation etc.). However in this paper either there are no such guarantees or at least they are not clearly highlighted in the writing.
That being said , the paper, does a very good task in the experimentation. The experiments are done on large real world datasets and also the subset is used with atleast 3 state of the art techniques. Most of the details and setup is stated clearly. The subset is compared with random sample, early stopping  and also CRAIG (which was proposed for supervised learning. Modifying that for SSL is also a minor contribution). Advantage is shown not only in terms of convergence time but also in tems of CO_2 emission.  Experiments for Robust SSL beat the state of art method both in terms of time and accuracy. 

Clarity and Quality:
I didnot go through the proof in details but it looks ok. I would suggest to highlight the theoretical contributions more in the writing and make them clear. Also in writing there is sudden change in tense (from present to past tense) on lines 266 and 268. In terms of empirical results this is a very good paper and will be of interest to ML community in general and specifically to the ML practitioners and ML engineers. Also the appendices are quite detailed and give a good overview of state of art SSL algorithms.

Question: Just as the authors have modified CRAIG for the semisupervised setting, is it possible to modify the CRUST algorithm used to work with nosiy labels ( Baharam et al Neurips 2020. ) for Robust SSL?
limitations_and_societal_impact: For limitations please refer to the above section.

I donot see any major potential negative impact of this work.

For improvement please highlight the technical challenges in theory and your theoretical contributions more in writing. Also would be good to give more details (related work) about traditional coresets and contrast your coresets with traditional ones.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work extends previous approaches for coreset selection to the problem of semi-supervised learning (SSL). Experimental results show the approach has good performance even in settings with data imbalance and out of domain training data

main_review: This work extends previous approaches for coreset selection to the problem of semi-supervised learning (SSL). Experimental results show the approach has good performance even in settings with data imbalance and out of domain training data

Originality/Significance: Straightforward application of previous ideas from methods like CRAIG and GLISTER to the problem of SSL. This may have impact in the SSL community but I think it's significance for data summarization/selection is rather limited.

Clarity: Well organized overall, good review of related work that unifies SSL algorithms under a common framework and distinguished the proposed approach from previous SSL methods

Quality/Methodology: Proposed method appears to improve consistently over previous work, but authors should compare against GLISTER and GRAD-MATCH

Questions:
- How would the method perform on data from non-image domains?

---
EDIT: The authors fully addressed my main questions about additional baselines and problem domains. Therefore I am increasing my score to 7
limitations_and_societal_impact: Limited discussion of limitations/societal impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes RETRIEVE to address the computational cost issue in previous semi-supervised learning (SSL) algorithms. The key idea is to train SSL on a subset of unlabeled data (coreset) instead of the entire unlabeled data. To do so, it must solve the problem of selecting the coreset efficiently and effectively. RETRIEVE tackles this problem by solving a two-level optimization problem such that the selected coreset minimizes the labeled set loss. This paper evaluates RETRIEVE on two settings---traditional SSL and robust SSL---and demonstrates its improvement in terms of speed, energy consumption, and CO2 emissions compared to prior work.
main_review: **Originality**: The proposed method is new. The novelty of RETRIEVE is to train on coreset of unlabeled data rather than on the entire unlabeled data. The challenge is then how to select the high-quality coreset effectively and efficiently. They use greedy selection and Taylor approximation. Generally speaking, the novelty of this work is incremental:
- The subset selection problem is NP-hard in general [1]. 
- The connection with submodular optimization has been demonstrated before [2].

[1] Balas Kausik Natarajan. Sparse approximate solutions to linear systems. SIAM journal on computing, 24(2):227–234, 1995.

[2] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950–6960. PMLR, 2020.

**Quality**: The submission is technically sound. The claims in the contribution list are well supported by theoretical analyses and empirically results.  I like the extra measurement in terms of energy consumption and CO2 emissions. The related work is adequately cited. 

**Clarity**: This paper is well-written and easy to follow. The problem is also well-motivated from the introduction by asking two questions. The experimental details are also very specific such that reproducing the results should be possible. 

**Significance**: Empirically, RETRIEVE outperforms prior work in two different settings. I would recommend the authors to add releasing code link in the revision.

Here are a few questions:
- Can RETRIEVE generalize to machine learning methods other than deep neural network? For instance, logistic regression, graident boosting tree? 
- If no to the above question, would it be a limitation of this work?



limitations_and_societal_impact: This paper does not discuss any limitation. I would recommend to add limitations in the revision, which definitely exist. The paper mentions positive societabl impact in terms of less energy consumption and CO2 emissions during training, which is good.
review_title: null

Meta-review: This paper proposes RETRIEVE to address the computational cost issue in previous semi-supervised learning algorithms. It is well-written and well-motivated. The proposed idea is incremental but technically sound. The claims are well supported by theoretical analyses and extensive experimental results.
Decision: Accept (Poster)