Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use data programming to address the semi-supervised continuous learning, achieving the performance close to the fully supervised continual learning.
strength_and_weaknesses: Pros:
* They consider a more realistic continual learning with only a few labeled data, and adopt a novel technique, data programming, to annotate these unlabeled data.

* From the perspective of engineering, they greatly reduce the computational cost.

Cons:
* This paper seems to lack a type of important baseline. They can incorporate the existing semi-supervised learning training strategies into the continual learning methods to overcome the weak supervision problem. They need to justify the advantages of DP-SSCL over such straightforward solutions.

* I have a doubt about the meta initialization process. They use the task sequence as the prior knowledge to initialize hyperparameters and model architectures. However, for continual learning, could we obtain the full task sequence at once? If yes, why do not we use the full task sequence to perform standard supervised learning.

* For more severe scenarios, we could not obtain qualified WLFs. Under such a setting, whether DP-SSCL is still useful?
clarity,_quality,_novelty_and_reproducibility: These are all seem fine.
summary_of_the_review: Please refer to Strength And Weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to apply Data Programming to semi-supervised Continual Learning (SSCL). The core idea is to first generate weak labeling functions (WLF) with existing tools like Snuba, where the labeling functions are then used to pseudo-label the unlabeled data points that are eventually all fed into training the downstream continual learning models. Experimental results show that this pipeline leads to improved performance over existing SSCL methods and performs more closely to the fully-supervised continual learning methods.
strength_and_weaknesses: Strengths:
- The idea of applying data programming to continual learning is new. The connection and synergy between the two are natural, i.e., similar tasks in continual learning could share similar WLFs, making data programming specifically suitable for CL.
- By adopting data programming, the underlying CL methods do not have to be modified. This makes the proposed pipeline flexible and able to enjoy improvements as new CL methods are developed.

Weaknesses:
- While the combination of CL and data programming is novel, the technical contribution appears to be more limited. “Transferability” could be one aspect where the authors could more carefully look into to improve over previously used metrics like LEEP. However, this seems to be not studied in depth.
- Since the combination of DP and SSCL is new, Sec 5.3 should be one main focus of the paper, but the current set of experiments shown appears to be somewhat weak. For example, only one Transferability metric, LEEP, is tested in the experiments.
- The entire DP-SSCL pipeline requires many different hyperparemeters to be set, including $\phi$ and $\rho$. How are the hyperparameters selected in the experiments?
- Table 3 lacks one baseline that readers would be interested in seeing, i.e., CL with only the labeled data. It is also encouraged to show the dataset statistics, i.e., size of the labeled set, unlabeled set, test set, etc.
- In the introduction, it is mentioned that applying DP to CL can be more robust to distribution shifts. However, it is not clear to me why this is the case, and I also didn’t find any experimental results supporting this.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is overall clearly written.

- Novelty and Quality: Applying data programming to CL is novel on its own, but there are limited additional technical contributions. More comprehensive experiments and deeper analysis can make the paper better.

- Reproducibility: The authors released the code anonymously.

summary_of_the_review: I like the idea of applying Data Programming to SSCL, the current empirical results also show that this is a promising approach. However, I believe more comprehensive analysis on the proposed DP-SSCL pipeline can make the results stronger, for example, answering questions like what are the important aspects on selecting the transferability metric.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a semi-supervised continual learning technique which leverages data programming to probabilistically label unlabeled data. The paper leverages methods proposed in Snuba to automatically generate labeling functions for new tasks. 
strength_and_weaknesses: *Strengths*
1. The paper solves an important problem and has impressive results, beating previous reported state-of-the-art methods. 
2. I like the fact that the authors have taken time to analyze the memory and time complexity of their methods with baselines. 
3. The authors have a theoretical justification of the quality of labeling function proposed by the mechanisms used by Snuba under the continual learning settings.

*Weaknesses*
1. A quick google scholar search brings up a number of papers on semi-supervised continual learning [1, 2, 3, 4] which seem to not have been covered by the authors in either the literature review or as baselines. 
2. Comparison on Tiny-imagenet: Previous methods such as DistillMatch also compared their performance on Tiny-Imagenet. I believe that the paper would benefit from experiments on more datasets. 
3. I believe that the paper does not have extremely novel methodology. Having said that, I believe that the paper solves an important problem and does it well using practical methods.

References:
[1] Brahma, Dhanajit, Vinay Kumar Verma, and Piyush Rai. "Hypernetworks for Continual Semi-Supervised Learning." arXiv preprint arXiv:2110.01856 (2021).
[2] Luo, Yan, et al. "Learning to Predict Gradients for Semi-Supervised Continual Learning." arXiv preprint arXiv:2201.09196 (2022).
[3] Boschini, Matteo, et al. "Continual semi-supervised learning through contrastive interpolation consistency." Pattern Recognition Letters 162 (2022): 9-14.
[4] Ho, Stella, et al. "Semi-supervised Continual Learning with Meta Self-training." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and is of high quality. The authors have released their code and have used public datasets which ensures reproducibility. As mentioned above, the paper is not extremely novel, but solves an important problem and does it well using practical techniques. 
summary_of_the_review: I think the paper is strong and would benefit from comparison with some related work, some more experiments. I would also like the authors to highlight the novelty of their approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors have proposed a  Data Programming (DP) based Semi-supervised Continual Learning (SSCL)  framework which they call as DP-SSCL. Data programming (DP) is a known technique and its objective is to assign pseudo labels to unlabelled data points as a part of the semi supervised learning (SSL) process. The main idea of this paper is to first label unlabelled data points using DP and  then use those as additional (to already available labeled data) labels to further consolidate the model.

The proposed DP-SSCL framework follow the following steps (in the cascading order) including a feedback loop: (1) Meta (parameter) initialization, (2) Weak Labeling Function (WLF) initialization via knowledge transfer, (3) WLF training and pruning, and (4) Pseudo-labeler ensembling and continual model update. The proposed framework has been tested on  MNIST and CIFAR benchmark datasets and  compared against (a) some existing SSCL methods such as CNNL, ORDisCo, DistillMatch, and (b) Fully supervised Continual Learning with labeled data only.  A set of the performance metrics: 1) peak per-task performance, (2) final task performance, and (3) forgetting scores have been also researched and used in model comparisons.  Reported result is found to be promising. 
strength_and_weaknesses: Strengths and weakness:
Strengths: The paper is well written clearly covering the problem definition, the scope, the experimental and evaluation protocol, the evaluation metrics and the setup to compare with existing techniques in the field. It provides a set of thorough experiments that highlight the effectiveness of approach and the methodology. The reported results look not only better than existing approaches but sometimes close to the completely supervised setup (although tested on simple and small datasets). Also, DP-SSCL reports less memory and time complexity when compared to other existing approaches. In addition, the proposed framework is compatible with other Continuous Learning (CL) approaches so it can be easily evaluated and compared with others. 
 
Weakness: As reported in the paper, the Ensembling Methods and the WLF Transfer are two very important steps of the DP-SSCL framework. It looks the corresponding parameters of these two steps (section 5.3) were learned/chosen using the same data that has been used to report and compare (with the competitors) DP-SSCL performance in section 5.4. These underlying data should have been exclusive for steps 5.3 and 5.4, and therefore the results of section 5.4 may not be valid. It is suggested that authors redo these experiments by setting a disjoint set of data for these two steps. Same comments are applicable for the setup in Appendix B.

clarity,_quality,_novelty_and_reproducibility: The paper is well written with minor linguistic errors. The structure is well organized and the content is easy to follow. The semi-supervised continual learning idea has been presented clearly with minimum fuzziness. 

There have been a number of innovations especially the DP-SSCL framework itself including the evaluation and experimental protocol that has been followed (mainly to compare against available techniques in the literature). A number of novel ideas can be traced in the DP-SSCL pipeline  steps:  (1) Meta (parameter) initialization, (2) Weak Labeling Function (WLF) initialization via knowledge transfer, (3) WLF training and pruning, and (4) Pseudo-labeler ensembling and continual model update. 

As the code has been shared, it is expected the results can be reproduced although there has been some experimental setup problem identified as explained in the weakness section. 



summary_of_the_review: I have gone through the paper more than once including the appendices. Overall, the idea is quite sound, well articulated through the document. The experiment is thorough and the reported results are encouraging. 

There has been a major flaw identified in the experiment section 5 as some of the model comparisons were done using parameters that were selected based on the same data (see the weakness section). This might have over judged/scored the model performance. 

The authors are suggested to redo their experiments with a proper setup for a valid comparison. I think this work has some value if this problem can be resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper has been assessed by four knowledgeable reviewers. It proposed an application of data programming to continual learning scenarios, which is novel, but the technical and experimental infrastructure surrounding the concept, as it stands right now, does not appear sufficient. This limits the perceived novelty of the work as well as it diminishes strength of the technical contribution. Three of the reviewers rated this paper below the threshold of ICLR acceptance while on voted to accept it as is. This work has some merit but it is not ready yet to be published at ICLR.
justification_for_why_not_higher_score: Most reviewers expressed reservations about various aspects of empirical evaluation part of the paper.
justification_for_why_not_lower_score: n/a
Decision: Reject