Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on a version of a continual learning problem where for each continual task a learner is given a set of labeled instance, and a set of unlabeled instances in which some are relevant to the task and some are not.  The authors make the claim that this version of the continual learning task is most realistic in practice and is not explicitly studied in prior work. The authors propose a method called URSL that 1) Trains a "reference" network using a contrastive objective on unsupervised data 2) Uses the reference network to select task relevant instance from the unsupervised data 3) Trains a separate supervised learning model using labeled instances, instances from a memory buffer, and instances determined as task-relevant (but used as negative samples).  The authors perform a series of experiments where the supervised labels are from one data set and the unsupervised data is from an entirely different data set.  URSL performs slightly better than most baselines when it comes to accuracy of the model after the final task is learned, but in some cases performs worse.

strength_and_weaknesses: Strengths
1. I feel the framing of the continual learning problem where many unsupervised instances are provided for each task is a practically useful framing of continual learning.
2. The ablation study is a necessary part of an evaluation for URSL as it is complex and includes a number of additional terms in the supervised learning objective.

Weaknesses:
1. Overall, the empirical results are not very compelling.  The proposed method is rather complex but simpler methods that use no unsupervised data are fairly competitive to the point where one method that uses no unsupervised data actually outperforms URSL.  It is hard to argue for the practical utility of URSL given the relatively low or nonexistent performance increase over some of the baselines.
2. I find the overall experimental set up is somewhat limited.  First, accuracy results are reported only for the final task.  In continual learning the goal is to learn as tasks are presented to the learner.  Accuracy of intermediate tasks is a relevant and important result to report in the main body of the paper.  Second, the continual learning task presented has many variants based on how much supervised/unsupervised data is given and what data specifically is given as unsupervised.  The authors explore only a limited set of these, so it is hard to really understand the breadth of applicability of URSL.  Even simplifying the experiments by synthetically including known task-relevant data at some percentage to the unsupervised data to confirm the selection criteria is finding that data to learn from would be enlightening.
3. URSL borrows considerably from prior work.  As far as I can tell the basic procedures used to train each model are taken from prior work (namely SimCLR and Co^2L).  The main contributions is the method for sampling task-relevant features for Co^2L for the supervised model and two additional objective terms for knowledge transfer.  Without much principled justification (though some intuitive justification is provided), I do not find this work particularly novel at a technical level.

A few additional comments:
1. This problem seems highly related to PU learning (Garg et al; NeurIPS 2021 for example).  It would be good to see it discussed even briefly.
2. I don’t understand the intuition behind how thresholds are set for selecting unsupervised instances.  The thresholds are set relative to a mean similarity.  This seems to result in cases where an entire batch could be dissimilar and still many instances can be selected.  Said another way, if one batch has a low mean similarity (many instances are dissimilar to the prototype) and another batch has high mean similarity (many instances are similar to the prototype), the same number of instances can be selected for both if their \eta parameters and variances are the same.  This seems counter-intuitive.
3. What does the following statement mean: “It is noteworthy that the representation space of the reference network is chosen for OoD detection since it provides better sample discrimination than any other representation space obtained by training over a small number of labeled samples.”? How can such a claim be justified?
4. Why two thresholds for selection?  Intuitively if an instance is deemed task-relevant, it should have a task-specific pseudo label, yes?
clarity,_quality,_novelty_and_reproducibility: Clarity - Overall, the paper is well-written an clear

Quality - There are no major flaws in the paper’s research quality besides those listed above.

Novelty - I find the overall novelty of the paper to be somewhat low (see above comments)

Reproducibility - Hyperparameter for experiments and other experimental details are provided in Appendix B.  Barring some bias from the selection of supervised samples to be included in the experiments, I feel this work is reasonably reproducible.
summary_of_the_review: While the authors propose to focus on perhaps the most realistic version of the continual learning problem, I feel that the proposed method is not strong in technical novelty and doesn’t introduce significant methodological or theoretic contributions, which makes empirical performance a focus.  Unfortunately, I also did not find the empirical results compelling by not performing consistently better than baselines where no unsupervised data is used.  For these reasons, I argue for rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose the URSL model “Unsupervised Reference network and a Supervised Learner network“, which consists of two parts: 1) The general task-agnostic reference network, which is responsible for absorbing information from unsupervised data in the environment, and 2) the learner network, which is designed to capture knowledge from a few supervised samples while it is also guided by the reference network. They utilize contrastive representation learning as a unified approach for training both the reference and the learner networks, which allows information to flow between these networks. They also combine with knowledge distillation techniques applied in the representation space to take advantage of unsupervised samples.
strength_and_weaknesses: Strength:  
- The paper is well organized and well written.
- The proposed model performs well compared with STOA. The experiments are fair and well executed. 

Weaknesses:
- The novelty of the proposed approach is limited, most of the contributions are based on previous work such as: the SimCLR (Chenet al., 2020a) loss function, and an instance-wise relation distillation (IRD) loss to transfer knowledge from the previous time step to the current model (Cha et al., 2021).
- In (Further Experiments section 1.1) the authors said “In this experiment, the reference network is first pre-trained with all unsupervised samples before starting the learning of the first supervised task. Next, the reference network is frozen, and its parameters are maintained throughout the entire learning procedure.” Does this apply to avoid catastrophic forgetting?
- How does increasing the number of stored samples in the memory affect the performance? 
- It would be nice if the authors can discuss the transfer and ability to scale to streams with a hundred tasks for the proposed model.
clarity,_quality,_novelty_and_reproducibility: This is a well-organized paper and the proposed approach is intuitive to follow. My concern is related to the novelty of the paper which seems incremental and based on previous work heavily. The authors provide different experiment settings that outperform other benchmarks.

summary_of_the_review: This is an ok paper that I feel is not strong enough. As I mentioned, my concern is related to the novelty. I like that the authors provide extensive experiments with different settings. Some discussion could be explained in more detail such as: if the model overcomes catastrophic forgetting by nature “the way it is constructed”, and the transferability and scalability of the model to a large number of tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new continual learning algorithm that exploits unsupervised data in the environment. The model consists of two main parts: (i) reference network that learns knowledge from unlabeled data, and (ii) learner network for solving specific task of interest by utilizing labeled data. The reference network is trained using self-supervised loss function and then out-of-distribution detection technique is applied to segregate unlabeled data. The learner network is trained using three loss functions that transfer knowledge through time, reference and utilize supervised samples. The method is tested on CIFAR-10, CIFAR-100 and Tiny-ImageNet datasets. It is shown to outperform existing methods. 

strength_and_weaknesses: Strengths:
- The paper is well organized and well motivated.
- The ability of continual learning methods to utilize unlabeled data is important and the proposed method solves the problem in an effective way. The proposed approach based on reference and learner networks is interesting.
- The experiments are well designed and the method achieves large improvements over existing methods. Moreover, ablation study shows that each of the components of the objective function is essential.

Weaknesses/questions:
- The method has many hyperparameters and systematic analysis of the robustness to their selection is missing. It is not clear how to select these parameters.
- Some parts of the work need to be better explained. All parameters defined in the equations need to be defined which is not currently the case. The intuition behind each part of the objective function should be presented.
- Are there any assumptions on the proportion of the in-distribution compared to out-of-distribution unlabeled samples? Can the performance be degraded if unlabeled data is very different from the data that the model was trained on?
- Is the model applicable to large-scale datasets such as ImageNet? How does it compare to baselines in terms of scalability?

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clearly written and the overall quality of the paper is high. The proposed approach is interesting, original and effective.


summary_of_the_review: Overall, this a good paper and presents an interesting and effective method for continual learning by exploiting unlabeled data in a realistic setting in which unlabeled data can contain out-of-distribution samples. The method outperforms baselines by a large margin.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a novel setting for continual learning called OSSCL, which assumes that the agent has access to a large number of unsupervised data in the environment with some of which are relevant to tasks. To solve the proposed task, the author introduced a dual network, consisting of a supervised learner and an unsupercised reference network. The dual model can effectively utilize both supervised and unsupervised samples in the data. Different elements in the model are evaluated by ablation studies. The proposed model also outperform some other SOTA continual learning models.
strength_and_weaknesses: Strength:

The paper is clearly written and easy to read. The experiment part is solid and the idea of using a reference network from the environment is interesting. Each part of the losses the author introduced in their model is discussed by ablation studies. 

Weaknesses:

1, Many of the components in the network training, such as the designing of loss functions, are borrowed from previous studies, which makes the work incremental. The authors should highlight the novelty they made compared to previous studies, instead of creating a problem (designing a paradigm, OSSCL) which has not be investigate previously, but only studied by the authors themselve. 

2, In the experiment, the main and the perioheral dataset do share some underlying common distribution, and hence the out-of-distribution samples are not truly out-of-distribution samples. The authors should justify this point. 

3, Citations need to be checked. Some of the citations are duplicate. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The idea of the dual network is not novel but the proposed OSSCL setting could be interesting. 
summary_of_the_review: The model itself is based on many previous studies but the performance seems good, compared to previous models run by the authors themselves.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Open-Set Semi-Supervised Continual Learning, where the experimental setting is essentially a class-incremental learning scenario with unlabeled data from both in- and out-of-distribution. The proposed method consists of two components: the unsupervised reference model learns with the SimCLR loss and in charge of training representation and thresholding unlabeled data, and the supervised model learns with the supervised contrastive loss, previous model, and the unsupervised reference model. Experimental results show the effectiveness of the proposed method in small image benchmarks.
strength_and_weaknesses: Strengths

+ The proposed scenario is interesting.

+ The proposed method outperforms other methods in most cases.

Weaknesses

- It is hard to say the authors introduce a new scenario on continual learning. To me, the authors are just re-branding the setting proposed in prior works. Please make a more thorough comparison with prior works, [Lee et al.] and [Smith et al.] on the scenario. The authors have not explicitly claimed novelty on the scenario, but writing in abstract and intro is somewhat misleading, if they did not mean to claim the novelty.

- In fact, I could not see much difference on the setting from the prior work from [Lee et al.]. Unless specifically focusing on few- or low-shot setting, limiting the number of training data per class would not be a concern in the real-world setting. Rather, the scalability of the method would be more interesting and important. Based on the experimental result on the first few tables, the scalability with respect to the dataset size is partially confirmed, but the largest setting is still too small; only 10% of CIFAR. However, based on the experimental result on Table 14, the proposed method is not scalable with respect to the model size.

- Similar to above, I could not find the reason why the number of unlabeled data should be limited to only 9k per "time step" (this is confusing, "time step" sounds like one iteration or even smaller unit. I personally think "stage" would be a better term). There are plenty of unlabeled data in the open world, and there is no reason to randomly sample only 9k of them. Indeed, based on Table 13, the proposed model seems not scalable much with respect to the number of unlabeled data.

- Ablation studies on the hyperparameters are required. For example, the choice of $\eta_{id}$ and $\eta_{pl}$ seems critical to the performance of the proposed method.

- Training time is different over methods. Are authors sure if all methods are converged or early-stopped to show their best performance?

- $L_{KD}$ takes $\theta_t$ as a fixed set of parameters but missed.

- FixMatch and other few papers appear multiple times in the reference section.
clarity,_quality,_novelty_and_reproducibility: Writing is clear and well-written in general. The novelty of the proposed scenario is not clear. The proposed method has a limited novelty, as it is combination of existing methods, but their combination produces a good performance in their experiment. I believe this work is reproducible with the provided code.
summary_of_the_review: I appreciate thorough experimental results, but the proposed scenario and the experimental setting do not seem really match, and ablation study on the hyperparameter choice is not enough. I think this is a good paper if they limit their scenario to low-shot learning. Please answer my concerns above.

**post-rebuttal**

After reading the rebuttal, I decided not to change my rating, as I cannot agree with the validity of the proposed scenario.

The authors claimed that "The difference between OSSCL and the setting of Lee et al. [1] is that their setting is fully supervised, which makes the setting unrealistic."

I don't specifically agree with this argument. Once you incorporate unlabeled dataset, there is no significant difference between fully-supervised and semi-supervised, as we have both labeled and unlabeled data in both cases.
The proposed setting assumes that a portion of unlabeled dataset is guaranteed to share the same data distribution with the labeled dataset, which is more unrealistic to me. Rather, exposing a large unlabeled dataset to each time step of continual learning is more realistic. [1] used 80M Tiny Images as an unlabeled dataset, which is the source of CIFAR datasets. This means that unlabeled data in their setting includes data sharing a similar inductive bias to CIFAR in a natural way.

The scenario of [2] makes sense in some cases, as they limited the setting of each time step to the conventional semi-supervised learning; as authors said in the rebuttal, "unlabeled data at all time steps are drawn from the same dataset as the labeled data had been drawn from" in [2]'s setting.

However, this paper distinguishes main-labeled as labeled dataset and a union of main-unlabeled and peripheral (OOD) datasets, but at last, we end up with having a pair of labeled and unlabeled dataset, which is essentially not different from [1].
But the size of both labeled and unlabeled dataset is significantly smaller than [1], so I think the experimental setting in this paper is somewhat toy-ish and seems not scalable.

Again, I could not find the reason why we need to use "only 10% of CIFAR samples" as a labeled dataset, to make a portion of unlabeled data to share the same data distribution with the labeled dataset.
This not only makes the setting unrealistic, but also toy-ish, as we have a very small number of 32x32 training data per time step.

Also, after reviewing the paper again, an additional comment on Table 3 is that, the overall performance is too low, so I do not think the comparison is so meaningful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on a version of continual learning where for each task a learner is given a set of labeled and unlabeled instances, and proposes a method called URSL to tackle this problem. The strengths of the paper include its framing of the continual learning problem and the design of experiments. However, the paper has several weaknesses, including limited and unconvincing empirical results, a lack of justification for its technical novelty, and a lack of discussion of related work. Additionally, method for setting thresholds for selecting unsupervised instances, among others, is not well explained and need to be further clarified. Overall, all reviewers recommended rejection therefore it cannot be accepted at this time.
justification_for_why_not_higher_score: No reviewer recommends acceptance
Decision: Reject