Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Unsupervised Domain Adaptation (UDA) is studied in a task sequential continual learning setting. The source domain is assumed to have supervised data and several target domains without labels are being sequentially adapted to. An algorithms is engineered for the specific case of supervised classification source datasets which can be neatly clustered in some high-level representation; then, learning on related (target) datasets which can be classified in the same (source) classes can be done sequentially and without the use of labels. Catastrophic forgetting is mitigated using a standard experience replay approach, but also through the assumptions of the conceptual framework, i.e. strong overlap between source and target classes, which is known to be the easier setting in CL works. Empirical validation is done primarily in the using classic domain adaptation benchmarks, with the difference of using a single source domain for multiple target domains in a sequential fashion. The theoretical analysis and ablation study round up the work, including a discussion of algorithm hyper-parameters.
main_review: The novelty of the approach is reduced in terms of continual learning research, especially due to assumptions about source and target datasets, hence I need to primarily assess the merits of the work based on domain adaptation literature. While labels are not used for the target datasets, the assumption of identical class set is very strong. 

The empirical evaluation datasets used here, with domains in the hundreds of examples, make it difficult to imagine what the method would do on more realistic current image data streams. Real datasets are not this neat and definitely not this small, even supervised datasets. It is also hard to separate the improvements due to ImageNet pre-training and those due to specific domain adaptation algorithms. I do not believe it is enough to say that this is what domain adaptation works use, but it may be necessary to educate the readers (and indeed the reviewers) on why these datasets are still challenging and why algorithmic improvement is needed; for example, with large scale source dataset learning (even unsupervised source learning with e.g. SimCLR(v1 or v2)), freezing such models and learning linear output layers is enough for learning many other datasets. Why do we need to keep learning from small source datasets? Why not focus on domains where there is no choice but unsupervised domain adaptation?


I need further clarifications for the following:
* What is the upper bound of performance? Multitask training on both source and target domains using target labels and an empirically weighted joint objective?
* When is "aligning internal distributions" possible or desirable? Many datasets have very different classes, e.g. MNIST and Flowers. Is aligning distributions useful there?
* How robust is the method when there is data imbalance between source classes in target datasets?



Misc
* Could you please explain what is meant in rows 30-31? Why are existing UDA algorithms not suitable for continual learning?
* Row 41: modal or model?
* Row 42: could you please explain what is meant by coupled learned distributions?
* Row 195: Damian or domain?
limitations_and_societal_impact: I do not believe the discussion of limitations is sufficient. I would suggest improvements and clarifications along these lines:
* Consolidating internal distributions from datasets which are very different seems counterintuitive; even if the target dataset is a subset of source dataset, say source=ImageNet, target=Dogs++, there are at least 100+ overlapping classes, but it is at least possible that aligning distributions may lead to problems. This should be discussed in the main text.
* Even if continual UDA is possible, it is not entirely clear whether it is needed or beneficial for performance improvements compared to target task specific  source -> target UAD experiments with tuned hyper-parameters.
* Self-supervised learning on data rich source datasets (e.g. ImageNet++) seems to be a strong alternative, certainly for the datasets where the proposed method was demonstrated; I can agree that there are cases where such transfer is not helpful, e.g. (I suppose) skin cancer classification, but such domains are not explored.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper aims to address an unsupervised domain adaptation (UDA) in the continual learning (CL) setting, specifically the sequential unsupervised domain adaptation task. The learned internal distribution estimated as GMM are consolidated using pseudo-rehearsal to mitigate the domain shift and catastrophic forgetting.
main_review: Clarity: The paper writing and formatting is clear and well organized. However, some grammatical errors and typos are found, e.g., line 72, “The challenge is CL is ...”; line 75, “Our goal is update ...”; line 134, duplicated “form”; line 196, the two sequential tasks are exactly the same. Also, the mathematical notations, especially super/subscripts, in the equations and algorithms have typos and sometimes are not consistent, which cause confusion and need to be corrected, e.g. in Eq(3), the superscript for $z_i$ should be $p$ instead of $t$; $e^J_t$ in line 149 and $e^t_J$ in line 16; $t$ instead of 0 is used as superscripts in Algorithm line 7; Algorithm line 13, the pseudo-dataset size “N” should be $N_p$. Please check the grammar through the paper and clean up the typos in the mathematical notations. 

Reproducibility: All the experiments are using public datasets, and the code and experiment settings are provided in the paper and supplements, so the proposed method and experiments are expected to be reproducible. 

Originality: The task of the paper is a combination of UDA and CL, and is formed as a sequential UDA task, which is a novel task setting in domain adaptation. The novelty of the method is the internal distribution estimation and pseudo-dataset sampling. However, the loss function and pseudo-rehearsal are existing techniques in UDA and CL and have limited novelty. 

Significance: Since this task is a novel setting, limited horizontal comparisons are made between the proposed method and other existing methods. From the results in Table 1-4, the proposed method is at least comparable to other classic UDA algorithms. But since there’s no comparisons with other existing continual learning methods, it’s hard to say whether this method has significant advantages in CL. Moreover, the accumulation of memory buffer limits the proposed method being used in CL in practice, since in reality, lifelong learning will take unlimited data, but you cannot accumulate unlimited representative data in the memory buffer. 

Quality: The sequential UDA task is novel, and the challenges such as catastrophic forgetting and domain shift in CL are mitigated from the learning curves in Fig.2 as well as the visualization in Fig.3. The proposed algorithm and loss function are based on existing techniques and have both theoretical and experimental support. 
However, I think several important questions are not fully explained in this paper: 
1. Although the motivation for improving the domain shift in CL with unlabeled data is valuable and significant, the sequential domain adaptation in the problem statement is simply an ideal situation and not a real task in practice. It’s very hard to find such a situation in real life where multiple domains are learned separately and sequentially. In most CL cases, after initial training on labeled source domain, the upcoming continual unlabeled data points are sampled from the source domain and other target domains in random order and are always mixed together, instead of the training setting in the paper, where only one target domain is trained at each time step and no data points from other domains are included. 
2. The paper does not explain how the internal GMM distribution is updated each step after initialization from the source domain. What data samples are used for updating the GMM at each time step? Pseudo-datasets or unlabeled data points from the target domain? Also, in the Algorithm 1, for $t=1$, how to get $p^1_J$ for pseudo-dataset sampling in line 13? I assume $p^1_J$ at $t=1$ should be the same distribution as $p^0_J$ for the source domain. 
3. Memory buffer are used and accumulated to store the representative data points from each domain. However, since lifelong continual learning takes unlimited data, the memory buffer will also become extremely large after longtime accumulation. Thus, the memory buffer strategy is not preferred in practice. Also, the paper stores 10 samples per class per domain, but how is this number determined? Since this number is quite important for the performance (obviously larger buffer will have better results), I think some explanation is needed. 
4. Although horizontal comparisons are made with other UDA methods in two-domain setting, however, since the motivation of the paper is also about improving CL method, I think some comparisons with other classic CL methods are needed in order to demonstrate the advantage of the proposed method for lifelong continual learning and sequential domain adaptation. 
5. As Results explained, “A balanced dataset makes the empirical distribution a less biased approximation of the real distribution.” as shown in Table 2. This shows a limitation of GMM internal distribution estimation: this kind of distribution estimation is sensitive to class-wise balance. So I’m a bit concerned about the performance of the method on unbalanced datasets, which is always the case in practice. 
6. Two loss terms are added from Eq(3) to Eq(4), however there’s no ablation study for each loss term in the experiments. More ablation study for the loss is definitely needed in order to show the advantage of adding memory buffer terms in the loss function. 

Despite the above questions, the paper presents a valuable task combining both UDA and CL, and domain shift is the main challenge for lifelong unsupervised learning which is mitigated in this paper. So I think the paper lies on the borderline and look forward to the authors' response.
limitations_and_societal_impact: The paper mentions some limitations in the results section. 1. Because of the distribution estimation, the algorithm works well on balanced datasets but the performance degrades when the dataset is unbalanced. (shown in Table 1-2); 2. As mentioned in line 266-268, the method has a possible improving direction to benefit from class-conditional alignment techniques.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel method for unsupervised domain adaptation in continual learning setting using only unannotated data. The proposed method models the internally learned distribution of initial annotated source domain as multimodal distribution and updates it using the learned distributions of subsequent unannotated target datasets to couple all the domains together i.e., consolidated internal distribution. 

To minimize catastrophic forgetting, the proposed method also performs experience replay during domain adaptation on target dataset using important representative samples of all historical tasks.
main_review: Originality:

1. This work proposes a novel method which combines Unsupervised Domain Adaptation and Continual Learning methods in a joint learning framework via consolidated internal distribution. 

2. In terms of novelty, the proposed method has the following strengths:
> - Creating internally learned distribution using initial annotated source dataset as multimodal distribution and consolidating it over a sequence of subsequent unannotated target datasets via domain alignment in shared embedding space using Sliced Wasserstein Distance (SWD) metric.
> - Identifying important representative data points for each historical dataset and using them during current dataset's training for knowledge retention of past datasets/domains and to minimize catastrophic forgetting.

---

Quality:

1. The motivation is well founded and the claims are sound.

2. Theoretical formulation and analysis are done in detail and look fine except for the comments below.
> - Step-19 in algorithm 1 denotes update of GMM parameters using newly learned unannotated target dataset $t$, but equation 2 only explains the initialization of GMM parameters using initial annotated source dataset. Therefore, there is a need of another equation for updating GMM parameters using target dataset $t$.
> - See comments regarding inconsistent use of notations below.

---

Clarity:

1. As the proposed method deals with unsupervised domain adaptation and continual learning for classification tasks only, it would be appropriate to mention this in the introduction itself before "costly and time-consuming manual data annotation" statement.

2. In Figure 1, if equations are also referenced in the steps (wherever applicable) as done in algorithm 1, then it should enhance the understanding of the proposed method.

3. A table of notations should improve the readability.

4. Inconsistent use of notations:
> - $N$ is used to signify the number of samples in annotated dataset $D_S$, but $M_t$ is used to signify number of samples in unannotated dataset $D_T^t$ (in line 78).
> - Pseudo-dataset samples are denoted with $z_i^t$ in line 116 and with $z_i^p$ in equation 3 and $z_i^{p,t}$ in algorithm 1.

5. Typographical errors:
> - Line 35: forgetting the CL literature -> forgetting in the CL literature
> - Line 41: update the modal -> update the model
> - Line 72: challenge is CL -> challenge in CL
> - Line 73: non-stationery -> non-stationary
> - Line 75: is update -> is to update
> - Line 134 form form -> form
> - Line 195: Damian -> Domain
> - Figure 2 caption: (a,c) -> (a,e)
> - Table 1 caption: MINIST -> MNIST

6. Methodology:
> 1. As selection of important representative samples for experience replay is done based on the distance to cluster mean in input representation space ($x$) and not in embedding space ($z$), using model prediction confidence scores in addition to distance to cluster mean might help in selecting better representative samples for experience replay.
> 2. While updating GMM parameters, updating parameters using all the samples of target dataset in embedding space might be sub-optimal due to low confidence model predictions. So, it would be interesting to see how model performance varies if parameters are updated based on the samples with model prediction confidence score above a certain threshold hyperparameter.

7. Empirical Evaluation:
> 1. In Figure 2, accuracy of base model trained on each dataset of UDA sequential tasks as initial source dataset can be added and compared with the learning curves of sequential UDA tasks to study the tradeoff between knowledge transfer from past domains and knowledge retention of past domains i.e., minimizing catastrophic forgetting.
> 2. As equation 4 does not contain any hyperparameter to control the intensity of minimizing catastrophic forgetting, the only control parameter is the number of samples $N_b$. So why store only 10 samples per class for experience replay? Is it supported by any ablation study?
> 3. There is no clear description of "Source Only" method shown in all the Tables. If "Source Only" method is training of base model using only initial annotated source dataset then, Figures 2(a,e) both suggest model accuracy after training 100 epochs on source dataset $S$ is around 90. So, why "Source Only" performance in Table 1 in $S \rightarrow M$ column is only 67.3 which is far less than 90?
> 4. Similarly, in Table 2, why "Source Only" performance so much different in columns $C \rightarrow I$ and $C \rightarrow P$ with the same source dataset $C$? However, performance of LDAuCID method on target datasets is similar with columns containing same target datasets.
> 5. A clear indication in Figure 3 (left) regarding which row belongs to which timestep would improve understanding better.

8. Evaluation Setup:
> 1. Why chose max epochs as 100 per task as 100 epochs might not be enough for convergence on initial annotated source dataset which builds the internally learned distribution in embedding space?

---

Significance:
	
1. Evaluation setup is comprehensive and results show significant improvement in 2 tasks out of 4.
2. This is a significant achievement given the fact that this is the first algorithm which implements Unsupervised Domain Adaptation in Continual Learning setting.
3. Evaluation datasets, tasks and base models have been selected based on the related works in UDA literature.
4. To keep the evaluation fair with other UDA algorithms without continual learning, pairwise tasks have been used.
5. Although, flat learning curves in Figures 2 (c,d,f,g,h) of the initial source dataset during adapting model on subsequent target datasets might indicate that the algorithm puts more weight on minimizing catastrophic forgetting. Hence, there might be a need for a hyperparameter apart from number of samples $N_b$ to control catastrophic forgetting during domain adaptation on subsequent target datasets.

--

Updates: 

(24-08-2021): increased score from 6 to 7.
limitations_and_societal_impact: Addressed by authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a problem formulation at the intersection between continual learning (CL) and unsupervised domain adaptation (UDA). It introduces a method to perform UDA throughout the lifespan of a model, mitigating catastrophic forgetting of previous information. The proposed method is validated on benchmarks proposed ad-hoc for this problem, as well as on standard UDA benchmarks - showing that is can compete with state-of-the-art approaches even in standard settings.
main_review: This is a well written paper on an important topic, that has not received the proper amount of attention yet (CL+UDA). The proposed solution builds on standard techniques typically used in UDA (minimizing distance between domains in the space of learned representations) and in CL (pseudo-rehearsal): they are merged in a very reasonable way, into a method that is empirically shown to perform well on sequences of tasks - where by "task" we intend "UDA problem".

I liked the idea of using GMM for pseudo-rehearsals, with respect to related work on CL that use more complex strategies based on VAEs/GANs: it is cleaner, simpler to implement, and it fits well with the Mean-of-Features memory approach that is used.

My main criticism is that other works have studied CL+UDA problems before, and this work claims to be the first of its kind (I'm reporting the main references below). In particular, [A] proposes a solution that is very similar to the one proposed in this paper. It is originally devised for domains that shift gradually, but it could be easily applied to the benchmarks studied in this paper. Can the Authors comment on this? While [B] is presented as a method for semantic segmentation, the proposed pipeline seems to be fully generalizable to classification problems.

Minor: in the Digits experiment, SVHN is never used as the final domain. MNIST $\rightarrow$ SVHN adaptation is very challenging in UDA, hence it would be interesting to see what happens when SVHN is the final domain in a learning sequence.

I enjoyed reading this paper, and I believe it will help bringing attention to a research direction that is currently a narrow niche, CL+UDA. I did not give a higher score because the related work section - and, in turn, comparisons and discussions - does not include the few works that have addressed the problem tackled in this paper. I look forward reading the Author response on this point, and updating my score accordingly.

References for related work section ([A] and [B] seem the most relevant to me, but the others can further enrich the section)

[A] Bobu et al., Adapting to Continuously Shifting Domains, ICLR Workshops 2018

[B] Wu et al., ACE: Adapting to Changing Environments for Semantic Segmentation, ICCV 2019 

[C] Wulfmeier et al., Incremental Adversarial Domain Adaptation for Continually Changing Environments, ICRA 2018

[D] Porav et al., Don't Worry About the Weather: Unsupervised Condition-Dependent Domain Adaptation, ITSC 2019
limitations_and_societal_impact: The theoretical analysis provided hints on the conditions in which the proposed methods will work. Societal impact is not discussed but I agree with the Authors that there is not much to discuss in that regard, for what concerns the submitted manuscript.
review_title: null

Meta-review: The reviewers identified a number of strengths and weaknesses in this submission. The authors response led to a detailed discussion, over the course of which some of the reviewers increased their scores. Ultimately, three of the four reviewers recommend acceptance, with the four still being in the borderline range. I see not reason to overrule this vote and therefore recommend acceptance.
Decision: Accept (Poster)