Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents, CLOPS, a new replay-based continual learning (CL) strategy for physiological signals.

The paper has the following merits. It is well-organized. I can understand its method, different settings of continual learning, and more ablation study on the model parameters. It did through experiments on the 3 datasets and get good results showing CLOPS beats naive baselines like multi-tasks learning, Fine-tuning as well as other CL methods like GEM, MIR.  

On the other side, I have the following concerns about the submission.

1. what is the task exactly?

Is cardiac arrhythmia label also time series? Does it have the same sampling frequency as physiological data?
In the Class-IL setting, are you doing multi-class classification or binary class? The notation is confusing. Is task [0,1], task [2,3] meaning binary classifications of class 0 v.s  class 1 and class 2 v.s. class 3? Then what does task [0-2] mean? It appears in the second paragraph of section 6.1.

2. not enough highlight for important parameters (storage and acquisition fractions, b, and a).

Storage and acquisition fractions, "b" and "a" are important parameters of replay-based CL algorithms. However currently, the definition of them is hidden in section 4 and figure captions. They are important parameters. The author even did an ablation study in section 6.4 on it. However, when I first read the paper, after reading the method section, I had no impression of "a" and "b".

3. not well explained experimental result. 

In Table 1, it shows Fine-tuning works better than Multi-task learning. It is a bit counter-intuitive to me since MTL has stronger supervision at all the time. Do there exist differences between task difficulty? It seems the task order influences the performance of fine-tuning according to the result in Table 1 and Table 9 in appendix G. Can the author provide any explanation?
In section 6.2, the author argues that CLOPS achieves an AUC after one epoch while fine-tuning requires 20 epochs to achieve that. First, I am not sure whether it is true since it seems the performance of the fine-tuning is not converged yet, its performance may still increasing. Second, clearly, CLOPS has a much larger performance variance than fine-tuning.  So I think the current result does not support that CLOPS is better than fine-tuning. 

4. unclear usage of task similarity.

In the appendix, the author shows the task similarity produced by the model. However, I am not sure what we can do with such task similarity. In the abstract, the author claim "this quantification yields insights into both network interpretability and clinical applications". However, in the paper, I can not see any arguments about what insight of network interpretability and clinical applications we can get.
review_title: interesting problem; unclear task description;

final_score: rating: 3: Clear rejection
review_content: ## Summary
This paper proposes a replay-based continual learning method and applies it to physiological data.
The proposed method consists of two heuristics to manage the replay memory:
- Learn a weight parameter (task-instance parameter) for each data point by minimizing a special loss function, and store the data with large weight parameters.
- Periodically compute epistemic uncertainty for all data in the replay memory, and replay the data with large uncertainty.

---

## Pros

This paper points out the importance of continual learning in the medical field.

---

## Cons

### Weak contribution

Since this paper deals with the continual learning of a specific type of data, I think there are two directions in which this study can be meaningful:
- Test various CL algorithms in the new domain, determine which method performs better than others, and analyze the reason.
- Use domain knowledge to design a new algorithm that performs particularly better on the domain.

However, I think this paper belongs to neither of the two. There is no domain-specific component in the algorithm, and only two baselines are compared.

If the method is general enough to be used in other domains, I think the authors should have also tested standard CL scenarios such as Split-CIFAR10 or Split-CIFAR100.

### Lack of theoretical justification

The proposed method is mostly heuristic and does not have a theoretical basis. Especially eq. (2) seems too arbitrary. Since this paper's empirical evidence is weak, I think a theoretical analysis is necessary to support the algorithm's efficacy.

### Vague and improper definition of time incremental learning

The authors seem to propose time incremental learning for the first time, but the definition is too vague:
> Time Incremental Learning (Task-IL) - the same dataset and prediction problem are used for each task, however the time of year at which the data were collected differs from one task to another. Such seasonality is most common in healthcare applications.

The authors should specify what changes over time. However, I think there is even a bigger problem:
- If the input distribution changes, it is not different from domain incremental learning.
- If the output distribution changes for the same input, it is not continual learning. The model *should* forget old tasks and adapt to new tasks.
- If both the input and output distributions change, it is class incremental learning.

Therefore, I think time incremental learning is not a novel branch of continual learning. The corresponding experiments should better be reformulated as domain incremental learning or class incremental learning.

---

## Overall evaluation

I do not think this paper proposes a novel idea with either a solid theoretical basis or strong empirical results. Therefore, I recommend rejection.

---

## Post rebuttal
During the rebuttal, the authors failed to handle the issues that I raised. Especially, the authors did not respond to my criticisms about the strange experiments. Therefore, I stick to my initial rating.
review_title: No theoretical basis, slack empirical validation

final_score: rating: 7: Good paper, accept
review_content: The authors propose a learning methodology designed to offset detriments to algorithm performance that arise when instances are not i.i.d (independent and identically distributed), focusing on cases in continual learning (CL) given physiological signals. They designed a replay-based learning method that handles an instance buffer using Importance-guided Storage and Uncertainty-based Acquisition strategies. They apply their method on Class, Time and Domain types of CL, and they introduce t-Step Backward Weight Transfer and  Lambda Backward Weight Transfer methods by which to evaluate their method. They conclude with two ablation studies to explore an explanation for their method’s performance and attempt to validate their hypotheses based on these studies.

Suggestions: 
* Make more explicit, in 4.1.2 (regarding equation 5), which type of task is being referred to, when comparing between prior and present tasks.
* If time permits, a figure for Experimental Results section 6.3 (to correspond to figures provided in 6.1 and 6.2) would be nice.
* A brief explanation should be included for the -1 in the regularization term in equation (2)
* In section 4.1.2, the need for a storage function was well motivated; however, the particular solution that the authors chose needs more motivation (and comparison with possible alternatives such as \beta^2. Not suggesting that this be empirically verified.)
* In Figure 2, SDs were calculated over 5 seeds. Was this low number due to the computational cost of experimentation?
* For a lay ML practitioner to appreciate the assertion made in section 6.5, "ST-elevation" needs to be defined more concretely.

review_title: a unique combination of applying Active Learning to physiological signals for CL

final_score: rating: 7: Good paper, accept
review_content: **Update**
I think the manuscript has been further improved now and I improve my score to 7. Also since I think results on public datasets in new domains other than images may be helpful for the wider research community. Keep in mind, as written below, that other reviewers more familiar with this research field may be more  able than me to judge the evaluation quality and whether the evaluation contains too substantial flaws to allow publication. 

**Disclaimer**
Upon reviewing the manuscript, I found I am unfortunately not well-acquainted enough with the research fields (active learning/continual learning) to be able to evaluate very well what this manuscript contributes to these fields. I hope my review can still be useful.

**Summary**
The manuscript describes and evaluates a continual learning algorithm that learns per-instance scalars beta that weight the loss of each instance. These betas are optimized together with all the other parameters, using an additional regularization term that prevents all betas to be optimized to zero. The betas averaged over time are used to decide which instances get into the replay buffer (easier ones are used). Additionally, to decide which instances  from the replay buffer to train on, Monte Carlo Dropout is used to estimate the uncertainties of the network on each instance and more uncertain instances are then more likely to be sampled for training. The approach is evaluated on three healthcare datasets, in either a setting where sequentially different tasks are learned by the network, the same task is learned with data sequentially used across time (affected by seasonal changes), or the same task is learned while sequentially shifting the domain/input distribution in a controlled way (different projections of same ECG signal). The proposed approach outperforms two baselines with regard to performance on previously-trained-on tasks. Ablation studies show the role of the storage strategy and the acquisition strategy while validating the interpretation of the betas as instance difficulties.

**Main Impression**
The approach is clearly described, it was fairly easy to understand for me, the writing is nice. The evaluation seems interesting; it would be more convincing to me if it was also evaluated on some dataset where published results of other works are available. Ablation studies are nice, the results of them could be discussed more.

**Major Points**

As written above, I would really like to see an evaluation of this approach where one can directly compare to published results by others. Reimplemented baselines always have a risk of being not tuned well enough etc. And as far as I could understand, there does not seem to be anything so healthcare-specific in the algorithm itself that comparison in other domains would not make sense, in case that is the problem.

I like the ablation studies and would have enjoyed more discussion of their results. 
- It seems for b=0.1 and b=0.25, random storage always outperforms CLOPS? Why? b=0.5 also mostly outperforms. 
- Similar, for a=0.5 and a=1.0, it seems that random acquisition mostly outperforms CLOPS, for a=0.25 it is still close, why?
- What happens for random acquisition and random storage? Was this tested?


 **Minor Points**

*Some parts of introduction I didn’t agree or find too broad*

“Deep learning algorithms operate under the assumption that instances are independent and identically-distributed (i.i.d.). “
-> not really specific to deep learning at all to me. Obviously one can apply deep neural networks also under different assumptions. Would either write: “Many machine learning and deep learning algorithms ...”, Or simply “Many deep learning algorithms…”

“Given the potential impact of designing such an algorithm and the machine learning community’s efforts towards achieving artificial general intelligence, research on continual learning has increased (Parisi et al., 2019).”
-> a bit too broad stuff to me, could be just removed (AGI etc.)

*Confusing beta zero part*

“In this setup, we discovered β_iT quickly decay to zero, and as a coefficient of the loss, prevents the network from learning the task”
This made me a bit confused: As per formula, if you set all betas to 0, the loss would become zero no? So it should be a direct consequence of the formula that yes, gradient descent or any optimization should set all betas to zero without any regularization/constraints? The “discovered” confused me here. Or am I misunderstanding something here?

*Figure improvements*

Figure 2: thicker curves? Hard to see

Figure 5 in appendix: legend would be helpful



As written above, I don’t feel well-acquainted enough with the literature on continual/active learning to judge the contribution of this manuscript to these fields with confidence; hope other reviewers can do this :)

review_title: Clearly described approach; not sure about evaluation quality

Meta-review: Reviewers could not reach consensus here and important concerns from one reviewer on empirical results could not be convincingly addressed. The authors have provided a comprehensive response to the reviews, yet failed to convince them.

Decision: Reject