Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposed an interesting continual learning approach for sequential data processing with recurrent neural network architecture. 
The authors provide a general application on sequential data for continual learning, and show their proposed model outperforms baseline.

It is natural that their naive baseline shows poor performance since they do not consider any continual learning issues like the catastrophic forgetting problem. Then, I hesitate to evaluate the model in terms of performance. In that sense, it would be much crucial to show more meaningful ablation studies and analysis for proposed model. However, there is a few of thing about them. 

Then, I decide to give a lower score that even the authors suggest that the main contribution is a definition of problem setting. It requires more detailed and sophisticated analysis.
          

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The goal of this work is to best understand the performance and benchmarking of continual learning algorithms when applied to sequential data processing problems like language or sequence data sets. The contributions of the paper are 3 fold - new benchmarks for CL with sequential data for RNN processing, new architecture introduced for more effective processing and a thorough empirical evaluation. 

Introduction: 
I think a little more insight into why the sequential data processing CL scenario is any different than the vision scenario would be quite helpful. Specifically, it would be quite impactful to tell us more about what the additional challenges with RNNs for CL vs feedforward for CL are in the intro. 

The paper is written as if the benchmark is the main contribution and the architecture improvement is just a delta on top of this, but it gets confusing when the methods section starts off with just directly stating the new architecture. 

The algorithm seems like a straightforward combination of recurrent progressive nets and gated autoencoders for CL. Can the authors provide more justification if that is the contribution or there is more to the insight than has been previously suggested in prior work?

Figure 1 has a very uninformative caption. It also doesn’t show how modules feed into one another properly. 

The motivation for why one needs GIM after one already has A-LSTM or A-LMN is not very clear?

Overall the contribution does seem a bit incremental based on prior work and the description lacks enough detail to properly indicate why this is a very important contribution?

Experiments:
What does it mean to be application agnostic but restricted to particular datasets and losses? This doesn’t quite parse to me. 

The description of the tasks is very informal and hard to follow. It’s not clear what exactly the tasks and datasets look like 

“using morehidden units can bridge this gap” -> why not just do it? Its a benchmark after all. 

Overall the task descriptions should be in a separate section where the setup is described in a lot of detail and motivated properly. 

The results in the experiments section are very hard to parse. The captions need much more detail for eg Table 2. 

Could we also possibly have more baselines from continual learning? For instance EWC (Kirkpatrick) or generative replay might be competitive baselines. 

Overall I think that the GIM and A-LMN and A-LSTM methods are reasonable although somewhat incremental. But the proposed benchmarks are pretty unclear and the results are a bit hard to really interpret well. It would also be important to run comparisons with more baselines and to provide more ablation/analysis experiments to really see the benefit of GIM/A-LMN or A-LSTM. I also think that the task descriptions should be much earlier in the paper and desribed in much more rigorous detail. 

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: Summary:

In this paper, the authors propose a new method to apply continual learning on sequential data. The model is constructed by combining an Autoencoder and LSTM/LMN for each task. The experiments on several datasets show the proposed model outperforms basic LSTM/LMN.


Strength:

+ Sequential data widely exist in the real world, e.g., text, health records. Thus, It is interesting to see that continual learning is used in sequential data. 

+ The motivation of the proposed model is clear. The authors save the learned knowledge in the hidden representation of LSTM/LMN.

Weakness:
- In this paper, the model size linearly increases since the number of LSTM/LMN and AE increases when a new task comes in. Thus, if the number of tasks is too large, the model size is quite big. In traditional continual learning settings, researchers may not always increase the model size for overcoming catastrophic forgetting. For example, if task 1 and task 2 sample from the same distribution, they can share the same LSTM/LMN and AE. Thus, it would be better if the authors can consider how to reduce the model size in the future version.

- In the experiments, the authors only compare the proposed model with simple LSTM or LMN. However, most continual learning methods can still be applied in this scenario, at least regularization based methods [1,2] can be simply applied in this scenario. The authors may need to compare the proposed method with them in the future version.

- It is better to compare it with a larger dataset. For example, in the natural language processing field, we can regard sentiment analysis on one language as one task. Then, we can construct the continual learning dataset for sentiment analysis.

Minor Comments:
It is better to improve Figure 3 by adding the x-axis label and y-axis label.


[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.
[2] Zenke, Friedemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
review_title: Official Blind Review #3

Meta-review: This manuscript describes a continual learning approach where individual instances consist of sequences, such as language modeling. The paper consists of a definition of a problem setting, tasks in that problem setting, baselines (not based on existing continual learning approaches, which the authors argue is to highlight the need for such techniques, but with which the reviewers took issue), and a novel architecture.

Reviews focused on the gravity of the contribution. R1 and R2, in particular, argued that the paper is written as though the problem/benchmark definition is the main contribution. R2 mentions that in spite of this, the methods section jumps directly into the candidate architecture. As mentioned above, several reviewers also took issue with the fact that existing CL techniques are not employed as baselines. The authors engaged with reviewers and promised updates, but did not take the opportunity to update their paper.

As many of the reviewers' comments remain unaddressed and the authors' updates did not materialize, I recommend rejection, and encourage the authors to incorporate the feedback they have received in a future submission.
Decision: Reject