Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
Summary:

The paper proposes to use the same language model to learn multiple tasks and also to generate pseudo-samples for these tasks which could be used for rehearsal while learning new tasks. The authors demonstrate that this idea works well compared to other SOTA lifelong learning methods for learning various NLP tasks using a single model.


My comments:

1. Please change the title! Language modeling is NOT all you need for lifelong language learning. Also, not every NLP task is a QA task. I do not want more papers to over-trivialize NLP by following Bryan McCann and Socher, 2018. I will not increase my scores until the title is changed.
2. A relevant model architecture based method is Sodhani et al. 2018 (Towards Training Recurrent Neural Networks for Lifelong Learning) who use Net2Net to do zero-shot expansion of the model parameters.
3. Section 3.2 - you mention that any pseudo-example which does not have only one ANS token is discarded. Can you comment on how much discarding is needed to generate the required number of pseudo-samples?
4. Why is it that every task was trained only for 9 epochs?
5. On page 5, you mention k=20. What is k? Where is this introduced?
6. On page 5, you mention that MTL is used to determine whether forgetting is caused by a lack of model capacity. I am not sure if it is correct. Can you explain?
7. Why not compare the approach with models like GEM? Keeping very few examples is ok. Even though you don’t beat GEM, it is good to see the comparison.
8. Page 7: Is there any reason why you choose to go from large to small tasks? I feel like this is a favorable order. I would like to see how the model performs if you do the reverse order.
9. Please remove the last line.
10. I assume that the authors will release the code upon acceptance of the paper.

Minor comments:

1. Page 2, 4th contribution: check the spelling for “pseudo-samples”
2. Page 2, 5th last line: “After a completing a task” - fix it.
3. Table 1: I think the description is not correct. 1fEM is for wikiSQL, not WOZ. Also, it is better if you can describe these metrics in detail in the appendix.

==================================

After rebuttal:

I am happy with the authors' response and name change. I am increasing my score.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of lifelong language learning. The core idea underlying the algorithm includes two parts: 1. Consider the NLP tasks as QA and then train a LM model that generates an answer based on the context and the question; 2. to generate samples representing previous tasks before training on a new task. 

In experiments, the authors demonstrate the efficiency and effectiveness of the proposed models based on the following perspectives:
1. Compare the proposed method with existing baselines. However, it seems that keep real data is missing from Table 3. 
2. Preliminary studies with 3 tasks and study the task oder.
3. Performance of the training epochs.
4. Hyper parameter tuning gamma. 

The weak points of this paper are as following:
1. The assumption of modeling all tasks as QA might be strong;
2. The baseline from using real data is missing;
3. There are many components that are missing from the discussion, such as the complexity of the language model, etc. For instance, when the model complexity is high,  TopK sampling could be expensive.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper presents a new NN architecture designed for life-long learning of natural language processing. As well depicted in Figure 2, the proposed network is trained to generate the correct answers and training samples at the same time. This prevents the "catastrophic forgetting" of an old task. Compared to the old methods that train a separate generator, the performance of the proposed method is noticeably good as shown in Fig 3. This demonstrates that the new life-long learning approach is effective in avoiding catastrophic forgetting.

The motivation of the paper is clear. The comparison to old methods seems fair. The proposed method is clearly different from previous methods.

One weakness of the paper is in the experimental results especially in section 5.4. The statistical significance of the results in table 5 is missing. As the authors have discovered, the performance is highly dependent on implementation. In addition, the resulting performance might have a high variance. From the data, it is hard to argue that LAMAL is better than MBPA++ by a significant margin.

I recommend having a more elaborate figure instead of Figure 2. The structure of the network might be particularly interesting for people who are not in this field. 

Overall, the results are very interesting and worth a publication.
review_title: Official Blind Review #2

Meta-review: This paper proposes a new method for lifelong learning of language using language modeling. Their training scheme is designed so as to prevent catastrophic forgetting. The reviewers found the motivation clear and that the proposed method outperforms prior related work. Reviewers raised concerns about the title and the lack of some baselines which the authors have addressed in the rebuttal and their revision.
Decision: Accept (Poster)