Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper is about continual learning on NLP applications like natural language instruction learning or machine translation. The authors propose to exploit "compositionality" to separate semantics and syntax so as to facilitate the problem of interest.

In summary, the current manuscript is clearly not ready for publication. The writing is not good, as I cannot see clearly the backbone of the paper. Honestly, I got very confused by the presented contents. What’s the problem/motivation? No preliminary background knowledge? What’s the classic or naïve method to solve the problem of interest? What’s the main advantage/novelty of the presented method compared to that classic/naïve method?

Please see the detailed comments below.

In the Abstract, you mentioned "One of the key skills … ability to produce novel composition’’. Do you imply your method can continually learn new compositions? If so, how does that reflect in the technical parts and the experiments?

The paragraph before Section 3 might be redundant.

Many typos exist. Such as the word "iuput’’ in the 1st paragraph of Page 4.

What’s the problem of S2S-CL? Increasing input number n and output number m?

What does the word "COMPOSITIONALITY’’ mean in Section 3.2? Also, what’s the relationship between the last two equations of Page 4?

How do you defend the simplifications adopted in the first Equation of Page 5?

The notation "{0,1}^{Uxn}’’ usually represents a binary matrix of size Uxn. It is not suitable to use them to represent a matrix containing one-hot columns.

At the beginning of Page 6. Why entropy regularization can be introduced via L2 norm on the embedding matrixes p and f? Also why that L2 norm regularizations can `achieve disentanglement’? Please provide the detailed proof or the reference. 

What are the detailed settings of the demonstrated experiments?

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: *Summary 

The paper proposes a continual learning algorithm for label prediction to deal with sequence-to-sequence continual learning problems. The proposed method is designed to leverage compositionality.  The key idea of the proposed method is to enable the network to represent syntactic and semantic knowledge separately. This allows the neural network to leverage compositionality for knowledge transfer while alleviating catastrophic forgetting.
The experiments showed that their method performed significantly better results than baseline methods.  The method was tested on two different datasets, e.g., instruction Learning and machine translation.


*Decision and supporting arguments

I think this paper has enough quality to be be accepted as a conference paper.
The main reasons of my decision are two-folds.
First, the proposal is quite insightful. The separation of semantics and syntax of an input sentence for using compositionality is an excellent idea.
Second, the proposed method improved the performance on two dataset significantly. This supports the usefulness of the idea.


*Additional feedback

My concern is about evaluation. Table 1 shows the significant difference between the proposed method and the baseline methods. It looks to nice. But, this suggests that the datasets might be too artificial for this evaluation. To my understanding, both of the datasets are artificial to some extent. Hopefully, the method should be evaluated on the more realistic dataset.  

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes an approach for continual learning for applications in sequence-to-sequence continual learning (S2S-CL). More specifically, the paper addresses the problem of growing vocabulary. The overall approach is intuitive yet quite effective. The method assumes a split between the syntax (f) and semantics (p) of the sequence -- in other words, each token is associated with two labels. Furthermore, the syntax is assumed to be shared across all the training sequences and the sequences that are not encountered during the initial training. A sequence model (LSTM) is used for learning the syntax f over the initial training and is then frozen for all the downstream tasks (i.e. continual learning). The sequence model predicts the correspondence between the output token and input tokens. Another network (a 1-layer  MLP) is used to predict the semantic label of the selected token in the target domain. Barring some notational confusion, I believe the method is very reasonable and should work well. They perform experiments on 2 datasets (Instruction learning & machine translation) in 3 continuous learning paradigms with varying difficulties and demonstrate that the proposed method significantly outperforms the baselines by an impressive margin.

I am leaning towards accepting this paper since I believe the proposed approach can be a promising direction for continual learning in S2S settings and the empirical results are convincing.

However, I have some concerns that might require clarification or additional experiments:
    1. The novelty of the proposed method is somewhat limited in my opinion. It seems it is a very simple adaptation of Li et al. 19 and the only addition to that work is the usage of a very simple label prediction scheme by fixing everything else. However, this might be okay since the experiments are quite compelling and the method is applied to a different setting.

    2. Notations and language in section 3.2-3.4 are hard to follow:
        - At the bottom of page 4, I believe some of the equations are not correct. For example, the first term of line 2 should be P(Y^f | X^f, X^p) rather than Y^p?
        - In the second equation of page 6, the second term should be v_j = p’ \cdot a_j
        - In Figure 1, k_r, E_r, and W are never defined (although their meaning can be inferred).
        - In section 3.4, it reads that new word embedding is appended to both semantic and syntactic embedding matrices but this doesn’t make sense because the syntactic network is already fixed so it shouldn’t be able to handle new symbols; therefore, I believe only new row is added to the semantic embedding.

    3. I believe that f_predict here is parameterized by \theta and \theta is also frozen during the continual learning phase which contradicts the claim at the end of section 3.2 that only \phi is frozen. Otherwise, it’s hard to see why f_predict does not suffer from catastrophic forgetting. From the provided source code, it seems that it is indeed the case that only the new embedding is being updated. In other words, the only thing happening at this stage is that the newly added embedding are optimized to adapt to the frozen f_predict.

    4. Due to point 3, I have some doubts about how scalable this approach is if the other embedding is not allowed to co-adapt. However, perhaps this problem could be alleviated if f_predict has extremely high capacity at initialization? More on this in point 6.

    5. Assuming all my assessment above is correct, it seems that the performance of the proposed method should *not* decrease at all so I would like to see more analysis on the decreasing performance in the instruction learning task (Figure 2, left column). Is this be due to the fact that f_predict or the other embedding is not allowed to update and the newly added embedding is mapped close to existing embedding? In that case, can increasing the model capacity solve this problem? I understand the paper makes argument about regularization but I believe this warrants thorough study for gauging the significance of this approach in more realistic settings.

    6. I would like to see a discussion of the short-comings of the approach and possible ways to overcome them. For example, freezing the syntactic network seems limiting in machine translation settings if a new language, say Italian, is added. Intuitively, knowing how to translate English-French should help translating English to Italian but fixing the syntax prevents this. Another example is that prior knowledge about the syntax needs to be known about the language for labeling the f and p and this can be expensive and cannot handle words that have more than 1 usages (e.g. run can be used as a noun but also as a verb).

I am willing to increase my score to accept if the revised manuscript can address the majority of, if not all of the concerns listed above.

========================================================================================================
Minor comments that did not affect my decision:
    - These sections could greatly benefit from an overall flow chart of how everything fits together
    - It says the experiments are repeated with 5 different random seeds so why not add error bars to the plots?
    - What characteristics of the 2 tasks cause the baselines to behave so differently?

review_title: Official Blind Review #2

Meta-review: The paper addresses the task of continual learning in NLP for seq2seq style tasks. The key idea of the proposed method is to enable the network to represent syntactic and semantic knowledge separately, which allows the neural network to leverage compositionality for knowledge transfer and also solves the problem of catastrophic forgetting. The paper has been improved substantially after the reviewers' comments and also obtains good results on benchmark tasks. The only concern is that the evaluation is on artificial datasets. In future, the authors should try to include more evaluation on real datasets (however, this is also limited by availability of such datasets). As of now, I'm recommending an Acceptance.
Decision: Accept (Poster)