Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Continual Post-training of LMs with Soft-masking (CPS) that adapts the pre-trained language models on different domains in a continual-learning way. CPS is designed such the language model can maximally preserve the knowledge learned in the general domain and previously-seen knowledge-specific domain. This is done by calculating the importance of submodules in the transformer models and using the importance score to calculate the soft-mask. The soft-mask is used to mask out gradients such that important units for the previous domains will not be changed too much. The experiment results of continual post-training on six different domains show that the proposed method, CPS, outperforms other continual learning and non-continual learning baselines.
strength_and_weaknesses: Update after discussing with the authors
===

After all the supplementary experiments added during the discussion period, I find this paper and the proposed method to be solid and convincing.
I highly appreciate the efforts done by the authors and I find this process to be very beneficial.
**All the weaknesses are perfectly addressed.** 
I am thus raising the score from 3 to 8.

-------------

Strengths
===
1. Propose a method that shows empirical performance gain in language model post-training
2. The paper contains a lot of baseline methods for comparison.

Weaknesses
===
1. The intuitions of the proposed method are not very convincing. These include the loss function in the initialization and continual learning, and the importance computation, which are the core of CPS.
   - I do not see why we need to assume "general domain datasets are not accessible to users of the LM". Those general domain corpora should be very easy to obtain in realistic, and that is why they are used for self-supervised pre-training.
   - I cannot agree with why using the domain-specific dataset and the loss function in Equation (4) is for learning general domain knowledge.
   - I do not understand why contrastive learning is used here. Why would we want the representation of full knowledge and previously learned knowledge to be pushed away in the representation space, and how does this connected with knowledge "integration"?

2. Some previous works and baselines should be investigated, this includes
   - "Jin et al. (2021) compared several CL methods for continual post-training", as stated in Section 2. In fact, the scenario discussed in this paper is highly overlapped with Jin et al. (2021). In this case, the methods mentioned by Jin et al. (2021) should be compared. However, this paper misses a comparison between most knowledge distillation-based methods in Jin et al. (2021), which are exactly the methods that achieve better performance. There are 5 different KD methods in Jin et al. (2021) but there is only one KD method in this paper.
    -  I also think LoRA [1] should be used as a baseline. 
    - Why is AdapterFusion not compared here? One can train domain-specific adapters using unlabeled data (MLM), and use the domain-specific downstream task dataset to learn the parameter of AdapterFusion and the downstream task's parameter.

3. It is unclear whether the soft-masking based on importance is really very critical to the proposed method.
    - I am curious whether CPS is successful because when updating the model, the soft-masking is calculated based on the importance, or it is just because we are suppressing the update of some proportion of the weights, and where the weights whose gradient update is soft-masked is not important. Gating the gradient during backpropagation sounds like some kind of regularization, and I would like to know how the regularization is placed is really important. To be specific, I would like to see an ablation study that constructs a random importance score $I_{l}^{(k)}$ that samples from a fixed probability distribution on $[0,1]$, and see whether the performance degrades or not.

4. The paper is not very clear, including:
    - What does $f_{LM}(o_l^{prev})$ mean? What does "plug it into the whole language model" means here? Is this simply using $I_{l}^{(\leq t-1)}$ in the transformer model?
    - How is NCL (Adapter) done? I understand that this is described in the appendix, but I find that most experiment details related to the baseline methods are placed in the appendix, making the paper's main content not very well self-contained. An additional question is why not add a new adapter in a new domain as in [2]? This is like DEMIX but different initialization method.
    - The intuitions for the methods are not very convincing, making me find those parts confusing and not very clear.
    - How is the model fine-tuned on the end tasks in CL baselines? Full model fine-tuning with a classifier head or using parameter-efficient fine-tuning methods such as adapter or LoRA? Does fine-tuning methods change the conclusion of what CL method is better?

5. Lack of comparison when the learning order of domains is permuted. In Table 4, only the performance of CPS and NCL are compared, but this Table should have compared other CL methods.

[1] Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." International Conference on Learning Representations. 2021.
[2] Jang, Joel, et al. "Towards Continual Knowledge Learning of Language Models." International Conference on Learning Representations. 2021.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The clarity can be significantly improved, including especially in Section 3. Also, more descriptions of the experiments should be included in Section 4.
- Quality: The quality can be improved, regarding the experiment setting, baseline comparisons, and the clarity of the paper.
- Originality: The training objectives of the CPS are actually quite novel but unintuitive and unconvincing.
- Reproducibility: Good, the authors provide the source code.

summary_of_the_review: Update after discussing with the authors
===
I think this paper provides a solid and convincing method for language model post-training. While there may be minor issues with the writing, the method itself is good. I think this paper should be accepted.

-------------
I think the paper requires significant improvements, including experiment setting and writing. I do not think the paper is ready to be accepted to ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper authors propose to improve continual post-training by the way of computing saliency of each unit or layer in transformer model. Average saliency is then used as a weight in the backward pass. Idea appears to be first time used in relation to self-supervised loss functions such as MLM as is the case here. Authors overcome multiple technical challenges in their system and the final experimental results are quite convincing. 
strength_and_weaknesses: Strengths: 
- Good empirical results in a very relevant task (self-supervised LM) with a number of different target domains. 
- Approach seems to make intuitive sense. 

Weaknesses: 
- Writeup is not finalized (see explanation in the clarity box)
- Theoretical grounding is missing. Final result appears to be empirical searched solution to a practical problem, while idea itself is sensible. In my opinion, authors should try to answer theoretically that in some restricted case the proposed method would get the "correct" answer. Other way would be to show that it cannot completely fail. 
clarity,_quality,_novelty_and_reproducibility: How about just pool all data and train only once? What would be that performance, obviously continual updating the model with newly acquired data would not be feasible with this
approach, but it is necessary to know how much we lose by doing continual learning. 

I feel that too many acronyms are used, such as CF and TIL, please consider eliminating unnecessary ones. And in the same vein, why the first TIL occurrence does not have a reference?

Introduction: About properties required, to me it seems like property (1) and (2) are same. Both essentially talk about catastrophic forgetting. Please clarify. 
Footnote 2: I would say instead that definition of catastrophic forgetting is that training on new domains causes performance on previous domain to degrade. So to strengten that statement. 

Section 2: 
Is difference between ELLE and the proposed method only that you start with pretrained model and ELLE starts from scratch? If so, I guess your novelty is then very limited. Please correct this statement. 

I would contest these statements: "(1) we compute importance for soft masking, not for pruning, and (2) post-training works on unlabeled data, while the above approaches work with labeled data."
First of all, computing importance for soft masking and pruning can be considered to be the same. If for pruning you need to compute a score each node and then apply some thresholding, well then you can apply same scoring technique for your soft masking. Same goes for labeled vs unlabeled case, in both cases you have loss function that is being optimized. There is no 
principal difference why method developed for labeled data could not be used for unlabeled data. 

In don't understand why contrastive learning is mentioned in this context. 

Section 3: How is KL in (4) computed? What exactly is the input to it? Mathematical details are missing here. 
In proxy loss you take subset of the _current_ domain data, how you can claim that gradient of the KL then represents general knowledge? 

Equation (5), I am confused, where is mean and Var taken of, over all I_l?  Please clarify. 

Equation (6), what is EMax? 
summary_of_the_review: Idea does make intuitive sense, but it is my opinion that paper at the current state is not of sufficient quality. 

After rebuttal:

I find authors rebuttal and changes to be substantial enough to change my score from 5 -> 8.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to continue adapting the language model to new domains without forgetting the previous domains. The main method first calculates the important neurons based on general language knowledge. Then it uses a KL divergence loss to to continued adaptation to other tasks and domains.
strength_and_weaknesses: 1. the general ideal of training different important parameters for each domain, and preserves the previous domains is pretty interesting

clarity,_quality,_novelty_and_reproducibility: 1. I cannot understand most of the technical details. It's not very clear what the actual method involves. The paper could be greatly improved by having a clear and general summarization of what each component of the model does.
Some areas that are difficult for me to understand are:
- I'm still not sure what the motivation behind proxy KL-divergence loss is. One could sample some pretraining data, and use the MLM objective to calculate eqn 3). I'm not sure if I misunderstood the explanation/motivation for proxy KL-divergence loss on page 4. 
- the first paragraph for proxy KL-divergence is also a bit confusing. The authors used I_{l, i}^{gen} to indicate the importance of the units, but this notation is not used anywhere else.  Is that basically just the gradient, which is delta_{gl}? There are many notations in this paragraph that are not used anywhere else, so it is very confusing for readers to understand.
- I think the second paragraph for proxy KL-divergence is also a bit difficult to understand. The formula in eqn 4) is helpful, but the language used to explain eqn 4) could use some improvements. I'm also not sure what "no label in post-training" mean.
- in eqn 6), I'm not sure what EMax means
- for section (3) of "Contrasting the previously learned and new domain knowledge", the author mentioned o_prev and o_full, but then used O_+ and O_- for the formula in eqn 11), which makes this section very hard to follow. It might be helpful to just use o_prev and o_full, which makes it clear to the readers what are the representations used in the contrastive loss.
- in eqn 12), the authors introduced another notation L_{constraint}, which is not used anywhere else. Maybe consider using some notations you already defined to tie everything together. 

2. It seems like the authors only tested on adapting from one domain to the next domain, but it's not clear whether the method would work well on multiple domains.
summary_of_the_review: The main issue with the paper is that the writing of the technical methods are very unclear, so it's difficult for me to understand or assess it. It might not be suitable for publishing at its current stage because the writing is not clear enough for people to replicate the method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of sequentially pretraining a neural language model on a sequence of domain specific training sets whilst preserving the original pretaining and avoiding catastrophic forgetting.  The basic idea is to maintain a soft mask over critical neurons (in this case the attention heads) such that backward error propagation is scaled down for paths which encode most of the existing knowledge.  The mask weights are learned by introducing a temporary gating function and using the normalised gate weights as the mask.  Mask weights are updated for each new dataset by taking the max of the mask for the previous iteration and the current.  In addition the normal LM Mask loss is augmented by a contrastive loss designed to encourage the LM to acquire knowledge from the new data that is not already in the knowledge extracted from previous domains.  In the experimental evaluation, the proposed system is extensively tested against a number of existing continual learning paradigms with impressive results.  Over a sequence of 6 datasets, the forgetting rate for the proposed system is negative showing that it not only prevented forgetting but it also facilitated knowledge transfer from one domain to another.  Ablation tests show that each component of the proposed system makes a positive contribution. 
strength_and_weaknesses: The paper provides a principled approach to mitigating the forgetting problem when learning over time using sequential data sets from differing domains.  The solution is elegant and practical.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and introduces several new and well-motivated ideas.
summary_of_the_review: Overall this is a very nice paper which introduces a novel approach to a problem which has significant practical importance for real world applications.  The methods here are presented in the context of large LMs, but they should be applicable to a wide range of neural network systems. The work is clearly presented and the experimental work is thorough and convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of continual post-training. Although there is previous work with the same setting, this work proposes a new solution. 

This paper proposes Continual Post-training of LMs with Soft-masking (CPS) to tune pre-trained language models across different domains in a continuous learning manner. CPS is designed such that the language model can maximize the retention of knowledge learned in the general domain or previous domains. This is done by computing the importance of submodules in the transformer model and using the importance scores to mask the gradients so that critical units from previous domains do not change too much. Using the importance scores to consolidate learned knowledge is common in continual learning. But the paper further augments the typical LM Mask loss with a contrastive loss, which aims to encourage LMs to acquire knowledge from new domains. Experimental results in six domains show that the proposed method CPS outperforms other continual and non-continuous learning baselines. The reviewers suggest more baseline experiments. The additional results provided by the authors make the results more solid and convincing. 

The main issue with the paper is the writing. The reviewers have questions about the intuition and technical methods in the beginning. But problems are solved after the explanation of the authors' feedback. Please carefully polish the camera-ready version based on the discussion with the authors if the paper is accepted.
justification_for_why_not_higher_score: The original version of the paper confused the reviewers. Hopefully, the authors will polish the paper according to the discussion with reviewers.
justification_for_why_not_lower_score: Overall this is a nice paper introducing a novel approach to a problem that has significant practical importance for real-world applications.
Decision: Accept: poster