Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors investigate whether or not pretrained language models like BERT have the ability to maintain previously learned knowledge in the long term. To do this, they track the encoding ability of BERT for specific tasks before, during, and after learning new tasks. They find that BERT can actually refrain from forgetting when learning a sequence of tasks, contrary to existing studies about catastrophic forgetting. The authors believe this is due to the fact that BERT has a strong potential to produce high-quality representations for previous tasks even without memory replay. They further investigate the topological structure of the learned representation sub-space within and among different tasks and find that forgetting can be interpreted as intra-task forgetting (forgetting what has been learned within a task) and inter-task forgetting (forgetting what has been learned across tasks).
strength_and_weaknesses: I like this paper, the results are neat, the exposition is clear (Section 4.2 could use some work as described later), and (almost) everything is easy to understand. But as I was reading this paper, I kept coming back to one paper [1] that I read last year which had done everything this paper is offering. They answered the same questions, they did an even more extensive set of experiments, and the only difference was in how they viewed why pretrained models maintain good representations even if the decoder needs to be retrained. I spent the last day reading both papers side by side and in my view, up until Section 4, there's hardly any difference between the two works. It appears that the authors may not be aware of [1] as it hasn't been cited in this paper so I invite the authors to read [1] and share if they disagree with my assessment. But in light of this, so far, I see the marginal value of this paper lying in Section 4 and would recommend the authors to amplify that as the core contribution of the paper (which is currently listed as their third contribution in Introduction), as the rest of their contributions (claimed contributions 1 and 2) have already been offered to the community previously. Additionally, please add standard errors in every result.

[1] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. "An empirical investigation of the role of pre-training in lifelong learning." arXiv preprint arXiv:2112.09153 (2021).

Errata:
- Introduction, first paragraph: "learning the xxx one". No idea what you meant to say here.
- Introduction, first paragraph: "learning a sequecne of tasks" ---> "learning a sequence of tasks"
- Section 3.2, second paragraph: Kingma and Ba (2015) should be \citep not \citet
- Section 4.1 "decoders,pre-trained" ---> "decoders, pre-trained"
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to understand but Section 4.2 could use another pass (a strategy that works for me is to read every sentence out loud and ask what does it mean?). The novelty is marginal at best. I'd recommend the authors put a section in Appendix about implementation details and add standard errors to the results.
summary_of_the_review: I like the paper and found the results to be clear and easy to understand. However, my current view is that much of what was presented in this paper had already been answered in another paper last year which the authors may be unaware of (as it hasn't been cited in this work). The main difference between the two papers, in the my opinion, is in Section 4. I recommend that the authors emphasize Section 4 as the main contribution of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work analyzes catastrophic forgetting during different multi-task learning strategies, specifically sequence training and replay. The authors test BERT on a suite of classification tasks and question answering, and use linear probes to test the ability of the model on each task before training, after training and after replay. They find that the sequential model does as well as the replay models however, t-SNE plots reveal that the sequential model often confuses representations across tasks. Next, they look at the angle between an exemplar and the mean of the class to characterize how the topological ordering changes due to replay. Finally, they look at the effect of replay in both the current dataset and the one being replayed.
strength_and_weaknesses: I appreciate the authors' detailed response. I've read through the comments and have updated my score accordingly.

##################################

Strengths:
The paper presents an interesting problem and novel use of the solutions prescribed to analyze it.

Weaknesses/Questions:
- On one hand the authors say that the task decoders are united in that the denominator for $P(\hat{y}=\alpha|x_i)$ is over all $y \in Y$. But for question answering it seems that the decoders for start and end are disjoint, and the softmax is over all tokens in the context not all tasks. Could the authors clarify? This also plays into the fact that catastrophic forgetting is evaluated by training new probes on the frozen encoder. If the decoder is separate for each class, could we not use that directly? One problem would be that perhaps the model has changed how it stores information after seeing a new task. This could be verified by testing if the original decoders work as well, in comparison to the newly trained decoders.
- Section 3 introduction: Please add standard errors here and everywhere. Just to be clear, the macro-averaged accuracy scores is based on SEQ and not REPLAY right?
- Figure 1: Nit: Put the replay order backward! Also indicate what the dashed lines mean.
- Figure 2: Where is the dashed red line for SQuaD? Nit: Order subpanels based on order of training tasks.
- Section 3.2: “However, the probing results (blue lines) are still much higher than the original scores measured before re-training de- coders (red dashed lines). Comparing the obvious gap between them4, we can find that BERT still keeps most of knowledge of previous tasks when learning new ones.” Yes but the information might be re-organized which is why the blue line does a bit better. (as the authors also note in the 4th footnote)
- Table 1 analysis: Just to be clear, the representation vectors for a particular dataset, say Amazon, is when one does inference on the Amazon training instances before training on it correct? This is done to define the cone axis and find narrowest angle? And the process is repeated after training on Amazon? I understand the method and motivation but this doesn’t test for inter-class separability no? Even if the topological order within a class might be high, the cone axes or the cones itself could be overlapping leading to reduced separability. This brings me to the point that we have no way of ascertaining that a given correlation is high or low without looking at what the value would be for two pairs sampled randomly from different classes. This would tell us about the ordering of the space entirely. Also, what are the variances for each entry in table 1? How much does this value vary for different examples?
    - I would also be curious to see how this ordering varies for different subsets of classes. Do some classes, for example, get closer/farther depending on the distribution of the most recent training set?
    - It would also be more interesting to see how this metric varies for the AGNews dataset before training, after training and then after incremental training.
- What is the expected result for the inter-task forgetting experiment and the interpretation?  If the value is positive, it means that the new angle is smaller than the old. So as you replay more, the new and old angles become almost equal and since the old angle is fixed, equality would mean that the new angle became bigger? On the contrary, a negative value means that the old angle is smaller than the new. So as you replay more, the difference between them becomes small and since the old angle is fixed, equality would mean that the new angle became smaller? If this is correct, I understand “reveal that memory replay obliges the vectors of previous tasks rotating to their corresponding column-vectors in decoder efficiently, while dragging those of current task to deviate from optimal position.” but it wasn’t clearly written in the text.
- I am confused by the contradiction between statements and results. For example, the first result (Fig. 1) suggests that the sequential task scheme does as well as memory replay but the authors go on to suggest it doesn’t for the rest of manuscript and do subsequent analyses based on this assumption. For example, the t-SNE plot suggests that under SEQ and not REPLAY, there is a high degree of inter-class overlap. How can we explain figure 1 results then? Additionally, aren’t the results in table 1 for SEQ? If this is the case, what does the high correlation tell us about purported catastrophical forgetting in SEQ? This is not clear from section 4.2 but section 4.3 leads me to believe it is replay. Please clarify.
clarity,_quality,_novelty_and_reproducibility: Clarity: While a lot of the methods were easy to follow, the conclusions or observations made form each experiment were a bit difficult to parse!
Quality: Interesting paper although it could benefit from analyses suggested above.
Originality: To the best of my knowledge, this seems like a novel step towards understanding replay mechanisms for multi-task learning.
summary_of_the_review: Overall, the paper presents an interesting set of tools to analyze the problem of catastrophic forgetting in neural LMs. However, the results and claims seem contradictory and it is not clear to me if the authors are claiming that purely sequential learning *suffers* from forgetting or it does not! To that end, I believe that these tools can be used to more systematically understand how the replay and sequence strategies differ in the geometry of the representations given that they both do well for the tasks described here.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors conduct a two-part study to investigate phenomenon of "forgetting", i.e. degraded prediction performance on past data, when a language model is trained on a sequence of tasks. First, they show evidence via a probing study that a BERT encoder base in fact largely retains representations capable of high performance on a given task, even after subsequent training on other tasks. Second, they propose a sequence of analyses based on the idea that same-class embedding vectors fall in a relatively narrow cone, showing that (1) embedding vectors from the same class for a given task have similar nearest-neighbors after training on another task, and (2) replay re-aligns cone axes with the relevant decoder column for past tasks while inducing some misalignment for the current task. 
strength_and_weaknesses: Strengths:
- The overall presentation and analysis are of good quality and rest on a sequence of interesting and informative experiments. These are unified under the general theme of understanding how the representations in the BERT encoding layer evolve during training. 
- The probing study is simple and effective in showing that "forgetting" is not a matter of representational degradation in the encoding layer, as decoder "probes" achieve similar accuracies both during and after training on a given task. This raises subsequent questions that the authors answer with a novel perspective.
- The authors expand on previous observations in the literature to argue that same-class sentence-level embeddings lie within a cone, then use this representation to propose and execute novel analyses relevant for the topic of task forgetting and replay. These methods represent potentially new and useful ways to study the evolution of learned representations over the course of training. The results offer new insights as to how learned representations change (or don't change) when a new task is learned or after replay.

Weaknesses:
- There is some lack of clarity in the presentation of the novel methods, particularly in $\S 4.2$. The authors should define precisely what it means for the "rotating process" to be "topological(ly) ordered". It is also unclear what it means to evaluate the "correlation between the relative positions of $v_{y,i}^{(1)}$ and $(v_{y,k}^{(1)}), k \in N_{y,i}$" or why this is "estimated by the Pearson correlation coefficient between $\cos(c_{i}^{(1)}, v_{y,i}^{(1)})$ and $\sum_{k \in N_{y,i}} \cos(c_{i}^{(1)}, v_{y,k}^{(1)})$" (in the paper the quantity $v_{y,i}^{(1)}$ appears inside the sum but I assume this is a typo). Overall, it is somewhat unclear why this specific approach was taken if the goal is just to establish that same-class embedding vectors have the same or similar nearest neighbors after training on some subsequent task.
- The authors spend relatively little time contextualizing the sequence of (interesting) results that they generate. How do these support or contradict existing hypotheses in the literature regarding the mechanism of task forgetting and replay? What hypotheses are the studies in $\S 4$ designed to evaluate, and what are the directions for future analysis?

clarity,_quality,_novelty_and_reproducibility: The paper is overall well structured and presented, though as noted above I feel that the approach and justification for some parts of $\S 4$ is unclear. The quality is good and could be improved by some further contextualization of the results. The work seems substantially novel, particularly in terms of the methods and results presented in $\S 4$. 

It does not seem that code or data was included in the submission, which raises questions for reproducibility, especially given that the paper proposes some novel analyses. Despite the relatively clear presentation, it might be difficult for a motivated reader to fully reproduce these method or results. I would encourage the authors to consider sharing their code.
summary_of_the_review: This paper presents a coherent, creative sequence of empirical analyses that illuminate how the phenomena of sequential task forgetting and recovery via replay are related to the representational capacity and within-class structure of embedding vectors from a BERT encoder. The authors could further improve on their contributions by clarifying some details of a novel analysis in $\S 4.2$ and releasing their code.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores whether BERT forgets representations of previous tasks over the course of being trained on new tasks. The method tracks the encoding ability of BERT for specific tasks before, during, and after learning new tasks. They find that BERT can actually refrain from forgetting when learning a sequence of tasks, contrary to existing studies about catastrophic forgetting. The authors believe this is due to the fact that BERT has a strong potential to produce high-quality representations for previous tasks even without memory replay

**Strengths:**

Overall, the reviewers seem to have positive opinions of the paper, with some praising the quality of the presentation and experiments and others noting the potential usefulness of the novel methods proposed.

**Weaknesses:**

There are some points of confusion or unclear presentation that the reviewers have raised, such as the lack of clarity around the definition of a "topological(ly) ordered" rotating process and the contradiction between some of the statements and results. It will be important for the authors to address these points in order to improve the clarity and coherence of the paper.

justification_for_why_not_higher_score: See meta-review
justification_for_why_not_lower_score: See meta-review
Decision: Accept: poster