Kati8/ICLR_2023_2025 huggingface dataset:
authors: Weimin Yin
authors: Chunzhao Xie
authors: Bin Chen
authors: Zhenhao Tan
reviews: 
Comments: In order to uncover the root causes of catastrophic forgetting in Continual Learning, this study analyzes phenomena that arise during single task training. This analysis reveals three key phenomena (Interleaved Representation Production, Task Representation Convergence Phenomena, and Forward-Interleaved Memory Encoding) inherent in the training process of neural networks.
Strengths: None
Weaknesses: W1. The paper presents unsubstantiated claims and contents. We provide some examples of unsubstantiated claims:

(Section 2.2) “This indirectly suggests that the model training is equivalent to training an encoder. However, we have not provided an explanation for these phenomenon. On one hand, we do not have the time to provide a comprehensive mathematical proof,”
(Section 2.4) “In the model initialization phase, all parameters are randomly initialized. At this point, regardless of the input received by the encoder, after undergoing multiple rounds of random parameter adjustment, they will almost all be mapped to similar representations.”
(Section 2.4) “The paper assumes default training using the stochastic gradient descent (SGD) algorithm. If other optimization algorithms are used, there may be differences in the formulation processes, but it does not affect the conclusion.”


W2. The phenomena claimed to be elucidated in the paper are already well-known facts, rendering them trivial. We provide some examples:

- The observation in Section 2.1, where distinct representations are formed based on classes, is a well-established fact.
- The practice of employing the expectation of a representation as a decoder in Class Incremental Learning is a widely recognized approach. For more details, you can refer to the following paper. [Rebuffi, S. A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017). icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 2001-2010)].


W3. The explanation in the paper is difficult to comprehend.

- The notations are confusing.
- The provided explanations for the experimental results are insufficient.
- Descriptions for the axes on the figures are absent.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: The authors present an empirical investigation on catastrophic forgetting, mostly by tracking changes in representations learnt by neural networks during training. In particular, they look at both the impact of last-hidden-layer representations ('encoder' in the paper), and at the changes in activation in all layers of the network during training.
For example, they observe that the last-hidden-layer representations become class-discriminative (which is however trivially implied, since the output layer acts as a linear classifier, and accuracy improves during training).
Strengths: The authors thoroughly look at the problem of catastrophic forgetting and in general of neural network training in a thorough fashion.

The article is written in an engaging way (but see 'Weaknesses' below).
Weaknesses: Overall, it seems that none of the results in the paper are particularly original, and it rather looks like that the authors are not aware of prior work in the field.

This is reflected by some mis-use of terminology (e.g., encoder/decoder) and by strong statements (e.g., "One such problem that has remained inexplicable since the advent of neural networks is catastrophic forgetting"; "However, we do not fully understand how they accomplish this process, which is why they are often referred to as black boxes.").

Likewise, language is informal and imprecise (e.g., introduction, and then at many places in the text like "To address this issue, we first introduce a definition here"; "Now, we have both good news and bad news. The good news is that we now have a clear understanding of how catastrophic forgetting occurs. The bad news is that it is intertwined with the concept of backpropagation".
The way the article is written would be suitable for a blog post, but not for a formal scientific article.

General remarks:
    - The split of any neural network in encoder/decoder parts seems mostly arbitrary and it does not add much to the paper.
    - It seems that most of the results presented have been known in the field for a long time?

    - Typos and general grammar issues, e.g., "After pre-trained, we trained a new decoder using", etc.
    - Minor: fig. 4 use of accuracy may be a bit misleading, since only 10 samples are used (it is also not clear whether the full test set is used, or also only 10 compressed representations like for the training set). In this context, `accuracy' can only take 11 possible values.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper carries on an analysis on the catastrophic forgetting issue in neural networks. The authors claim that the provided explanation, referred to as **Forward explanation**,  sheds light into the forgetting problem.
Strengths: Investigating the catastrophic  forgetting issue is very relevant to the community.
Weaknesses: Despite the relevance of the topic, the paper contributions are very poor. The paper lacks a proper structure and the experimental  investigation is not sufficient for a conference like ICLR. There are many bold claims that are not supported by sufficient experiments (the conducted experiments are not well described and are extremely simplicistic). 
Even the main intuitions are misleading (e.g. dissimilar representeation are referred to as **interleavead**, which stands for the opposite) or already known from the dawn of the field [1,2]

The authors only give intuitions about the main concepts without any experimental validation: in the the Task Representation Convergence section, the authors claim that  *"the parameters of the decoder converge to the expectation of the representation"* . What does this mean? 

Similarly, the procedure of training the decoder  using the *expected representation*  is only mentioned without a proper definition.
Overall, the paper lacks a sufficient structure and carries on a story-telling rather than a proper and rigorous theoretical/experimental analysis.

[1]  McCloskey, Michael, and Neal J. Cohen. "Catastrophic interference in connectionist networks: The sequential learning problem." Psychology of learning and motivation. Vol. 24. Academic Press, 1989. 109-165.

[2] French, Robert M. "Catastrophic forgetting in connectionist networks." Trends in cognitive sciences 3.4 (1999): 128-135.
Recommendation: 3: reject, not good enough
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: In this work, the authors try to understand the working on NNs to understand better why NNs catastrophically forget. They conduct experiments on mnist and cifar-10 to show how representation changes over time on various tasks.
Strengths: 1.	Tackles important problem of understanding forgetting in NNs.
Weaknesses: 1. Weak Related work section
2. Novelty is limited 
3. Writing can be improved-- Text is copied from some thesis, as it talks about chapter, conclusion 1, 2 etc, and none of those are mentioned in the main paper.
4. The experimental section is weak.
5. No validation set in the code shows that the hyper-parameter is optimized on the test set, which would create a biased outcome
6. The test set is shuffled
testloader = data.DataLoader(DATASET(train=False,task=task),batch_size=64,shuffle=True,num_workers=4), why would you shuffle the test set?
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This work investigates catastrophic forgetting through empirical analysis of representation changes in neural networks across training iterations. The authors examine both penultimate-layer representations (termed 'encoder') and activation patterns throughout the network architecture. A notable observation is that final-layer representations develop class-discriminative properties, though this finding is somewhat expected given that the output layer functions as a linear classifier with improving accuracy. While the investigation into catastrophic forgetting and neural network training dynamics is conducted with reasonable depth, the contributions lack substantial novelty and suggest incomplete familiarity with the existing literature. The presentation style, though readable, employs colloquialisms and casual language patterns more aligned with expository writing than rigorous scientific discourse. Additionally, there are inconsistencies in technical terminology and several grammatical errors throughout. These factors collectively indicate that the paper requires substantial revision before meeting the standards expected for a formal academic venue.
Strengths: The authors conduct a comprehensive examination of catastrophic forgetting and associated aspects of neural network training with appropriate rigor. The manuscript demonstrates clear and accessible writing that engages readers effectively.
Weaknesses: The core findings appear to lack originality, with evidence suggesting limited engagement with relevant prior research in the domain. Terminology is used inconsistently (particularly 'encoder/decoder'), and several assertions are overstated given the existing body of knowledge. The writing style relies on informal constructions and imprecise language unsuitable for academic publication, including phrases like 'good news and bad news' and awkward formulations such as 'To address this issue, we first introduce a definition here.' The dichotomy of the network into encoder/decoder components appears somewhat arbitrary and provides limited analytical value. Multiple spelling errors and grammatical mistakes are present throughout (e.g., 'After pre-trained, we trained'). Technical issues include potentially misleading accuracy reporting in Figure 4 using only 10 samples, with unclear specification of whether the full or subset test set was employed—particularly problematic given that accuracy can only assume 11 discrete values under these constraints.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair