Kati8/ICLR_2023_2025 huggingface dataset:
authors: Michael Kleinman
authors: Alessandro Achille
authors: Stefano Soatto
reviews: 
Comments: This paper demonstrates that critical learning periods in deep learning models depend on the depth of the model and the structure of the data distribution, offering an analytical perspective on the learning dynamics. This study provides both analytical and simulation evidence that the learning of features is closely tied to the competition between different sources of information, highlighting the competitive dynamics within the learning process.
Strengths: 1. This paper is well-written, technically solid and has a concrete flow.
2. This work offers insights into the reasons behind the emergence of critical learning periods in both biological and artificial networks.
Weaknesses: 1. I think the assumption of the deep linear network is a bit strong. Since in real-world applications, most neural networks require non-linear activations. A deep linear network can be actually approximated by a single-layer linear network.
2. Critical periods in artificial deep neural networks (DNNs) may be due to specificities of the optimization process, such as an annealing learning rate, or from defects in the artificial implementation and training, like ReLU units becoming frozen or gradients vanishing.
3. What's the model's generalization ability on more sparse data settings?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: This paper is a continuation of the study of critical learning periods in deep learning. This work focuses on why critical learning periods emerge in deep linear network models. The paper investigates the critical learning period that depends on the depth of the model and structure of the data distribution, with supporting experiments. Meanwhile, they also show the learning of features is tied to competition between sources. They analyze the impact of pre-training on some tasks, and there are some simple experiments used to prove the “critical learning periods”. This work provides analytical understanding of critical periods in deep linear networks and draws connections between artificial and biological learning.
Strengths: 1. This paper continues previous work (on the deep network), the experiments correspond to research on the depth of the network, data distribution, competition between sources, and pre-training.
2. This work on studying the competition of different data sources is solid, and it seems to be a relatively good job through the linear multi-pathway network.
3. Analytical and minimal models provide fundamental insight. Intuition and empirical observations match well.
Weaknesses: 1. The paper is less readable and requires a higher theoretical foundation. The description of the “linear multi-pathway framework” (Sec.3.1.) is not clear enough and lacks corresponding details.
2. Only studying deep linear networks seems not enough to clearly understand why deep networks have critical periods, and network depth and data sources cannot guarantee sufficient persuasiveness.
3. There are fewer categories of experiments, and it would be better if the authors could provide more types of experiments to prove their claims.
4. For the experiments related to pre-training, I cannot guarantee sufficient correlation with key learning practices. I hope I can explain the motivation for doing this part of the experiment. By the way, this is not to say that this is an obvious weakness.
5. I don’t find any support in the paper for what the abstract said: “Why critical learning periods emerge in biological and artificial networks”.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good