Investigating Layer Importance in Large Language Models
Authors: Yang Zhang, Yanfei Dong, Kenji Kawaguchi


Reviewer paper summaries: This paper investigates the significance of Attention (ATTN) and Feed-Forward Networks (FFN) in each layer of the latest Large Language Models (LLMs), specifically, LLama3 and Mixtral, utilizing Shapley values with a truncated subset and single-layer ablation. The findings reveal that the initial layers contribute substantially to the models' performance across most datasets. In contrast, the removal of subsequent layers results in only marginal changes to the outcomes.
Summary of strengths: * The paper is written with clarity and precision, making it accessible and understandable.
* The narrative is well-structured, ensuring a logical progression of ideas.
* The study employs a timely choice of models.
* By incorporating various datasets, the paper demonstrates the generalizability of its results.
Summary of weaknesses: The paper's related work sections in both the main text and the appendix could benefit from placing the current paper in the context of existing research, highlighting how it differs from and adds to those works. This would help clarify its novelty and contribution to the field.
Comments, suggestions and typos: The pie chart in Figure 1 is somewhat confusing due to the varying positions of FFN and ATTN for different layers across each subplot. Clarifying the layout or using a different visualization method could enhance understanding.
Overall assessment scores: 3