Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
Authors: Wataru Ikeda
Kazuki Yano
Ryosuke Takahashi
Jaesung Lee
KeigoShibata
Jun Suzuki
Venue: COLM 2025 (2025)
Keywords: Feed-Forward Networks
Model Architecture
Knowledge Representation
Pre-training
TL;DR: FFNs in 70% of the consecutive middle layers of Transformer-based LM contribute more to model performance than other layers.

Abstract: This study investigates the layerwise importance of feed-forward networks (FFNs) in transformer-based language models during pretraining.
We introduce an experimental approach that, while maintaining the total parameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers.
Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch to examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done.
Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70\% of the consecutive middle layers consistently outperforms standard configurations for multiple downstream tasks.

Published as a conference paper at COLM 2025
Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
Wataru Ikeda", Kazuki Yano", Ryosuke Takahashi", Jaesung Lee", Keigo Shibata", & Jun Suzuki
"Tohoku University, RIKEN,, 'NIT LLMC
i ikeda. wataruêdc. tohoku.ac.
Abstract
This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that, while maintaining the total pa- rameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers. Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done. Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70% of the con- secutive middle layers consistently outperforms standard configurations for multiple downstream tasks.
1 Introduction
Language models based on Transformer architectures (Vaswani, 2017) have rapidly evolved and are now central research topic in the fields of natural-language processing and artificial intelligence. In such Transformer-based language models (hereinafter, "Transformer LMs"), many detailed model designs have been proposed and implemented. Conceptually, most Transformer LMs comprise stacked Transformer layers, each containing two main components: a self-attention mechanism and feed-forward network (FFN) (Vaswani, 2017; Touvron et al., 2023). Figure (a)) illustrates a standard Transformer layer. The computational process follows a specific pattern, particularly the Transformer LMs with the pre-language- normalization (pre-LN) (Xiong et al., 2020a). Each layer sequentially processes the input vectors through both self-attention and FFN components, and the resulting output vectors are added to the original token embed dings through residual connections. This process is repeated across all layers to progressively refine the representation and ultimately produce the final hidden-state vectors.
Numerous previous studies have attempted to investigate the roles of the self-attention and FFN components, both individually and simultaneously, to understand what Transformer LMs compute internally. Most studies conclude that the self-attention mechanism mainly handles the mixing of information obtained from token embeddings, while the FFN pri- marily serves to store knowledge from the training data (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022).² Assuming that FFN layers embed knowledge, functioning similarly to key-value memory, many questions remain, such as whether FFNs are the most effective form for acquiring knowledge and where exactly within the multiple layers of a Transformer this knowledge embedded.
To better understand the role of FFNs within Transformer LMs, we use an original approach to uncover some of their roles and functions. More Mpeciiiiilllllyy instead of evaluating publicly
Although some empirical and theoretical studies (e.g., Kobayashi et al. (2024)) have demonstrated that FFNs may include other functions and effects, they do not prove that FFNs do not store knowledge.

---

Published as a conference paper at COLM 2025
Baseline Model
Experimental Model
FEN
FEN FEN
DOOOO
OOOOOOOOO
Self-Attn
Self-Attn
SelF-Attn
(a) Standard Layer
(b) FFN-expanded Layer
(c) FFN-deactivated Layer
Figure 1: Layer Structure of the Baseline Model and Our Experimental Model. (a) Stan- dard Transformer layer: The baseline model is standard stack of Transformer layers as implemented in LLaMA. (b) FFN-expanded layer: In the experimental model, certain layers have an expanded intermediate representation dimension in the FFN. (c) FFN-deactivated layer: In the remaining layers, the FFN is removed. In our experimental model, while maintaining the overall parameter count of the baseline model, the FFN's computational capacity (i.e., the number of parameters) is concentrated in specific layers.
available (static) pretrained models, we evaluate Transformer LMs trained from scratch with several nonstandard FEN configurations within the Transformer layers, such as models with removed or enlarged FFNs. By comparing the task performance of such nonstandard FFN configurations, aim to clarify whether the importance of FFNs depends on their positions within the Transformer LMs during the pretraining phase.
Related Works
Several studies have demonstrated that FFNs store knowledge and that specific neurons play an important role in representing and recalling factual information (Geva et al., 2021; Dai et al., 2022; Meng al., 2022). Additionally, Kobayashi et al. (2024) proposed different interpretation of the role of FFNs, showing that FFNs, together with layer normalization, contribute to contextualizing inputs.
More recently, studies have analyzed LMs at the layer level and have reported that mid- depth layers provide robust representations, whereas the final layers tend to overspecialize toward the pretraining objective (Skean et al., 2025).
While many important insights have emerged, these findings derive from analyses of pretrained LMs and are limited to standard model architectures. The verification method used in the present study original because it examines the impact of modifying the model architecture itself, such as removing FFNs from specific Transformer layers or increasing the dimension of the remaining FFNs. Adopting such a structural approach model investigation rather than conventional analytical methods should lead to new insights into the functions and roles of FFNs.
3 Model Settings for Identifying the Position-based FFN Importance
This study examines the importance of FFNs as function of their layer position within a Transformer LMM To focus our investigation, we limit this study to the LLaMA architec- ture (Touvron et al., 2023), which has become the de facto standard model for Transformer LMs. Specifically, LLaMA integrates modern architectural improvements, including pre- LN (Xiong et al., 2020b), SwiGLU activation function (Shazeer, 2020), and RoPE positional encoding.
Based on this model architecture, Transformer layer, which in this paper refers to each layer in a Transformer LM, primarily consists self-attention mechanism followed by an FFN. Although the Transformer layer includes layer normalizations with pre-LN setting,
2

---

Published as a conference paper at COLM 2025
Final Layer mom
Final Layer
Middll Layer
Middle Layer
First Layer
First Layer
first
middle
final
10%
50%
90%
Figure 2: Different Positional Configura- tions of the FFN-extended Layer. In our experimental model, the FFN-expadded layer is placed in one of three positions: near the input layer (first), in the mid- dle layers (middle), or near the output layer (final) and we evaluate the effects of these different placements.
Figure 3: Different Placement Ratios of the FFN-expanded Layer. In our experi- mental model, vary the proportion of FFN-expanded layers and evaluate their effects. For instance, when placing FFN- expanded layers near the input layers (first), we define what percentage of all layers they represent.
we omit explicit explanations about layer normalization our subsequent discussions because these details are not critical to our investigation and analysis.
3.1 Transformer Layers Characterized by Different FFN Types
Figure shows the three types of Transformer layers used in this study: standard, FFN- expanded, and FFN-deactivated. These are characterized only by the FFN settings, which are explained below in detail.
Standard Layer (baseline models). The standard layer refers to the standard setting of the Transformer layer in LLaMA and serves as the baseline setting. The LLaMA uses the SwiGLU activation function. Let (-) be the element-wise sigmoid function whose input is vector. The FFN then takes input vector Rd, where is the dimension of the hidden input and output vectors, and processes it by expanding it to an intermediate representation dimension d; through projection matrices Wgate and Wup E xd followed by projection back to the original dimension via Wdown E Rddd;;
FFN(x) = (Swish(W,WWhh @ Wapx), where Swish(x) = xo(xx.
(1)
FFN-expanded and FEN-deactivated layer (experimental models). We define the FFN- expanded layer essentially identical to the standard layer, except that the intermediate representation dimension of the FFN, de is expanded [see Figure 1(b)]. In other words, the relation d; < de holds. We define the FFN-expanded layer FFN expanded as
FFNexpanded (x) W' Wowwn (Swish(W's W
Wupx),
(2)
where W "gtte Wip E d and W Wrown R Section 4.2 explains how we determine d'.
We define the FFN-deactivated layer as the standard LLaMA layer, with the FFN removed entirely from the Transformer layer. Note that this approach means that the FFN-deactivated layer consists only of the self-attention mechanism (see Figure 1(c)].
3.2 Layer Placements within the Transformer LMs
Using the three types of Transformer layers explained in Section 3.1, we define three types of Transformer layer placements: {first, middle, final). Conceptually, each label indicates the approximate relative position of the FFN-expanded layers. Figure 2 illustrates this configuration. More specifically, first means that assign FFN-expanded layers to the first (next to the input) up to a specified percentage of subsequent layers and assign FFN- deactivated layers to the remaining layers. Similarly, final assigns FFN-expanded layers
3

---

Published as a conference paper at COLM 2025
starting from the final (just before the output) through a specified percentage of preceding layers and assigns FFN-deactivated layers to the remaining layers. Finally, middle assigns specified percentage of the FFN-expanded layers symmetrically about middle layer at 1./2 and assigns the FFN-deactivated layers to the remaining positions. These three layer placements (first, middle, final) essentially redistribute the computational capacity and parameters within the model, removing the FFNs from some layers while expanding them in others. This redistribution can be viewed as concentrating the FFN's s representational capacity in specific layers while maintaining the same overall parameter budget.
The design of our experiment was motivated by the goal of structurally verifying during pretraining the hypothesis that FFNs store knowledge. Specifically, our primary operation of "removing FFNs" intended to examine whether FFNs in other layers may serve as alternative storage for the knowledge that would normally be stored by FFNs in specific layers under standard uniform FFN placement (in situations where FFÑs in those layers are absent). Furthermore, by combining this approach with the "expansion operation" that redistributes the parameters lost through removal to FFNs in specific layers, we investigate whether knowledge accumulation can be concentrated.
By strategically placing these modified layers, we can investigate whether certain positions within the network benefit more from enhanced FFN capacity than others tthars-that is, which layer positions allow FFNs to effectively store knowledge. By maintaining the total number of parameters in all experimental models, we can measure the effects of placement and systematically verify the layerwise nature of the FFNs' knowledge-storage function.
4
Experiments
This section explains the experiments we conducted in this paper. Using the standard pretraining procedure, we trained the Transformer LMs from scratch using different model sizes and layer counts with baseline, first, middle, and final settings. Next, we evaluated the pretrained models based on the standard benchmark datasets, which are often used to assess Transformer LMs.
A performance degradation for certain model configurations indicates that the positions in the model of the FFN-deactivated layers are important for maintaining performance. Using this approach, we investigate whether removing FFNs from certain layers degrades or improves performance, thus revealing the importance of layers to the function of FFNs.
4.1 Baseline Model Setup
We constructed three baseline models with different numbers of parameters and layer configurations to ensure that the results are robust across varying model architectures. Our baseline configurations include (1) a 285M parameter model with 12 layers, hidden dimension d of 1280, and an FFN intermediate dimension d, of 4480; (2) a 570M parameter model with 24 layers, maintaining the same hidden and intermediate dimension sizes; and (3) a 570M parameter model with 40 layers, using a smaller hidden dimension size of 992 and FFN intermediate dimension size of 3472.
The rationale behind these diverse baseline configurations is twofold: First, comparing the 285M and 570M models allows us to detect whether any trends regarding FFN importance are independent of model sizes. Second, comparing the 24-layer variant with the 40-layer variant of the 570M model enables to examine whether any observed patterns are independent of the number of layers, which is particularly important because our approach removes FFNs from certain percentage of layers. Given that major models such as LLaMA 8B have 32 layers (Grattafiori et al., 2024) and Qwen3 14B has 40 layers (Yang et al., 2025), this 40-layer configuration covers a practical range of the layer count. This multiconfiguration approach helps explore the consistency of the behavioral patterns across different model architectures.
AAppendix A provides detailed configurations of these baseline models.
4

---

Published as a conference paper at COLM 2025
4.2 Experimental Model Setup
The experimental models maintain the same basic configuration as the baseline models, including the number of layers and hidden dimensions. In the experimental models, we replace the baseline model's layers with either FFN-expanded layers or FFN-deactivated layers acg to the layer placement positions described in Section 3.2 and the ratio of FFN-expanded layers described below.
Ratio 5% of FFN-expanded layers to total layers, where r E {10, 310,30,50 50, 70, 90, 100). As illus- trated in Figure 3, the total number of FEN-expanded layers is determined by the product of the total number L of layers and r, rounded down to the nearest integer ([rL/100J).
Combining the placement positions from the previous subsection with these ratio configura- tions, we generate experimental models for each baseline model. For example, in the 285M, 12-layer baseline model with r = 30% and the mi ddle position, we place FFN-expanded layers in layers 6, 7, and 8 (because 12 x 0.3 3 layers), with the remaining layers being FFN-deactivated.
Importantly, all experimental models maintain the same total parameter count as their corresponding baseline models. This parameter parity is achieved by expanding the in- termediate dimension size d', of FFN-expanded layers to compensate for the parameters removed from the FFN-deactivated layers. The intermediate dimension d' of FFN-expanded layers is recalculated based on the ratio of the FFN-expanded layers in each experimental model configuration and determined such that the total parameter count remains nearly identical to the baseline model.
By combining the two factors above, we established 18 different experimental configurations (six ratios of FFN-expanded layers times three positions) for each baseline model (285M, 12 layers; 570M, 24 layers; and 570M, 40 layers). Note that, when the ratio of FFN-expanded layers is 100%, the model architecture is identical to the baseline architecture, so we simply use the baseline results rather than training redundant model for these configurations.
4.3 Pretraining and Evaluation
For pretraining the baseline and experimental models, we used standard pretraining methods with the FineWeb- dataset (Lozhkov al., 2024). 5 We evaluated the pretrained models in terms of the downstream task performance and knowledge capacity. For downstream task evaluation, we used the framework (Gao et al., 2024) with a diverse set of tasks: LAMBADA (Paperno et al., 2016) for contextual next-word prediction, Wikitext (Merity et al., 2017) for language modeling, Winogrande (Sakaguchi et al., 2020) and PIQA (Bisk et al., 2020) for commonsense rea- soning in a binary choice format, HellaSwag (Zellers et al., 2019) for selecting the most natural continuation of a context, and ARC (Clark et al., 2018) for scientific knowledge and reasoning. We used accuracy (Acc) as the evaluation metric for the choice-based tasks (ARC, HellaSwag, PIQA, and Winogrande), and we evaluated both accuracy (Acc) and perplexity (PPL) for LAMBADA. Finally, we used perplexity (PPL) for Wikitext.
Pretraining and Evaluation.
Knowledge Capacity Evaluation. Prior research hypothesizes that FFNs serve as knowl-
edge storage components in Transformer LMs (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022), so it is natural to investigate how architectural modifications to FFNs might affect the amount of knowledge stored in model. To quantitatively evaluate this aspect, use the Zero-Shot Relation Extraction (zsRE) dataset (Levy et al., 2017) to measure knowledge
3For the middle configuration with an odd number FFN-expanded layers, we place one more layer in the latter half of the model with respect to the middle layer (L/2).
4 'Appendix B shows the specific intermediate dimension sizes df for each experimental model.
DDetailed pretraining configurations are provided in Appendix
6 Hereinafter, we refer to Winogrande as WinoG.
7 7RC consists of two subsets: the Fasy set and the Challenge set, which we refer to as ARC-e and
ARC-c, respectively.
5

---

Published as a conference paper at COLM 2025
capacity, following the methodology of Mitchell et al. (2022); Cao et al. (2021). The dataset consists of knowledgeebaset question-answer pairs, allowing us to evaluate the model's ability to retrieve factual information." 8
Performance Metrics. For each evaluation result, we calculate the relative improvement (RI) with respect to the baseline to facilitate comparison between the experimental models and the baseline. This calculation is done as follows:
mI T) s(T) metric(m, T) metric(baseline, 100[%]
(3)
metric(baseline, T)
where sT) is a sign-correction factor defined as
{1 if task T uses accuracy-based metrics
sT)) (-1 if task T uses loss-based metrics (e-g., perplexity)
(4)
Here, metric(m, T) gives the metric of experimental model m applied to task T, and metric(baselin T) gives the metric of the baseline model applied to task T. The sign- correction factor s(T) ensures that a positive RI consistently indicates better performance regardless of whether the underlying metric follows a "higher-is-better" convention (e.g., accuracy) or a "lower-is-better" convention (e.g,, perplexity). Zero RI indicates that the performance is equivalent to that of the baseline model, positive (negative) RI indicates that the performance exceeds (is inferior to) that of the baseline.
Results
Figure compares the RI of various FFN configurations with the RI of the baseline." To ensure fair comparison, we excluded the results of certain downstream tasks when either the baseline model or experimental model performed below the chance level because such results would not provide meaningful insights into architectural differences.
Hereinafter, we refer to individual results using the following format [ppos]_[pct].(sss [size] [layers]", where [pos] is the position of the FFN-expanded layers (first, middle, or final), [pct] is the percentage the FFN-expanded layers, [size] is the model size (e-g., 285M), and [layers] is the total number of layers. For example, aaddle_3e_28MM.. 121 refers to a model with 285M parameters and 12 layers, where 30% of the layers are FFN-expanded layers positioned in the middle of the architecture.
In this section, we analyze these results with respect to (1) the ratio of the FFN layers and (2) their posítion within the model.
Effectiveness FFN Expanded Layer Ratio
Our results reveal clear relationship between the ratio of the FFN-expanded layers and the model performance. Models with low layer ratios (10%-30%) consistently performed worse than the baseline model in nearly all evaluations, and the performance degradation is substantial for numerous tasks: HellaSwag produces a relative degradation ranging from -0.5% to - 6.53% excluding final 285M produces a relative degradation from 199.07% to -1 -1.57% for all model sizes [Figure 4(a)). Although LAMBADA accuracy and zsRE improve the performance with respect the baseline model in the 570M_401 configuration, the overall trend remains negative across most experimental settings [Figures 4(c) and 4(d)).
121 [Figure 4(b)), and Wikitext perplexity
As the FFN-expanded layer ratio increases, performance improves for almost all tasks. Ra- tios of 70%-90% produce a consistent trend whereby increasing the configuration produces
Appendix D provides detailed procedures for this knowledge assessment.
9 AAditiioal evaluation results for tasks not shown in Figure are provided in Appendix F. More- over, to validate the meaningfulness of our RI-based analysis, we show the absolute metric of our baseline models with metrics of models from the literature in Appendix E.

---

Published as a conference paper at COLM 2025
first
middle
final
(1) 285M, 12-layer
IRSEONAONTTT (50)
(2) 570M, 24- 24-layer
atre propremmentt. 186)
ausanded bowctl))
Perzertage
ofTTs
(3) 570M, 40-layer
(a) Wikitext (PPL)
(b) HellaSwag (Acc)
(c) LAMBADA (Acc)
(d) zsRE (Acc)
Figure 4: Relative Improvement across Tasks by FFN-expanded Layer Ratio. Relative improvement across tasks as a function of FFN-expanded layer ratio for different placement positions. Each row represents a different baseline configuration (model size and number of layers), while each column shows results for different evaluation task. In each graph, the red dashed line highlights zero relative improvement, representing performance equivalent to the baseline model. Note that at 100% ratio, all configurations converge to the baseline performance regardless of placement position.
model that outperforms the baseline model, although the gains vary by task configu- ration." 10 Particularly for the @85M_121 and 570M_401 models, most experimental models within this range of FFN-expanded layer ratio outperformed the baseline model for all tasks [Figures 4(1) and 4(3)]. These findings suggest that extreme concentration of FFN parameters in very few layers compromises model capability, likely because reducing the layers applying nonlinear transformations limits the representational capacity, even with individually larger FFNs.
5.2 Effectiveness FFN-expanded Layer Position
Focusing the 70%-90% FFN-expanded layer ratio range where the models tend to outperform the baseline model, we observe notable differences based on the position. The middle and final configurations consistently outperform the first configuration across most tasks in the 285M (12-layer) and 570M (24-layer) models.
For the 285M model, Figure 4(1)(b) shows that, at 90% ratio, the first position yields only +0.23% improvement with respect to HellaSwag, whereas the middle and final positions yield 1..4% and +2.04%, respectively. This pattern repeats for the LAMBADA accuracy
lask performance gains naturally vary due to differences in task difficulty. We address this challenge by examining the consistency of performance for multiple evaluation tasks and model configurations with different sizes and layer counts.
7

---

Published as a conference paper at COLM 2025
285M (12-layer)
570M (24-layer)
570M (40-layer)
Model
Avg RI (%)
Model
Avg RI (%)
Model
Avg RI (%)
final_90
+4.72
middle.70
+1.81
first.90
+3,41
middle_90
+4.35
final_90
+1.34
middle.70
+3.36
middle_70
+3.78
middle_50
+0.70
middle_90
+2.62
fi
+1.94
middle
+0.26
first_70
+2.35
middle 50
+1.37
final
+0.12
first 50
+2.15
Table 1: Average Relative Improvement (Avg RI) for Top Models by Model Size. Avg RI shows the mean value across six downstream tasks (Wikitext, LAMBADA, HellaSwag, zsRE, ARC-e, PIQA).
Model
Wikitext LAMBADA LAMBADA
ARC-e ARC-c
WinoG PIQA HellaSwag
zsRE Avg Acc
PPL
PPL
Acc
Acc
Acc
Acc
Acc
Acc
first.70
+0.30
0.30
+2.21
+1.89
0.60
+0.61
+0.71
0.64 +2.04 +0.69
middle.78 70
+1.09
+6.47
+2.89
+1.55
3.60
1.83
+1.27
+0.72 +3,06 +1.29
final 70
-1.71
+1.55
+0.74
-054
+3.90
1.22
+0.64
-1.25 +2.55 +0.52
Table 2: Relative Improvement (%) Experimental Models over Baseline for 1.2B Param- eter Models.
Figurre 4(1)(c) and Wikitext [Figure 4(1)(a)), where first 90 increases RI by +2.59% and -1.66% compared whereas final 90 increases by +.115% and +3.49%, respectively.
The 570M, 24-layer model produces similar trends, with middle 70 achieving +1.12% on HellaSwag, whereas first_70 achieves only 1.14% [Figure 4(2) In the knowledge assessment through zsRE [Figure 4(2)(d)], middle produces remarkable +3.80% im- provement, significantly outperforming first_70, which finishes at -.54%.
Curiously, this pattern becomes less consistent in our larger 40-layer experiments. As shown in Figure 4(3), the final configuration occasionally underperforms both the middle and first configurations. For example, final_90 achieves +4.72% improvement, whereas first. 90 achieves +8.13% in the LAMBADA accuracy [Figure 4(3)(c)..
These findings suggest that FFNs positioned in the middle to later layers contribute more to model performance than FFNs in earlier layers, particularly models with moderate layer counts (12-24).
5.3
Top 5 Experimental Models
Although our analysis reveals general trends across different FFN-expanded layer positions, identifying specific high-performance configurations is crucial. Therefore, calculated the average RI for all downstream tasks for each experimental model. Table presents the top five configurations based on the average RI for each model size. Notably, the middle 70 configuration consistently performs well for all model scales, ranking third (+3.78%) for the 285M model, first for the 570M, 24-la (+1.81 %) model, and second for the 570M, 40-layer (+3.36%) model. This consistency suggests that concentrating FFNs in approximately 70% of the layers around the center of the network a robust architectural choice.
Based on these findings, we identify middle.1 70 the most promising configuration and extend our analysis to larger models, including first_70 and final_70 for comparison to further validate the effect of position when scaling.
5.4 Scaling to 1.2B Parameter Models
To further validate our findings and determine whether the observed patterns persist at larger scales, conducted additional experiments with 1.2B parameter model using
8

---

Published as a conference paper at COLM 2025
40-layer configuration." 11 Table 2 presents the RI of each configuration compared with the baseline RI across the eight downstream tasks. The middle_70 configuration produces the highest average improvement (+1.29%) for all tasks, outperforming both first 70 (+0.69%) and final 70 (+0.52%). This pattern is consistent with our results for both the 285M and 570M models, for which the middle_70 configuration consistently ranks among the top-performing models.
Considering individual tasks, the middle_74 70 configuration excels particularly in language modeling and knowledge-intensive tasks, demonstrating the biggest improvements on Wikitext (+1.09%), LAMBADA (PPL: +6.47% Acc: +2.89%), and zsRE (+3.06%).
These results from the 1.2B parameter model provide strong evidence that concentrating the FFN parameters in specific layers rather than distributing them uniformly across all layers can significantly improve the downstream task performance. Our experimental approach notably demonstrates that the middle_70 configuration (i.e., concentrating FFNs in 70% of consecutive middle layers) consistently performs best for model scales from 285M through 570M to 1.2B parameters. This remarkable consistency across different model sizes suggests that the advantage of strategic FFN layer positioning represents fundamental architectural property of Transformer LMs rather than a scale-dependent phenomenon.
The underlying rationale for these results is consistent with prior research showing that the most significant information processing for downstream tasks occurs primarily in the mid-to-final FFN layers of the model (Meng et al., 2022; Geva et al., 2021). Our middle. 70 configuration effectively concentrates the parameter budget on the parts the model that matter most for downstream tasks, thereby utilizing the limited parameters more efficiently.
Layerwise Importance Analysis
To quantify the contribution FFNs in each layer to the overall model performance and metric
visualize the layerwise importance of FFNs, we developed layerwise-importee derived from the experimental results of Section 5. This metric is based on the idea that performance degraded upon removing specific layer's FFN, that layer's 's FFN must be particularly important to the model's capabilities.
To analyze different configurations of the FFN-expanded layers, we designed our importance metric based on a methodological starting point and a computational procedure. First, given the technical challenge of directly quantifying the individual contribution of each layer within specific configurations, we assume that when a set of FFN-deactivated layers degrade performance in terms of RI, this degradation is spread equally among all deactivated layers in that configuration. Second, for each layer, we sum its importance over all configurations where it was deactivated through normalized average.
For example, consider layer index 2 in our 570M, 40-layer model in the final_50 configura- tion. With this setting, layers 1-20 are FFN-deactivated, and this configuration produces an average RI of -2.04% over all evaluation tasks. Applying our first assumption, we attribute an importance of +0.102% to layer 2 (and the other 19 layers) from this configuration be- cause the - 2.04% degradation is distributed equally among the 20 deactivated layers (i.e., 2.04/20- = 0.102). This process is repeated for all configurations where layer 2 is deactivated, and the results are averaged to obtain the final importance score.
The metric is designed such that higher values indicate greater importance. When FFN deactivation in certain layers leads to larger performance drops in downstream tasks and knowledge assessment, those layers are assigned higher importance scores.
The bar plot in Figure 5 shows the computed importance scores for different layers, where each score has been standardized (zero mean and unit variance). Positive values (shown in blue) indicate layers where FFNs exert an above-average importance on model performance,
11 "Detailed model configurations and pretraining configurations are provided in Appendices and C, respectively. The detailed mathematical derivation of this metric is provided in Appendix H.

---

Published as a conference paper at COLM 2025
1/2
L/2
L/2
Layerwwise Imporaacce
Layorarse Importance
105
-1
-10
1.0
-16
-1.
-20
10
15
Layer Index
Layer Index
Layer index
(a) 285M, 12-layer
(b) 570M, 24-layer
(c) 570M, 40-layer
Figure 5: Layerwise Importance Scores. The horizontal axis represents the layer index, and the vertical axis represents the corresponding standardized importance score, where higher values indicate that the layer is more important.
whereas negative values (shown in red) indicate layers where FFNs exert below-average importance. The magnitude of each bar reflects the importance of the FFN at the given layer with respect to the importance averaged over all layers.
Analysis of the layerwise importance scores revealed several key patterns across the model scales. First, all three configurations demonstrate clear concentration high-importance layers in the middle portion of the networks, while the very first and final layers consistently show below-average importance. In the 12-layer model [Figure 5(a)], layers 3-10 show positive importance scores. Similarly, layers 7-18 of the 24-layer model [Figure 5(b)) are highly important. The 40-layer model [Figure 5(c) produces positive importance scores for layers 5-28. Second, the distribution of layer importance shifts systematically the model depth increases. To illustrate this pattern, Figure 5 includes black dotted lines marking the middle position (L/2) for each model configuration. Examining the importance distribution relative to this midpoint reveals a clear trend: the 12-layer model concentrates importance somewhat toward the latter half of the network, the 24-layer model produces more balanced importance distribution about the middle with a slight bias toward the latter half, while the 40-layer model shifts importance toward the earlier portion of the network. This progressive forward shift in the FFN importance distribution, from 12 to 24 to 40 layers, suggests that, as a model deepens, FFNs may become more effective when positioned earlier in the network architecture. This phenomenon might occur because, in deeper networks, hidden states execute more self-attention functions before reaching the middle layers, potentially resulting in overcontextualized representations that FFNs may struggle to process effectively.
7 Conclusion
This paper investigates the layerwise importance of FFNs, one of the component elements of Transformer LMs, focusing on their position-dependent significance within the overall model architecture during the pretraining process. By evaluating multiple models and various layer sizes, we found that concentrating FFNs in 70% of the consecutive layers around the middle of the Transformer LMs tends to yield superior performance for multiple downstream tasks compared with the baseline model using the standard FFN configuration. Interestingly, these results also suggest that FFNs in the first and last few layers may be redundant and that their functionality can be replaced by FFNs in the middle layers. These results suggest that optimized model configuration exists other than simply placing FNNs evenly in each Transformer layer. We hope that the results of our experiments and our new findings will encourage further model analysis and the development of new Transformer LM configurations.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This work was supported by the "R&D Hub Aimed at Ensuring Transparency and Reliability of Generative AI Models" project of the Ministry of Education, Culture, Sports, Science and Technology, and JST Moonshot R&D Grant Number JPM]MS2011-35 (fundamental research).
In this study, we mainly used ABCI 3.0 and the computer resource offered by Research Institute for Information Technology, Kyushu University under the category of General Projects. ABCI 3.0 provided by AIST and AIST Solutions with support from "ABCI 3.0 Development Acceleration Use". Additionally, we partially used "mdx: platform for building data-empowered society" for part of this research work.
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL wwwphhtps///aaaeeee .rg/ ht//bb//2200 16609.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxin org/abs/ 001.001111 373..
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6491-6506, Association for Computational Linguistics, 2021. doi: 10.18653 1116633/11/221. EMNLP-MAIN.52. URL https::/too.i..ooo /ttp:////.o..r////.../ 18653/v1 22/53//112221.111 mamlp-main... 522.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, 0bs/1803.00457, 2018. URL http://aaxxl..org/bbb/ 05457.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neu- rons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume Long Papers), ACL. 2022, Dublin, Ireland, May 22-27, 2022, pp. 8493-8502. Association for Computational Linguistics, 2022. doi: URL hnttps://dol: www.oo.g//o/../.. 11653/11/2222..1..2 acl-.lng..1115555
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. wtt.://nnddo..ggeeerrr////////// 12688002.
11

---

Published as a conference paper at COLM 2025
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp- 148445499 Association for Computational Linguistics, 2021. doi: 2206663///2222.EMNVV emnlp-main. 446.
446. URL https://dol http://oo..rgg////// org/10. 18653/v1/2021.
Aaron Gratta Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chem, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappal Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamín Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo WU, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic,
12

---

Published as a conference paper at COLM 2025
Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa lamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, llias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg,June Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim shev,Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024 URL https://arxxv.orr//bb/////// .27833.
Wang, Kai Wu, Kam Hou U, Karan Saxena,
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: htt.htttp/ppper...... /ttpp//apppppa/aa/eee nips. cc/paper files/paper /ateeere/222/aaae
13

---

Published as a conference paper at COLM 2025
html.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference Language Modeling, 2024. URL https: http////ppeeeeeeeeeeeeeeeeeeeeee nett
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Analyzing feed-forward blocks in transformers through the lens of attention maps. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https://openreeiiew wttpe/rwwwwwiieweefeeeeeeeee
net wttp://wwwe..oooommmomooo//srraa
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction Via reading comprehension. CoRR, abs/ 1706.0411,, 2017. URL http://arxiv.. http://aaxxvv..rgaabbba/e org/abs/ 1706. .26155111.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://hugging:: wtt.r/naaacccoocatttttttttte
HusginggaaceFW/Lineeet
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations GPT. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, uSA, November 28- December 9, 2022, 2022. URL h papers. nips. cc cc/paper /paper_files/ _files/ 2tte://22222mmas/11f11aaaaaa2222 (45117/4555222273366663333333333
paper/2022/hashh 6f
html.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openrevview net weee//orrridddByy22iee
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Man- ning. Fast model editing scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openrevieww net
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https:/ http:://arxi..or//bb///222
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset, Aug 2016
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Fhirty-Fourth AAAÍ Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposiumm on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY, USA, February 7-12, 2020, pp. 8732-8722 8740. AAAI Press, 2020. doi: 10.1609/ AAALV34105.6399 URL httpt::t//opi.oooo http:///doo.ogg///./// (609//aai v34i05. 6399.
Noam Shazeer. GLU variants improve transformer. CoRR, abbs/2002.05202, 2020. URL
https://arxiv.ort wwtp:iii.org/////////e/ 05202.
14

---

Published as a conference paper at COLM 2025
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by Layer: Uncovering Hidden Representations in Language Models, 2025. URL http::///aaniv.orrga///222222 02013.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/ 1302.13971 2023. doi: 10,48550/ARM) 2302.13971. URL https: /twii//o..oo/// org/10 48550/ arviv. 2302. 13971.
Vaswani. Attention you need. Advances Neural Information Processing Systems, 2017.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the trans- former architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learn- ing Research, pp. 10524-10533. PMLR, 2020a. URL http://proceedinngg mlr. press/v119/ xiong20b.. html.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Yan Liu. On layer normalization in the trans- former architecture, 2020b. URL https: / wwwp://////abb/////////
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxivv https://arxv..rr/aab/2228., 01068.
15

---

Published as a conference paper at COLM 2025
A
Detailed Model Configuration
285M, 12-layer
570M, 24-layer
570M, 40-layer 1.2B, 40-layer
Layers
12
24
40
40
Hidden Dimension
1280
1280
992
1440
Intermediate Dimension
4480
4480
3472
5040
Attention Heads
16
20
Key/ Value Heads
20
20
16
20
Activation Function
SwiGLU 50257
Vocabulary Size
Table 3: Hyperparameter Configurations for Model Architectures.
B Intermediate Dimensions and Layer Placements of Experimental Models
The intermediate dimension d' of FFN-expanded layers is recalculated based the ratio of the FFN-expanded layers in each experimental model configuration and determined such that the total parameter count remains nearly identical to the baseline model. Table 4 shows the intermediate dimension d' of FEN-expanded layers for each ratio of FFN- "FN-expanded layers. Note that when the ratio of FFN-expanded layers is 100%, the configuration is equiv- alent to the baseline model, so the dimensions correspond to those shown in Appendix A.
C Pretraining Settings
Table presents the detailed pretraining configurations. To enable comprehensive explo- ration under computational resource constraints, this study conducted training with 20 times the number of tokens relative to model size, following the Chinchilla optimal (Hoffmann et al., 2022).
Regarding learning rate, for the 285M and 570M (24-layer) models, we adopted x 10 which is consistent with models reported in the literature including Pythia 410M (Biderman et al., 2023), OPT 350M (Zhang et al., 2022), Qwen 1.8B (Bai et al., 2023), OLMo 2 7B (OLMo et al., 2025), and Llama 3 8B (Grattafiori et al., 2024). For the 570M (40-layer) model, we initially experimented with 3 10 but observed loss spikes and training instability in some experimental configurations. Since our study requires comprehensive comparisons across all configurations shown in Figure 4, we adopted 1 10-4 to ensure stable training across all settings for fair comparison. For the 1.2B (40-layer) model, we also set the learning rate to 1 x 10 based on this observation.
Note that 1 x 10 4 is not extremely small value, reference models such Pythia 1.4B (Biderman et al., 2023) and OPT 1.3B (Zhang et al., 2022) use 2 10-4 placing our choice within a reasonable range.
D Evaluation of Knowledge Capacity
Each instance the Zero-Shot Relation Extraction (zsRE) dataset (Levy et al., 2017) consists of a knowledge-based question and its corresponding answer pair. In the task designed to measure knowledge capacity (zsRE task) (Mitchell et al., 2022; Cao et al., 2021), during evaluation, the model is provided either with only the question with both the question and portion of the answer and is tasked with generating the subsequent token.
Specifically, the process begins by prompting the model solely with the question to generate one token, which is then compared to the first token of the answer. Subsequently, the first token of the answer is appended to the original prompt to form second prompt; the model
16

---

Published as a conference paper at COLM 2025
Model
Ratio of FFN-expanded layers (%)
Intermediate dimension dj
285M, 12-layer
10
53765
30
17921
50
8961
70
6721
90
5377
570M, 24-layer
10
53765
30
15361
50
8961
70
6721
90
5121
570M, 40-layer
10
34723
30
11575
50
6945
70
4961
90
3858
1.2B, 40-layer
70
7201
Table 4: Intermediate dimensions di of FFN-expanded Layers for Each Experimental Models.
285M, 12-layer 570M, 24-layer 570M, 40-layer 1.2B, 40-layer
Global Batch Size
288
560
560
1152
Peak Learning Rate Tokens Laerning Rate Scheduler
3 10
3 x
x 10
x 10
5.8B
11B
11B
23B
cosine
Sequence Length
1024
Training Steps
20000
Warmup
1000
Table 5: Hyperparameter Settings for Pretraining.
then generates one token, which is compared to the second token of the answer. This process iterated until the entire answer has been generated.
The proportion of matching tokens computed relative to the complete answer is defined as the accuracy for that instance, and the average accuracy across all 19086 instances is then used indicator of the model's knowledge capacity.
E Baseline Model Performance and Literature Comparison
Since this study compares experimental models with baseline models using relative im- provement (RI), we present the absolute performance of baseline models and comparison results with literature models to ensure the validity of RI-based discussions.
Baseline Model Performance Table 6 shows the absolute performance of each baseline model on all downstream tasks used for evaluation as described in Section 4.3.As mentioned in Section 5, for the 285M and 570M model sizes, some of the baseline and experimental models did not achieve metric values above chance level (ARC-c: 0.25, Winogrande: 0.50) for ARC-c and Winogrande tasks, we excluded them from our discussion for fair comparison.
Comparison with Literature Models While direct performance comparison with litera- ture models challenging because many recent models employ extensive computational validated the appropriateness of our model
resources and often involve overtraining, performance using Pythia models (Biderman et al., 2023), which provide numerous interme- diate checkpoints specifically for research purposes.
17

---

Published as a conference paper at COLM 2025
Model
LAMBADA Wikitext ARC-CARCH HellaSSagg LAMBADA PIQA WinoGzsin
PPL
PPL
Acc
Acc
Acc
Acc
Acc
Acc
Acc
baseline_285m.
87.6
35.9
22,9
55.6
30.8
26.2
64.7
49.5
16.7
pythia-11Om-step3000
87.3
46.8
18.8
41.2
27.0
26.3
60.0
50.7
14.5
baseline mewelinee.70m.... 51mmm_70m444
41.9
28.2
34.1
32.9
67.6
49.5
baseline 500mm577mmm000
75.0
34.1
23.5
56.4
31.6
26.7
66.1
53.0
16.8
pythia-1b-step5000
30.8
30.4
18.0
44.7
29.1
34.8
62.7
52.6
18.2
basel ine 1b 401
34.2
25.4
28.4
62.4
35.9
34.3
68.5
51.9
19.6
Pythia-14A-step11000
15.6
22.6
21.8
50.0
31.9
44.6
66.6
49.4
20.8
Table Absolute Performance of Baseline Models and Pythia Models.
Specifically, we evaluated Pythia-410M (300M non-embedding parameters), Pythia-1B (806M non-embedding parameters), and Pythia-1.4B (1.2B non-embedding parameters) available on Hugging Face Hub, corresponding to our 285M, 570M, and 1.2B models, respectively, using checkpoints trained with equivalent token counts.
Table presents the comparison results. Across all model sizes and evaluation tasks, our baseline models achieve performance equal to or superior to the literature models (Pythia). Even considering that the Pythia models used for comparison were intermediate checkpoints and may not have fully converged, these results clearly demonstrate that the performance of our baseline models under our training settings falls within a thoroughly reasonable range.
These results validate the reliability of experimental findings and architectural compar- isons, ensuring the validity of RI-based discussions.
F Additional Evaluation Results
Figure presents the results across all downstream tasks not shown in Figure 4.
G
Consistency of Results under Over-training Conditions
In this study, conducted pre-training following the Chinchilla optimal (Hoffmann et al., 2022) with 20 times the number of tokens relative to model size to enable comprehensive exploration under computational resource constraints. However, many recent models em- ploy over-training using large-scale computational resources. While over-training deviates from compute-optimal settings, it known to potentially achieve higher performance improvements. Since our training configuration may be analyzing models at stage where performance has not fully converged, is necessary verify the consistency of results under longer training periods.
Therefore, in this section, we validate that the main findings presented Section maintain consistency under over-training conditions. Due to computational resource constraints, we only focus on 285M (12-layer) and 1.2B (40-layer) models, conducting experiments under conditions with significantly extended training tokens for each of the baseline, first_70, middle.76, and final_70 configurations.
G.1 Modifications to Experimental Settings
For this experiment, we modified the training configuration from Appendix C in the follow- ing two aspects:
Total Training Tokens. For the 285M model, we 8.8B tokens (approximately 20 times the model size of 413M including embedding and unembedding parameters) 1x Chinchilla, and conducted training with 1x, 2x (17.6B 6B tokens), 4x Y3..2B tokens), and 8x (70.4B tokens). For the 1.2B model, we set 26B tokens (approximately 20 times the model size of 1.3B
18

---

Published as a conference paper at COLM 2025
first
middle
final
(1) 285M, 12-layer
2.0
De alive improvemnentt www.eeeentttttt (96)
Re stlve Improvementt (55)
20
0.0
-40
-0.9
70
80
13c
20
30
60
60
100
70
63
80
130
Parcontaga FFN expanded layers (8)
Percentage of FPN expanded layers (%)
Parcentaga of FFN exparded layers (S)
3.0
(2) 570M, 24-layer
Reletive improvement. (%)
Relelvee introveeentt (8)
Relative imprevement. (76)
-1.0
-1.
60
60
100
20
60
60
80
100
120
Percentage of FFN excanded byers (%)
Percentage of FFN oxpanded lyyers (h)
Percentage of FFN expanded layers (96)
3 570M 40-layer
Rulative In provvemmntt 186
Relative Impr improvementt (3s)
Relelve improveneett
0..5
-1.0
-10
-1.5
-2.0
-300
-2.
layers
layers
Percentage IPN-essanded
LAMBADA (PPL)
(f) Arc-e (Acc)
(g) PIQA (Acc)
Figure 6: Relative Improvement across Tasks by FEN-expanded Layer Ratio. Relative improvement across tasks as function of FFN-expanded layer ratio for different placement positions. Each row represents a different baseline configuration (model size and number of layers), while each column shows results for a different evaluation task. In each graph, the red dashed line highlights zero relative improvement, representing performance equivalent to the baseline model. Note that at 100% ratio, all configurations converge to the baseline performance regardless of placement position.
including embedding and unembedding parameters) as 1x Chinchilla, and conducted training with 1x, 2x (52B tokens), and 4x (104B tokens).
Learning Rate Scheduler. In this experiment, we employed the Warmup-Stable-Da Decay (WSD) scheduler (Hu et al., 2024) as the learning rate scheduler. The WSD scheduler maintains constant learning rate for the majority of training and applies decay rapidly toward the end. A key advantage of this approach is the ability to resume training from checkpoints before the cooldown phase without changing the learning rate. Consequently, when extending training steps, there is no need to train from scratch, making this method highly efficient for over-training scenarios with excessive training steps.
Note that the models in the Section 5 used a cosine scheduler as described in Appendix C. To verify that the WSD scheduler functions appropriately, compared validation loss
19

---

Published as a conference paper at COLM 2025
Loarning
Validation Loss
12350
Chinchile
Chinchills (Cosina)
4x Chinchilia (WSD)
22015
Chinchilia (Cosine)
Chinchilla (WSD)
24 Chinchalla (Casino)
42001
2.0
Chinchilia
(Cosina)
2980 12800
1003 5083
1988 1806 1868 1862 2033 0880
102
18:00
19:0
12900
1900
Iraining Step
Tramnnn Step
Figure/ 7: Comparison of WSD and Cosine Schedulers. Left panel shows learning rate curves, right panel shows validation loss curves.
285M,
12-layer
1.2B,
40-layer
Model
1x
2x
4x
8x
1x
2x
4x
Chinchilla
Chinchilla
Chinchilla
Chinchilla
Chinchilla
Chinchilla
Chinchilla
first_70
1.08
-0.99
0.51
2.59
1.39
-1.41
-2133
middle 70
+3.48
+3.12
+4.28
43.18
0.70
+0.09
+0.04
final 70
+3.23
+3.14
+3.15
+1.65
-0.88
0.04
+0.62
Table 7: Average Relative Improvement (%) under Over-training Conditions. Values represent performance averaged across the same downstream task sets used in Tables for 285M (12-layer) and Table 2 for 1.2B (40-layer) models under different training token scales.
for the 285M baseline model with fixed warmup steps (1000) and peak learning rate (3e-4), using both cosine and WSD schedulers (Figure 7). The results confirmed that the WSD scheduler achieved lower loss than the cosine scheduler, validating that discussions be conducted within an adequate performance range when adopting the WSD scheduler.
G.2 Results
Table 7 presents the results of over-training experiments for each configuration. We con- firmed that the main findings regarding the layerwise importance of FFNs remain consistent even under conditions with significantly increased training tokens. Specifically, examining Table 7, the middle 70 configuration demonstrates consistent advantages: for the 285M 12-layer model, it outperforms other configurations across all training token scales from 1x to 8x Chinchilla; for the 1.2B 40-layer model, while the performance gains are modest, the middle_70 configuration achieves positive average relative improvement from 2x to 4x Chinchilla training, indicating superior performance on downstream tasks compared to the baseline. These results demonstrate that our key insights remain robust even under conditions that exceed compute-optimal training settings.
H Layerwise Importance Metric
Here, provide the detailed formulation of the layerwise importance metric described in Section For each layer I, we first calculate a raw importance score based on the performance impact when that layer's FFN is deactivated:
20

---

Published as a conference paper at COLM 2025
1
Raw (ww.Importance()
E ((p))
(5)
Ci
(pr)ES
R((p.r)
if E (pv)
(pr) (1)
D Pirii
(6)
otherwise
where:
r) is the average relative improvement across evaluation tasks for a configura-
tion with position p and ratio
(pr) is the set of FFN-deactivated layers in configuration (p,r)
D (pr) denotes the number of FFN-deactivated layers in configuration (p,r)
C is the number of configurations where layer was deactivated
S represents all FFN placement configurations, defined by position
p E [first,midee middle, final] and ratio r E (10%, 30%, 50%, 70%, 90%]
To facilitate comparison across different model sizes and configurations, we standardize these raw importance scores. Let H and o be the mean and standard deviation of the raw importance scores across all layers. The final standardized importance score for each layer is given by:
Importance()) (I) Raw Importance()) -
This standardization ensures that the importance scores have zero mean and unit variance across all layers, making it easier to identify which layers contribute more or less than average to model performance. The standardized scores are used in the visualization presented in Figure 5, where positive values indicate layers with above-average importance and negative values indicate layers with below-average importance.
21


Reviews:
[{"review_id": "oOGiNLwPlx", "reviewer": "Reviewer_8WSS", "summary": "This paper is addressing a very fundamental and interesting problem of identifying how does the placement and the properties of FFN influences the model's overall performance. This is interesting as most of the computation and params are allocated to FFNs and not to self attention blocks. The authors have evaluated transformer-LMs which they they trained from scratch with various FFN configurations.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper studies an important and under studied problem.\n\n2) The experiments are reasonably large scale and lots of empirical results are shown.\n\n3) The results look strong and although the scaling law is not what is explicitly mentioned but it seems the best expanded configurations should work in large scale as well.", "reasons_to_reject": "1) The takeways are not clearly mentioned, and the writing is a bit weak.\n\n2) Since the authors are training the model from scratch can they show the impact of the best configuration (i.e. middle layer expansion) on pre-train val loss/ train loss. How the convergence and generalization looks like in these scenarios.\n\n3) The results in Fig 4 are not conclusive as the gains are highly dependent on the dataset.\n\n4) Not much intuition is discussed through out the paper.", "questions_to_authors": "1) When FFNs are expanded how attention behaves?\n\n2) Why in case of smaller models 285M (Table 1) first layer expansion outperforms the middle layer expansion?", "extra_scores": {}}, {"review_id": "IxRsXG0o2l", "reviewer": "Reviewer_XrFb", "summary": "This paper concentrate on the niche topic of FFN layer importance, given a compute budget. The author devices an effective searching strategy to decide FFN. The paper is mostly clearly written after one understand the notation of different models configuration. The experiment and evaluation setup are relatively solid and comprehensive for them to make claims. \n\nThe down side of the work is that it focus much on a niche topic; and it might be hard to connect it to a boarder use case like \"how would the message in the paper help people do neural architecture search?\" or \"what's the next neural architecture?\".", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Design effective searching strategies to decide on which FFN to increase size or decrease size. \n2. Found interesting observation that concentrating FFN on the 70% of middle layers will lead to best performance; and observe that this configuration holds across scale", "reasons_to_reject": "1. The authors assumes FFN-expanded are equally important. However, this likely might not be the case as reflected in Figure 5. For example, could the author redistribute the size of FFN-expended on the \"blue\" layers according to Figure 5, and obtain even better performance?\n2. Have the author used techniques from [1], would the layer importance conclusion differ?\n\n[1] Studying Large Language Model Generalization with Influence Functions", "questions_to_authors": "1. Hypothetically, do you think using a different search heuristics will arrive at the same conclusion?\n2. Figure 3, how could the percentage be 90%? if 90% FFN is expanded, where does the author reduce the parameter to make sure the compute budget is the same? \n> when the ratio of FFN-expanded layers is 100%, the model architecture is identical to the baseline\n\nI am confused that an expansion ratio of 100% is equal to baseline (no expansion). I felt the largest ratio one could do is 50%. Also, what's the strategy to remove layers? One possible explanation is: the intermediate size d'_f of FF-expanded is recalculated per ratio of  FFN-expanded layers. If this is the case that makes sense.\n3. Figure 5 axis label are not visible.", "extra_scores": {}}, {"review_id": "kNoZtjmDFn", "reviewer": "Reviewer_dJhd", "summary": "This paper investigates the importance of feed-forward networks (FFNs) in Transformers during pre-training. By studying different positioning and capacities given to FFNs (i.e. FFN-expanded and FFN-deactivated layers), the authors conclude that concentrating FFNs in 70% of the middle layers outperforms other configurations on multiple downstream tasks, including evenly placing FFNs in each Transformer layer.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Studying the mechanisms behind Transformer components is an important research direction to promote safety and increase the overall understanding of Transformers models.\n- The paper proposes a new configuration of FFNs in Transformers (as opposed to evenly placing them in every Transformer layer, as traditionally done) that works well across several downstream tasks.\n- All models were pre-trained from scratch, ensuring a controlled environment.", "reasons_to_reject": "- No reported standard deviation: The differences in performance between first, middle, and final configurations reported in Figure 4 are mostly overlapping, and it is hard to say that the trends are statistically significant without performing multiple runs and reporting standard deviations. This also applies to the results reported in Tables 1 and 2.\n- Only one architecture was used throughout the study, for which the authors state that \"the LLaMA architecture has become the de facto standard model\". In my opinion, this claim should be edited to avoid confusion since there is currently a clear trend toward mixture-of-experts (MoE) architectures (even Llama-4 now adopts MoE). I understand that there are limited resources, but the authors could have investigated additional base architectures such as the one used in GPT-2 (i.e. LayerNorm and GELU).\n- Small-scale experiments: I understand that access to computing resources may be limited, but 1.2B parameters is still considered a very small model in today's terms.\n- No hyperparameter tuning: As I understand it, there is no mention of any hyperparameter tuning for pre-training. Even though this can be related to limited resources, there are efficient alternatives to optimize hyperparameters on small proxy models (e.g. maximal update parameterization) or by simply training for a reduced number of iterations and resuming the best hyperparameter configurations. \n- The FFN-expansion method could have been further investigated. For example, regarding whether to add depth instead of width to the network. The authors claim that the \"extreme concentration of FFN parameters in very few layers compromises model capability, likely because reducing the layers applying non-linear transformations limits representational capacity\". Adding FFN layers interleaved with non-linearities could have been done to further study the impact of having fewer or more non-linearities while maintaining the same number of parameters as the width-based expansion proposed in the paper.", "questions_to_authors": "- What are the performances of the baseline architecture for the different benchmarks shown in Figure 4? The results are reported only in terms of relative improvement, but the performance at 0% improvement should also be reported for fair comparison with existing works.\n- How do such performances compare with LLMs of similar size in the literature? \n- What are the standard deviations across multiple runs for the reported results? Do the reported trends still hold in such a setting?\n- Did you explore gradually increasing the capacity of the FFN layers until the middle layer, and then reducing it gradually?\n- How can your findings be applied to MoE-based models?\n- Did you experiment with adding depth instead of width for FFN expansion?\n- Did you perform any hyperparameter tuning? In particular, how were the learning rates in Table 4 obtained?", "extra_scores": {}}, {"review_id": "xYrz3FFSiL", "reviewer": "Reviewer_Nedb", "summary": "This paper makes an investigation into the importance of feed-forward layers (FFNs) in the multi-layer Transformer models, through allocating more parameters than others.  Through experiments on up to 1.2B parameter models up to 40 layers, the authors identified that FFNs in the middle to later layers are more beneficial to achieve higher downstream tasks, and thus more important than those in shallower layers.  While existing work is exclusively stacked the same shape of layers, this paper empirically shows the potential of searching for better shapes.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper brings to the community a new insight; stacking the same shape of layers is sub-optimal, and there can be a better way.\n\n- Through a comprehensive comparison, although the search space is limited, the paper identified that FFNs in the middle layers are more important than those in the shallow and last layers, irrespective of the total numbers of layers.\n\n- It is intriguing that FFNs for some layers can be completely removed.", "reasons_to_reject": "- Experiments are not thorough.  For example, only two types of modifications, i.e., expansion and removal, have been proposed.  Why not narrow down but removal?  Why do not leave the original width at all? -> Would be taken down if the rationale of the choice of the two types is explained.\n\n- The sizes of experimented models (with 285M to 1.2B parameters) are relatively small, compared to standard sizes of decoder-only models, and thus the paper does not guarantee that the findings can be applicable to much larger models, such as 80-layer and 126-layer models investigated under the name of Llama (70B and 405B models). -> Would be taken down considering the limitation of computes, but the generalizability of the findings to larger models should be discussed further.\n\n- As listed below, the paper has several substantial issues in presentation.", "questions_to_authors": "- First of all, the paper does not explicitly state whether they are investigating encoders or decoders.  Readers can know they are decoder-only models only when they see \"LLaMA\" in line 65.\n\n- Propose method modifies several layers.  Then, can the resulting models still be called \"LLaMA architecture\" (ll.70-71) ?\n\n- How the authors come up with this \"novel\" approach (l.3)?  In other words, the rationale should be explained more precisely.\n\n- The expanded width, i.e., $d_{f}'$, in the r% layers is determined on the basis of the number of (100-r)% of FFN-removal layers.  Please precisely describe that equation.  I'm wondering if the variants models can have exactly the same number of parameters.\n\n- Accuracy and perplexity of the baseline models, i.e., the denominator for computing relative improvement (RI), are not shown, making it unclear if the RI is worth discussing and its scale.\n\n- Is it possible to automatically identify the optimal (or at least reasonable) widths allocation?  This is especially important because the paper currently only calls more labor of future research.\n\n- Some numbers in Section 5 are not corresponding to those in Figure 4; for example, +2.5% and +3.9% in l.209 look lower and higher in Figure 4 (1)(c) and Figure (1)(b), respectively.  +5.7% in l.224 looks lower than +5.0% line in Figure 4 (1)(a), and -1.8% in l.228 so does higher in Figure 4 (2)(d).  +8.5% in l.210 and %8.1% in l.232 do not match while they refer to the same point in Figure 4 (3)(b).\n\n- Although it is commonly done, I'm not still sure if one can take \"average\" of different metric scores with different scales, as in Sections 5.3 and 5.4.  Could you give a justification?\n\n- The superiority of \"middle_70\" for Wikitext, LAMBADA, HellaSwag, and zsRE is consistent with those in Table 1, and thus expected.  In contrast, how can the authors explain the largest performance drop in the new downstream tasks, i.e., ARC-c and Winogrande?\n\n- The IDs in l.280 cover 9, which does not match 10 in l.283.  Similarly, mentions about layers in ll.298-302 often do not match with those in Figure 5.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the role and importance of feed-forward network (FFN) layers within Transformer language models, focusing on how different placements and sizes of FFNs affect overall model performance. The authors conduct experiments by training Transformer-based language models from scratch with varying FFN configurations, including expanding or removing FFNs in specific layers. Their empirical results suggest that concentrating FFN capacity in approximately 70% of the middle layers yields better performance across multiple downstream tasks compared to evenly distributing FFNs across all layers. The study explores both FFN expansion and removal to find more optimal architectural configurations rather than uniformly sized layers. The goal is to provide insights into more effective parameter allocation within Transformer models to improve efficiency and accuracy.

Pros:
- The paper addresses an underexplored aspect of Transformer architecture related to FFN placement and sizing.
- It provides extensive empirical evaluation with experiments on models trained from scratch.
- The findings show that concentrating FFNs in the middle layers improves performance across multiple tasks.
- The study challenges the standard practice of evenly sized FFNs, offering new insights into efficient parameter allocation.
- The experiments include both FFN expansion and removal, broadening the scope of architectural exploration.

Cons:
- Experiments are limited to relatively small model sizes and a single base architecture.
- There is no reported standard deviation or multiple runs to assess statistical significance.
- The impact of FFN expansion on attention behavior is not analyzed.
- Hyperparameter tuning and exploration of alternative FFN modifications, such as adding depth, are missing.
- Some results are dataset-dependent and not fully conclusive.
- Reviewers complain that the presentation contains inconsistencies and unclear explanations in parts.

The meta-reviewer notes that the contents of the reviews are somewhat misaligned (i.e., more negative) with the final scores in some cases.
Decision: Accept