How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers
Authors: Xin Lu
Yanyan Zhao
Bing Qin
Liangyu Huo
Qing Yang
Dongliang Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Pre-trained Language Models
Base Capabilities
Mixture of Experts
Primary area: deep_learning_architectures

Abstract: Pre-trained language models have been proven to possess strong base capabilities, which not only excel in in-distribution language modeling but also show powerful abilities in out-of-distribution language modeling, transfer learning and few-shot learning. Unlike existing work focusing on the influence of scale on base capabilities, our work examines the influence of architecture on those. Specifically, our concern is: How does architecture influence the base capabilities of pre-trained language models? In this work, we attempt to explain and reverse the decline in base capabilities caused by the architecture of FFN-Wider Transformers, seeking to provide some insights. Through analysis, we found the contribution ratio of Multi-Head Attention (a combination function) to pre-trained language modeling is a key factor affecting base capabilities. FFN-Wider Transformers reduce the contribution ratio of this combination function, leading to a decline in base capabilities. We confirmed this by experiments and proposed Combination Enhanced Architecture (CEA) to address the decline in base capabilities of such models. Significantly, we extended our explanation and CEA to Mixture of Experts (MoE) Transformers. We successfully achieved significant improvements in base capabilities on a 14B parameter MoE model, demonstrating the practical application value of our work. This also indicates that our analysis has a certain guiding significance for architecture analysis, architecture improvement and architecture design.

How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers
Xin Lu', Yanyan Zhao Bing Qin', Liangyu Huo?, Qing Yang", Dongliang Xu²
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology Du Xiaoman (Beijing) Science Technology Co., Ltd.
[xlu, yyzhao, qinb}@ir. hit. edu.
Chuollianyu, yangqing, xudongllang//duxxiaome com
Abstract
Pre-trained language models have been proven to possess strong base capabilities, which not only excel in in-distribution language modeling but also show powerful abilities out-of-diittrbuutin language modeling, transfer learning and few-shot learning. Unlike existing work focusing the influence of scale on base capabil- ities, our work examines the influence of architecture on those. Specifically, our concern is: How does architecture influence the base capabilities of pre-traine trained language models? In this work, we attempt to explain and reverse the decline in base capabilities caused by the architecture of FFN-Wider Transformers, seeking provide some insights. Through analysis, we found the contribution ratio of Multi- Head Attention combination function) pre-trained language modeling a key factor affecting base capabilities. CFN-Wides Transformers reduce the contribution ratio of this combination function, leading decline in base capabilities. We confirmed this by experiments and proposed Combination Enhanced Architecture (CEA) to address the decline base capabilities of such models. Significantly, we extended our explanation and CEA to Mixture of Experts (MoE) Transform- ers. We successfully achieved significant improvements in base capabilities on a 14B parameter MoE model, demonstrating the practical application value of our work. This also indicates that our analysis has certain guiding significance for architecture analysis, architecture improvement and architecture design.
1
Introduction
Recent research has discovered that pre-trained language models possess strong base capabilities 244 They can not only address in-distribution language modeling which usually their pre- training objective, but also unexpectedly excel in out-of-distribution language modeling, transfer learning, few-shot learning, etc. This has attracted the attention of many researchers.
3 22
However, it has also been observed that the cost of pre-training a language model is substantial, with the trial-and-error approach based empirical improvements proving be very expensive. Consequently. there is a desire to gain insights into the final performance by analyzing factors like scale and architecture that directly determine the base capabilities of models.
In this process, much attention has been focused on analyzing scale, leading to the formulation of compelling scaling laws [14] [1] that drive the trend of enhancing base capability by increasing parameter numbers, data volume and training tokens. In contrast, the impact of architecture has not
"Email corresponding.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Comdination Adjusable Acchiboture
Duter FFN Contribution Base Capabllity (OOD)
MoF Models Performatio (DOD)
Layerfoore
Laye Nere
1279
315
Wider FFN
Gotee-IFN
335
wiltl ideaers
DiyerMorm
1222
Laverhio
MilA Drasialtant
LayerWorm
Iniez FFN
1006 EU2 150%
1225
2225
2500
12.6
FPM Wereertttteett Condonat an www.raaaeeeeneeeeoo Becture
OKRN Rate
Pao Todted Toless (3)
Figure I: Illustration showing that: 1) the synchronous improvement in model base capability as the contribution ratio of the Outer-FFN layer (a transformation function) decreases, that is, the contribution ratio of the MHA layer (a combination function) increases. This reveals a key factor affecting model's s base capabilities. 2) Combination Enhanced Architecture (CEA) was designed based on this factor and applied to MoE. models, resulting in an improvement in base capability.
received sufficient attention. According to the basic principle of inductive bias in machine learning, model architecture is also a crucial factor affecting base capabilities, and its impact could be equally decisive.
Some studies have already noted the significant influence of architecture the base capabilities. For instance, [31 have observed considerable differences in base capabilities among various Transformer architecture variants. Some variants, though larger in scale and more powerful in pre-training performance than vanilla Transformers, exhibit significantly reduced performance in downstream tasks. This suggests different architecture models vary greatly in converting pre-training performance into base capabilities. Simply increasing scale does not resolve all issues, and exploring the impact of architecture on base capabilities is crucial.
However, despite these studies demonstrating the key influences of model architecture on base capabilities, the understanding of the underlying mechanisms of these influences remains limited. In this work, we attempt to explain the base capability change caused by a specific model architecture change and identify the underlying influencing factor, then design experiments to validate explanation and influencing factor, and finally propose a generalizable enhancement method.
Specifically, we first focus on FFN-Wider Transformers. This architecture has wider FFN layers, but we found that the change leads to a significant decrease in base capabilities compared to vanilla Transformers. As shown in Figure 2 under similar pre-training performance, the FFN-Wider models exhibit a noticeable decline in base capabilities compared to the vanilla models. We believe such simple change, leading to significant differences in base capabilities, makes for a good object of study in exploring how architecture impacts base capabilities.
Then, we attempted to explain this change in base capabilities. Through analysis, we concluded that the MHA (Multi-Head Attention) layer in the Transformer is combination function, and the FFN Freed-Forwwar Network) layer is transformation function, with the former being a focused expression of the combinability of language. During their contribution to pre-trained language modeling, the actual contribution ratio of the MHA layer, as a combination function, is key factor affecting the model's base capabilities. The FFN-Wider Transformer models directly enhance the FFN layer, indirectly reducing the combination function's 's actual contribution ratio to pre-trained language modeling, thereby lcading to a significant decline in base capabilitics.
To validate our explanation for it, we designed a Combination Adjustable Architecture (CAA), as depicted in Figure a)) [a). This architecture bifurcates a wider FFN into two parts with adjustable width ratios: one remains in its original position as a transformation function, known as the Duter-FFN, and the other relocated within the MHA layer, transformed through a special design into an Inner- that solely enhances the combination function. We controlled the width ratio of the Outer-FFN, reducing it gradually from 100% to 0%. In Figure we observed that the actual contribution ratio of the Outer-FFN progressively decreased, indicating corresponding increase in the actual contribution ratio of the MHA layer. At the same time, we also observed gradual improvement in base capabilities. These reveal key phenomenon that confirms explanation: the actual

---

contribution ratio of the MHA layer (a combination function) increases, there is a general synchronous improvement in the model's base capabilities.
Subsequently, we identified the optimal width ratio of two parts of FFN for the Combination Adjustable Architecture (CAA) and established it as Combination Enhanced Architecture (CEA), and comparing this architecture with the original FFN-Wider Transformer. We conducted experiments on various scales of BERT and GPT models, with all results robustly supporting our explanation.
Importantly we also noticed that existing work has observed similar decline base capabilities in the Mixture of Experts (MoE) Transformers. We applied our explanation and CEA to MoE Transformers as well. We pre-trained a 14B parameter GPT architecture MoE model and its With the
improved version with CEA on 100B tokens (using CC and C4 from SlimPajama 2 same number of parameters, computational load, and pre-training steps for both versions, the improved version with CEA showed significant improvements in out-of-distribution language modeling and few- shot learning. The performance of the -ff-----iitiiuttoon language modeling (average performance across all domains in SlimPajama except for CC and C4) is shown in Figure [c), which can prove the practical application value of our work.
Overall, the actual contribution ratio of the MHA layer (a combination function) is likely universal factor affecting model's 's base capabilities, which can provide valuable insights for architecture analysis, improvement and design.
2 Background
Subsequent sections will involve specific analyses and experiments WFder and MoE trans- formers. Therefore, in this section, we introduce the relevant background of base capabilities, as well as the unified evaluation schemes and evaluation tasks.
2.1 Base Capabilities
Pre-trained language models not only address in in-distribution language modeling but also unexpectedly show strong base capabilities. In this work, we focus on the following base capabilities:
cot-of-Distributtio Language Modeling This reflects the out-of-distribution generalization capability of pre-trained language models. Models that learn more essential language features often outperform others, which is good measure of the base capabilities of pre-trained language models.
Transfer Learning This is recognized base capability of pre-trained language models. The works proposing GPT (24) and BERT 7 have established the "pre-training and fine-tuning" transfer learning paradigm under the Transformer architecture.
Few-shot Learning This is also a recognized base capability. Some works 255 3
found large-scale
pre-trained language models could complete many NLP tasks with or only few demonstrations, revealing their remarkable few-shot learning capabilities.
2.2 Evaluation Schemes
In this work, we primarily explore how the architecture of a model influences base capabilities in pre-trained language models, which requires designing sound approach for quantitative analysis.
Comparing the base capabilities of one architecture model against another is straightforward: we simply compare their performance on variety of representative tasks. However, changes in base capabilities cannot be directly attributed changes in architecture alone, model capabilities are also influenced by factors like scale.
This leads to question: under what conditions can compare the base capabilities of two different architecture models and be more confident that the differences are due to architecture variations?
Our scheme to this question: we compare the models when they use the same pre-training data and objectives, and have similar levels of pre-training performance (i.e., language modeling performance on an in-distribution development set).
3

---

BERT (H=129)
(H=7G3
GPT (H=129)
GPT (H=7G9)
(471
-10
538
-2.0
122
+220
ALIR
S4.41
475
-3M
45
321
230
SuperELUE
Pib
Midippl (D-lo0)
Midtile (loder)
(OOD)
(P))
Vanilla Model
FEN-Wide Mooel
Figure 2: Comparison of the base capabilities between
and Vanilla Transformers.
The underlying rationale: two different architecture models, when trained on the same corpus and objectives, both gain base capabilities by reducing the loss in pre-trained language modeling. If one architecture model achieves greater base capabilities with the same reduction language modeling loss, it is highly likely that this additional benefit stems from the inductive bias of its architecture. In other words, when pre-training performance aligned, differences in base capabilities models are likely reflecting the impact of architecture inductive biases.
This scheme is more appropriate for aresteereie analysis compared to aligning pre-training steps, parameter numbers, or computational load, and we explain it in Appendix A
Unless specified otherwise, all comparisons of base capabilities across architecture models in this work are conducted under the condition of aligned pre-training performance.
The experiment with MoE transformers is exception, because need to demonstrate practi- cality. Therefore, the comparative experiments between the vanilla MoE model and the improved MoE model, not only is pre-training performance aligned, but pre-training steps, parameter numbers, and computational load are also kept consistent.
2.3 Evaluation Tasks
Due to subsequent experiments involving dutttf-ddittibuttion language modeling evaluation, transfer learning evaluation and few-shot learning evaluation, it is necessary to clarify the selection of pre- training corpus and -iito--distribution test corpus, as well as the selection of downstream tasks. We provide detailed introduction in Appendix B
3 FFN-Wider Transformers vs. Vanilla Transformers
This work focuses the base capabilities of FFN-Wider Transformers. A typical Transformer model has Forward Network (FFN) layers. Assuming the hidden dimension d, the standard intermediate dimension of an FFN layer is 4d. However, an FFN-Wider Transformer model means the intermediate dimension of its FFN layer exceeds 4d.
We conducted experiments on various models with the intermediate dimension set to 32d, aligning pre-training performances with those of vanilla models. The results are shown in Figure 2
We found that, at the same level of pre-training performance, the Transformer models with wider FFN layers exhibit noticeable decline in performance on most downstream tasks, indicating deterioration in their base capabilities. This presents us with good research object.
4
Why FFN-Wider Transformers Have Worse Base Capabilities?
4.1
Combination and Transformation
Transformers consist of MHA and FFN layers. Considering certain position in a sequence, updated representation obtained after an MHA layer is combination of the entire sequence context; whereas the updated representation obtained after an FFN layer is from a transformation of that position representation alone, mnntext-insensitivee

---

When considering models composed of only one MHA layer or one FFN layer, we find that they can both be used for language modeling. If treated as black boxes focusing only on input and output, there is no difference. However, the way they accomplish language modeling is different, or rather, their inductive biases differ: the FFN layer directly maps the previous token to the target token, representing a one-to-one transformation function; while the MHA layer uses the entire sequence to calculate the target token, representing a many-to-one combination function. The latter aligns more closely with the combinability of language.
Further considering the multi-layer stacking MHA and FFN layers in a Transformer model, we easily understand that although all layers contribute to the final language modeling objective, they do so with different inductive biases. Among these, the MHA layer might be the central embodiment of the model architecture's 's expression of the combinability of language.
With this understanding, when revisit the scenarios where the FFN layer is widened or its capacity is increased, we have reason to suspect this will lead to a change in the actual contribution ratios of the transformation and combination function, thereby affecting the model's is expression of the combinability of language, which is ultimately reflected in changes in basic capabilities.
Our hypothesis: the actual contribution ratio of the MHA layer (a combination function) is key factor affecting the model's 's base capabilities. The Wider Transformers directly enhance the FFN layer, indirectly reducing the combination function's 's actual contribution ratio to pre-trained language modeling, thereby leading to a significant decline in base capabilities.
4.2
Contribution Ratio Analysis
To verify our hypothesis, we first need confirm whether the actual contribution ratio of MHA layers truly decreases, requiring quantitative analysis.
4.2.1 Mutual Information
The first method is Mutual Information (MI). For positions in the sequence where a predicted token to be output, we can focus on the representations of these positions after each layer and calculate the MI between these representations and the target tokens. Then, since we have MI before and after passing through any layer, we can calculate the contribution ratios of the MHA and FFN layers.
We adopted the mutual information estimate method proposed by [32), which mainly involves converting the representations into discrete variables through clustering, and then calculating the mutual information. For the definition of MI and more specific details, please refer to Appendix
We analyzed four small-scale models (H=128): vanilla BERT, FFN-Wider BERT, vanilla GPT and an FFN-Wide GPT. The pre-training performances of the vanilla models and the FFN- Wider models aligned. We plotted the MI results in Figure B where the blue, orange and grey lines represent the cumulative MI increment contributions of the MHA layer, FFN layer and Block, respectively. It can be seen that in the FFN-Wider models, the MI contribution of the FFN layer is significantly higher than that in the vanilla models, preliminarily validating our hypothesis.
4.2.2 Token Prediction
Since the MI estimate requires clustering a large number of representations and is costly when the hidden dimension is high, we tried another more direct method called Token Prediction (TP). It involves predicting tokens directly from the hidden representations. The approach is as follows:
We still obtained representations from each layer, first dividing them into sets based on the corre- sponding output token. Then, we normalized all representations within a set and use their mean as the category vector for that token. During this process, smaller than 50. For the representations of each layer, we then calculated the cosine similarity with the category vectors of tokens in that layer, selecting the token with the highest similarity as the prediction result for that representation. In this way, we could calculate a token prediction accuracy for each layer. The subsequent approach was identical to that of the MI method.
eliminated tokens where the set size was
We analyzed four small-scale (H=128) and four large-scale (H=768) models concurrently. We plotted the accuracy increment contribution ratio of the FFN layer for each model, as shown in Figure 4 It
5

---

FPN Contribuioo Radio (T Tkkas Prodictio)
68 57%
PEN TTME Autoecy Rnsoonend
Layss
43 91%
41.97%
td Gardla (antat)
33 10%
30%
25.20%
27%
375
0%
(3+TEB)
(3+-18))
Figure 3: Contribution ratio analysis based on Mutual Information(MI) for various transformers.
Figure 4: Contribution ratio analysis based on Token Prediction (TP) for various transformers.
can be seen that the contribution of FFN layers in FFN-Wider models remains higher than that in vanilla models, reaffirming our hypothesis.
Combination Adjustable Architecture
Although the FFN layer in FFN-Wider Transformers has higher contribution ratio, cannot assert this is the reason. This may merely be a correlation rather than a decisive factor in base capabilities.
Therefore, designed architecture that directly intervene in the contribution ratios of the transformation and combination function, named Combination Adjustable Architecture (CAA).
5.1 Model Architecture
The new architecture, shown in Figurd5 differs from the FFN-Wider Transformer in two aspects: one is partially transferring the FFN into the MHA; the other is adding a direct pathway inside the MHA that bypasses the Inner- mnner-FFN, which will introduce in detail below.
5.1.1 Inner-FFN and Outer-FFN
wider FFN would increase the contribution ratio transformation function to pre-trained language modeling. natural idea: is possible devise variant FFN serves only combination function?
We found the primary reason that the FFN does not serve only combination function due to the residual connections. The representation transformed by the FFN does not necessarily have to go through the MHA combination process; the FFN can bypass the MHA via residual connections and directly transmit the information in the representation subsequent layers.
Therefore, we divided an FFN into two parts with adjustable width ratios: one remains its original position as transformation function, known as the Outer-FFN, and the other is relocated within the MHA layer, transformed an Inner-FFN. This design ensures that the representation transformed by the Inner-FFN must undergo the MHA combination process before proceeding, initially achieving the goal of the Inner-FFN serving only for the combination function.
5.1.2 Direct Pathway in MHA
Although considered the impact of residual connections and designed an Inner-FFN, the Inner- FFN still has a hidden pathway to bypass the combination function and directly transmit uncombined information to subsequent layers.
The attention mechanism involves weighted summation over values, and typically, the value at the current position also participates. This provides possibility for the Inner-FFN to directly transmit information. Therefore, we designed a direct pathway the MHA for current position computation to circumvent the Inner-FFN, Specifically, during the self-attention computation the

---

Capobityy (OCO)
LayerNorm
Log Lowthooell O9M
Outer-IFN
Troen ENN and Outor TPN (Witth Rebio is Adjussatai
Quary
Key
Score
Layerfoom
MHA X Siec hetwent
Caodalle (0C0)
Layerform
Combenatinn Gusaable rttiieetumm (CAA)
Dirad Pullowyy MHA iSiegla Head)
O8a 525 Mill
Ou WoN
Figure 5: Overview of our proposed Combination Adjustable Architecture (CAA).
Figure Outer-FFN contribution ratio and base capability under different width ratios.
current position, the query, key and value from the current position the input representation that has not been processed by the Inner-FFN. Only the context representations from non-current positions are transformed by the Inner-FFN. This design largely eliminates the possibility of the Inner-FFN bypassing the combination function.
5.2 Width Adjustment Analysis
In this section, we control the width ratio of the Outer-FFN, reducing it gradually from 100% to Confoniii our hypothesis by examining the trends of contribution ratio and base capability.
5.2.1 Trend of Contribution Ratio
We conducted experiments on large-scale (H=768) BERT and GPT models. Initially, series of models with different width ratios were pre- pre-trained, ensuring their pre-training performance was aligned. Then, we calculated the contribution ratio of Outer-FFN using Token Prediction (TP) and plotted these trends, shown in Figures a) and 6 c)..
can be seen that the contribution ratio of Outer-FFN decreases as width ratio is reduced. Conversely, the contribution ratio of the combination function increases with the reduction in the width ratio of Outer-FFN, demonstrating that controlling the width ratio indeed directly influences the contribution ratio of the combination function.
5.2.2 Trend of Base Capability
We tested the performance of each model above in ouitt--distribution (OOD) language modeling and plotted the trend in Figures b)) d)).
For BERT, the OOD performance improves the Outer-FFN width decreases. The performance of the optimal model has already surpassed the vanilla BERT. Most models not only outperform the FFd-Wider BERT in terms of performance but also require fewer pre-training steps. For GPT, only the model without Outer-FFN showed anomalous results, other models also follow this pattern.
After the analyses, conclusion is drawn from the overall trend: the model's base capabilities generally improve as the contribution ratio of combination function increases. holds over a wide range of width ratios and proves our hypothesis.
However, this overall trend does not mean the contribution of transformation function should be reduced to zero. A minor contribution from transformation function might still be crucial, as indicated by the sole anomalous result in GPT. BERT was unaffected, which is related bidirectional attention still being able to leak few transformation contribution. More details are in Appendix G

---

GLUB
SGLUR
Others
Modd
Model
(22 puzk))
tanks)
B taka?
(5 take)
(22 ports)
(9 makes)
@ bodes)
(R-128, Pre- Pre-Trinnng Performanee/s -2,61)
(H-128, Pre-Training Performoncs -406)
Vanilla
3,211
78.45
Vanilla GPT
-4.706
FFN-Widr
76.66
REN-Wider GPT
-4.725
ITN Wider BERT w/ CBA
-3,202
76.20
PIN Wider GPT wt CDA
-4,85
77.29
(11-688 Pre-Training Porformances:
-3.19)
(M:768, Pre-iinnnng Performoaneett -1.33)
Vasilla BERT
2.158
83.89
64.41
$2.14
Vanilla GPT
-5878
45.82
46,09
PFN-Wide
82.60
PFN Wider GPT
-3.913
44.60
44.84
FFN Wide: BERT CEA
-2.149
83.86
64.22
53.12
FFN- GPT wt CFA
-3,82
48.52
45.84
Table I: The results of various BERT models.
Table 2: The results of various GPT models.
Combination Enhanced Architecture
In this section, we determine the Combination Enhanced Architecture (CEA) and pre-train more steps to verify whether can reverse the decline base capabilities of FFN-Wider Transformer.
6.1 Width Ratio Selection
We selected the optimal width ratio for Combination Enhanced Architecture (CEA). The Outer-FFN width ratio for FFN-Wider BERT w/ CEA set to 0%. For FFN- Wider GPT w/ CEA, it is 12.5%.
6.2 Further Experiments
We conducted more pre-training steps on vanilla models, FFN-wider models, FFN-wider models w/ CEA, and aligned the pre-training performances.
For BERT, tested fine-tuning performance GLUE, SuperGLUE and other datasets. For GPT, tested zero-shot and one-shot performance on multiple datasets. For both, we tested OOD language modeling on Pile. For small-scale models, we removed some experiments beyond their capabilities. The brief results are shown in Table I and 2 and the detailed results are in Appendix H
The results show the FFN-wider models w/ CEA improve in base capabilities, not only surpassing the FFN-wider models in most aspects but also nearly reaching the level of vanilla models. In addition, the new architecture models w/o the direct pathway in MHA show significant decline in base capabilitics. These all confirm our explanation.
Additionally, we conducted similar experiments for other width ratios that can align pre-training performance and steps simultaneously, which can indisputably prove improvements come from architecture, detailed in Appendix
7 From FFN-Wided Transformers to MoE Transformers
The Mixture of Experts (MoB) Transformer 18 g is practical architecture that introduces mix of experts, which enables expanding the capacity with lower computation expense.
However, base capability decline also be observed MoE Transformers. perform lower on SuperGLUE compared vanilla models when achieving same pre-training level. Similar issue can also be analyzed from the results of [T]
g found MoE models
We found the MoE layer can be seen enhanced version of the FFN layer. Therefore, believe the previous explanations could also apply to MoE Transformers. Consequently, we directly transplant our CEA MoE Transformers, resulting in new MoE architecture.
We selected a 1.3B GPT model as the backbone model, which incorporates Rotary Embedding (30 and RMSNorm 7 on the GPT3 3. Then, we added MoE layer with 64 experts (top-1 activation) before the FFN layer, resulting in a 14B parameter MoE baseline model. Finally, our improved version with CEA involves transforming the MoE layer into Inner-MoE layer within the MHA,

---

Vanilla GPT 1.3B
Vanilla MoE 14B
MoE 14B
Metric
Shol
w/ CEA
Total Params
N/A
1.3B
Activated Params
N/A
1.3B
1.5B
1.5B
Total Experts
N/A
64
64
Activated Experts
N/A
Training Tokens
N/A
100B
100B
100B
SlimPajama-CCen CC&C4 (Loss))
N/A
2.382
2.315
2.303
SlimPajama-Arss (Loss)
N/A
2.353
2.320
2.239
SlimPajama-Bout (Loss)
N/A
2.796
2.761
2.585
SlimPajama- (Loss)
N/A
1,989
1.989
1,840
SlimPajama-Stane (Loss)
N/A
2.653
2.588
2.465
SlimPajama- Wiki (Loss)
N/A
3.137
3.155
2.964
LAMBADA (PPL)
O-shot
23.7
24.8
19.3
LAMBADA (Acc.)
O-shot
36.6
36.4
39.7
MMLU (Acc.)
5-shot
30.8
30.6
31.4
OpenBookQA (Acc.)
5-shot
36.6
35.7
37.2
ARC Easy (Acc.)
5-shot
57.5
56.8
59.2
ARC Challenge (Acc.)
5-shot
31.6
30.0
32.3
BoolQ (Acc.)
5-shot
62.2
62.2
62.1
RACE Middle (Acc.)
5-shot
42.3
44.4
45.5
RACE High (Acc.)
5-shot
36.1
36.0
36.6
SIQA (Acc.)
5-shot
41.0
41.7
43.7
SCIQ (Acc.)
5-shot
78.1
72.7
84.1
HellaSwag (Acc.)
5-shot
43.7
COPA (Acc.)
5-shot
69.2
69.2
70.4
PIQA (Acc.)
5-shot
70.4
70.6
71.5
StoryCloze (Acc.)
5-shot
67.7
68.0
69.5
WinoGrande (Acc.)
5-shot
52.7
$3.1
53.4
Winograd (Acc.)
5-shot
67.2
66.7
66.8
Table 3: The results of Vanilla GPT 1.3B, Vanilla MoE 14B and MoE 14B w/ CEA. indicates the test set drawn from the same distribution as the pre-training data.
while retaining the original FFN layer as the Outer-FFN layer. To accommodate FlashAttention-2 [6] we simplified the direct pathway. Instead of preventing transformation leakage by replacing the key and value at current position, we chose to directly mask the current position to achieve the same effect. The specifications, pre-training procedures and few-shot learning procedures are detailed in Appendix [ and E
We conducted pre-training on CC and C4 subsets within SlimPajama 299 training all models from scratch with 100B tokens. Due to the de-duplication across subsets achieved by SlimPajama, we used the remaining subsets as out-of-distrrbution sets. We carried out comprehensive out-of-distribution language modeling and few-shot learning tests, with the results presented in Table 3
From the results, is evident that method significantly enhances the base capabilities of the MoE model. This fully demonstrates the effectiveness of our analysis and improvement methods.
8 Limitations
This work mainly analyzes the models that use language modeling as the pre-training objective, lacking experiments models with other pre-training objectives. Hence, the conclusions are limited to pre-trained language models. Therefore, the current applicability of our findings is relatively narrow, and we consider conducting more experiments in the future.

---

Conclusion
This work explores how architecture affects the base capabilities of pre-trained language models. FFN-Wider Transformer is our research object and we try to explain and reverse the decline in base capabilities caused by its architecture. We found the contribution ratio of combination function is a key factor, while FFN-Wider Transformer reduces it, leading decline in base capabilities. We solved by proposing CEA. In addition, we extended our conclusion to MoE Transformers, proving our work can offer guidance for architecture improvement.
Acknowledgments
This work was supported by the Fundamental Research Funds for the Central Universities (project number: 2022FRFK060002), the National Key RD Program of China via grant 2021 YFF0901602 and the National Natural Science Foundation of China (NSFC) via grant 62176078.
References
[1] M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L.. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang. L. Zettlemoyer, M. Diab, Z. Kozareva, and V. Stoyanov. Efficient large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11699-11732, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
[2] Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[4] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Explor- ing the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[5] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv, abs/ lats/8003.555, 2018.
[6] Dao. 2lashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. June 2019, Association for Computational Linguistics.
[8] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 1-39, 2022.
[9] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Naheshima, S. Presser, and Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
10

---

[10] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021.
[11] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark. T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre. An empirical analysis of compute-optimal large language model training. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016-30035 Curran Associates, Inc., 2022.
S. Koyejo, S. Mo-
[12] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and Gelly. Parameter-efficient transfer learning for NLP. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019.
[13] A. Jiang Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de las Casas, Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023.
[14] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[16] G. Lai, Q. Xie, H. Liu, Y. Yang, and Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics.
[17] Z. Lan, M. Chen, S. Goodman, K. Gimpel, Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
[18) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. (GS) Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.
[19] H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[20] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics.
[21] N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01669 2016.
[22] OpenAl. Gpt-4 technical report, 2023.
[23] D. Paperno, G. Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Compula- tional Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
[24] A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding by generative pre-training, 2018.

---

[25] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[26] M. Roemmele, C. A. Bejan, and A. S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
[27] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[28] M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 4463-4473, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
[29] 7. Shen, T. Tao, I.. Ma, W. Neiswanger, Z. Liu, H. Wang, B. Tan, J. Hestness, N. Vassilieva, D. Soboleva, and E. Xing. Slimpajama-dc: Understanding data combinations for IIm training, 2024.
[30] J. Su. Lu. S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
[31] Y. Tay, M. Dehghani, S. Abnar, H. Chung, W. Fedus, J. Rao, S. Narang, V. Tran, D. Yogatama, and D. Metzler. Scaling laws model architectures: How does inductive bias influence scaling? In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Compula- tional Linguistics: EMNLP 2023, pages 12342-12364. Singapore, Dec. 2023. Association for Computational Linguistics.
[32] E. Voita, R. Sennrich, and I. Titov. The bottom-up evolution of representations in the transformer: study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNN.P-JCNLPP) pages 1396-4406 Hong Kong, China, Nov. 2019, Association for Computational Linguistics.
[33] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman Superglue: A stickier benchmark for general- general-purpose purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[34] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics.
[35] Welbl, N. Liu, and M. Gardner. Crowdsourcing multiple choice science questions, 2017.
[36] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics.
[37] B. Zhang and Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019.
12

---

A Other Evaluation Schemes
Aside from the pre-training performance alignment scheme adopted, there three other schemes. However, none of these are suitable for our work.
Pre-training steps alignment Differences in model's base capabilities might be due to differences in model capacity, not solely from architecture changes. Especially in models with different architectures and parameter scales, models with pre-training loss to a low level, showing good metrics on various tasks, and appearing to have strong base capabilities. But this is more due large model capacity, and not much related to architecture inductive biases.
parameters have capacity advantage, rapidly reducing
Parameter numbers alignment It imposes restrictions on the model that make it inapplicable to any two different architecture models. More critically, this scheme has clear counterexamples, as it is not suitable for ALBERT models [17]. Under the same parameter numbers, ALBERT models exhibit strong base capabilities, but this is due to the large computational load, not much related to architecture inductive biases.
Computational load alignment imposes restrictions on the model that make it inapplicable to any two different architecture models. More critically, this scheme also has clear counterexamples, as it is suitable for MoE models [8 8. Under the same computational load, MoE models demonstrate strong base capabilities, but this is due to large parameter numbers and model capacity, not much related to architecture inductive biases.
Although these schemes have shortcomings, they also have reasonable aspects. To indisputably prove base capability improvements come from our architecture changes, we also conducted experiments that align pre-training performance, pre-training steps, parameter numbers and computational load simultaneously, detailed in Appendix !!
In addition, the experiment with MoE transformers also adopts a stricter setting similar to the above. In the comparative experiments between the vanilla MoE model and the improved MoE model, not only is pre-training performance aligned, but pre-training steps, parameter numbers, and computational load are also kept consistent.
B Evaluation Tasks
B.1 FEN-Wider Transformers
All experiments with FFN-Wider transformers follow the unified settings described here:
The pre-training corpus is consistent with that of BERT 77, namely Wikipedia and BooksCorpus. We partition portion of the corpus to serve in-distribution development set, and use it to align pre-training performance.
We conduct experiments on four specifications: BERT (H=128). BERT (H=768), GPT (H=128) and
GPT (H=768). The specifications and pre-training procedures are detailed in Appendix C
For out-of-distribution language modeling capability, we evaluate on the development set of Pile dataset
For transfer learning capability, we only evaluate BERT models on GLUE 34 SuperGLUE
33
HellaSwag (36), PIQA (2) OpenBookQA [20]. ARC Easy & Challenge 5 and WinoGrande 27
The experimental settings are detailed in Appendix[D
For few-shot learning capability, we only evaluate GPT models. Limited by the maximum sequence length (128) of our pre-trained models, we conduct 0-shot and 1-shot experiments on HellaSwag, PIQA, OpenBookQA, ARC Easy & Challenge, WinoGrande, Winograd [19] COPA (26) and
StoryCloze (21). The experimental settings are detailed in Appendix
B.2 MoE Transformers
All experiments with MoE transformers follow the unified settings described here:
13

---

The pre-training corpus is the CC and C4 subsets within the SlimPajama dataset [29]. The specifications and pre-training procedures are detailed in Appendix
For outtr--distriuution language modeling capability, due to the de-duplication across subsets achieved by SlimPajama [29], used the remaining subsets (Arxiv, Book, Github, Stack and Wiki) as out-of-distribution test sets.
For few-shot learning capability, we conduct 0-shot experiments on LAMBADA (23 and conduct
5-shot experiments on MMLU [T0], OpenBookQA, ARC Easy & Challenge, BoolQ 4 RACE
Middle & High [16]. SIQA 28 SCIQ (31, HellaSwag, COPA, PIQA, StoryCloze, WinoGrande and Winograd. The experimental settings are detailed in Appendix
C Model Specifications and Pre-training Procedures
C.1 FFN-Wider Transformers
All experiments were conducted in English only. We concurrently pre-trained both BERT and GPT models, where the architecture design of the BERT model adheres to the work of (7]. and the GPT model structure follows that established by [24]. Both models employ a post layer normalization scheme.
We trained small and large scales of both BERT and GPT. The small-scale models have a hidden dimension of 128, 12 layers, and 2 attention heads; the large-scale models possess hidden dimension of 768, 12 layers, and 12 attention heads. vanilla BERT and GPT, the FFN intermediate dimension is 4 times the hidden dimension, while the ratio of FFN-Wider models is 32 times. The small-scale vanilla models have 6.3M Parameters and the small-scale other models have 17.3M Parameters. The large-scale vanilla models have 110M Parameters and the large-scale other models have 506M Parameters. All models have a maximum sequence length of 128 and utilize the BERT vocabulary released by W. comprising 30,522 tokens. We used Py PyTorct and transformers libraries.
The BERT model employs a masked language modeling task for pre-training, masking 15% of tokens in the sequence 80% are replaced with [MASK], 10% with random tokens, and 10% remain unchanged. Differing from [7]. we removed the next sentence prediction task, using long and continuous text for pre-training inputs and applying different masking schemes to the same input sequence various epochs. The GPT model is pre-trained using language modeling task without additional special settings.
The maximum epoch set for all model pre-training 40, but in practice, it was not reached; mid- training checkpoints were used for alignment and experimentation. All models used the Adam optimizer [13] for pre-training, with learning rate of le-4, 31 0.9, B2 0.999, L2 weight of 0.01, warm-up over the first 10,000 steps, followed by linear decay. The small-scale models were pre-trained with a batch size of 512 on four Nvidia Tesla V100s, and total GPU days are approximately 55 days; the large-scale models with batch size of 1024 on four Nvidia Tesla A100s, and total GPU days are approximately 87 days.
To fairly compare the base capabilities of different architecture models, all models were pre-trained from scratch. However, as mentioned earlier, due limited computational resources, our pre- training steps generally fell short of those in the original papers discrepancies in downstream task performance compared to the original research.
3], which may result in
C.2 MoE Transformers
All experiments were conducted in English only.
We selected the GPT model as the backbone model (Vanilla GPT 1.3B), which incorporates Rotary Embedding [30 and RMSNorm (37 the GPT3 B3 Then, we added a MoE layer before the FEN layer, resulting in our MoE baseline model ( (Vanilla MoE 14B). Finally, our improved version with CEA (MoE 14B w/ CEA) involves transforming the MoE layer into Inner-MoE layer within the MHA, while retaining the original FFN layer as the Outer- FFN layer. To accommodate
Phtrps//pytorch..org/
14

---

FlashAttention-2 [6], we simplified the direct pathway. Instead of preventing transformation leakage by replacing the key and value at current position, we chose directly mask the current position to achieve the same effect.
The Vanilla GPT 1.3B has hidden dimension of 4,096, 6 layers and 32 attention heads; the Vanilla MoE 14B has a hidden dimension of 4,096, 6 layers, 32 attention heads and 6 MoE layers with 64 experts (top-1 activation, intermediate dimension is 4,096) before the corresponding FFN layers; the MoE 14B w/ CEA is similar to the Vanilla MoE 14B, just the position of the parameters has changed. All models have maximum sequence length of 2,048 and utilize the Mixtral vocabulary released by 13. comprising 32,000 tokens. We used Py Torch, transformers and fastmod libraries.
These models are pre-trained using a language modeling task without additional special settings. We conducted pre-training on CC and C4 subsets within SlimPajama [29]. training all models from scratch with 100B tokens. Due to the de- de-duplication across subsets achieved by SlimPajama, used the remaining subsets as pre-training, with a learning rate of 5e-4, 31 0.9, B2 0.95, L2 weight of 0.01, warm-up over the first 2,500 steps, followed by linear decay. All models were pre-trained with a batch size of 256 on 8 Nvidia Tesla A100 80G cards, and total GPU days are approximately 280 days.
sets. All models used the Adam optimizer [15] for
D Fine-tuning Settings
We conducted fine-tuning experiments on BERT models across various datasets. For the small-scale (H=128) models, only conducted experiments GLUE, while for the large-scale (H=768) models, we experimented with all datasets.
The maximum number of training epochs during fine-tuning was 10, with a batch size of 32. The optimizer was Adam [15), with a warmup ratio 0.06, a linearly decaying learning rate, and a weight decay of 0.01. We reported the average performance of multiple runs.
For the small-scale (H=128) models, we observed models with a wider FFN might overfit when fully fine-tuned with all parameters. Thus, we performed both full parameter fine-tuning and efficient fine-tuning based on adapters [12] for all models, choosing the better result of the two for reporting. For full parameter fine-tuning, the learning rates were [le-5, 2e-5, 5e-5): for efficient fine-tuning, the adapter size was 128, with learning rates of (le-4, 2e-4, 3e-4). For the large-scale (H=768) models, we directly conducted full parameter fine-tuning for all models, with learning rates of (le-5, 2e-5, 5e-5).
E Few-shot Learning Settings
In our study on GPT and MoE models, we conducted few-shot learning experiments on multiple datasets. Since small-scale (H=128) models have limited capabilities and struggle with few-shot learning, we focused our experiments only on large-scale (H=768) models for 0-shot and I-shot learning and MoE models for 0-shot and 5-shot learning.
We followed the method of 33. which involves transforming the classification into comparisons of probability magnitudes.
Specifically, the unified format of these datasets involves selecting one option from multiple choices, given context. We concatenated the context with different options and compared their respective generation probabilities. This comparison could be based on either the probability of the option alone the entire sequence text, with choice of normalizing the probabilities by length or not. B mentioned a method of normalizing the probability of each option unconditionally, which we also included in our options. We then experimented with all possible combinations of these choices for each dataset, determined the best combination for each, and reported the results of all models under these optimal conditions.
For I-shot experiment, we randomly selected one demonstration each time, repeated the experiment 10 times, and reported the average results. For 5-shot experiment, we randomly selected five demonstrations each time (for MMLU, randomly shuffling the order of demonstrations), repeated the experiment times, and reported the average results.
15

---

F
Mutual Information Estimate
The formal definition of mutual information for two discrete random variables is as follows:
(ax.))(a,)) (x(x))y
(X;Y)= = -x: P(x) v) yEY DEX'
where X and Y are discrete random variables, P(x,Y) is the joint probability mass function of X and Y, and Px and Py are the marginal probability mass functions of X' and Y respectively.
For estimating the mutual information between intermediate representations and target tokens, adopted the method proposed by [32]. which primarily involves transforming the representations into discrete variables through clustering and then calculating mutual information.
We first randomly sampled 6.94 million input tokens and their corresponding output tokens from pre-trained development set and collected all intermediate representations of the model these locations. Then, for the GPT model, we performed clustering directly at each layer and calculated mutual information using the labels obtained from clustering with the output tokens. For the BERT model, we also conducted clustering at each layer; however, since masked language modeling only considers masked positions the actual output, mutual information masked positions. Clustering was done using the mini-bate batch k-means algorithm, initialized with the k-means+ method, with batch size of 1024 and class number set to 2000.
calculated solely at these
G Additional Transformation Function Contribution in BERT
While it appears that BERT can reduce the actual contribution ratio of transformation function to zero, in reality, some contributions of transformation function remain uncovered by our method.
For BERT, although we have eliminated most of the pathways through which the Inner-FFN directly leaks independent transformation information to subsequent layers, there still one way this can happen: due to the bidirectional nature of attention, in the MHA of the current block, the entire sequence context has already been infiltrated by information about the current position at lower levels. Therefore, the Inner-FFN can independently enhance this part of the information, which can then leak out again through the weighted summation over values. In contrast, the GPT models employ unidirectional attention, so the context definitely does not contain any information about the current position, hence there is truly no independent transformation without the Outer-FFN.
Therefore, while we are fairly certain that reducing the contribution ratio of independent transfor- mation function is general trend that enhance base capabilities, completely eliminating the contribution of it could be harmful, as indicated by the trend GPT; the absence of this phenomenon in BERT's trend merely due to the likelihood that BERT still retains small part of independent transformation in the Inner-FFN.
H Detailed Results (Main Setting: Pre-training Performance Alignment)
In Section we present the brief experimental results Table I and 2 and here we provide the corresponding detailed results.
For BERT, out-of -fsttriibutii language modeling results on Pile are shown in Table fine-tuning results on GLUE Benchmark are shown in Table 7 fine-tuning results on SuperGLUE Benchmark are shown Table fine-tuning results on multiple other tasks are shown in Table 9
For GPT, out-of-distribution language modeling results on Pile are shown in Table 10 zero-shot results and one-shot results multiple datasets shown in Table 11
I Detailed Results (Extra Setting: Pre-training Performance Alignment & Pre-training Steps Alignment)
The main experiments in this work are based on the pre-training performance alignment scheme. Although we have introduced the rationale behind this scheme, the difference in pre-training steps
16

---

Model
Pile paris)
GLUR
SGLUE tasks)
Others
usks)
(H=128, Pre-Training Performances2222 -2,61, Pre-Training Stepa 155.2k)
FFN-Wder
FFN-Wikr (12.5%)
-3.203
(11-768, Pre- Pre-Iraining Performance::: -153, Traiiigg Stepe 164.%
FFN-Wider BERT
-2.196
82.00
62.82
51.61
(125%)
Table 4: The results of BERT (Pre-training Steps Alignment & Pre-training Performance Alignment).
0-Shot tasks)
(22 parts)
tasks)
(H=128, Pre-Training Performance:: -.066 Pre- Pre-Training Steps 74.2K)
FEN-Wider GPT
-4.728
FEN-Wider GFT w/ CAA (37.5%)
4.706
Pre-Training Performancess -.19, Pre-Truining Steps
RPN-Wider GPT
-3911
4.60
44.84
FFN Wider GPT (37.5%)
-38993
45.27
45.82
Table 5: The results of GPT (Pre-training Steps Alignment & Pre-training Performance Alignment).
between original FFN- Wider models and new architecture models may still raise question: is the base capability improvement of new architecture models really due to architecture changes?
To answer this question, we chose the Outer-FFN width ratios that allow CAA align with FFN- Wider Transformer for both pre-training performance and pre-training steps, and conducted exper- iments similar those in Section 6.2 Moreover, the parameter numbers and computational load of CAA are also aligned with the FFN-Wider Transformer. Therefore, we achieved alignment in four aspects, and under this setting, the improvements of base capabilities can be incontrovertibly attributed to our architectural changes.
The Outer-FFN FFN width ratio for FFN-Wider BERT w/ CAA is set to 12.5%, and for FFN-Wider GPT w/ CAA, it set to 37.5%.
The brief results are shown in Table 4 and 5
and the corresponding detailed results are as follows:
For BERT, out-of-distribution language modeling results Pile shown in Table T2 fine-tuning
results on GLUE Benchmark are shown in Table 13 fine-tuning results on SuperGI IUF Benchmark
shown Table 14 fine-tuning results on multiple other tasks are shown in Table 15
For GPT, out-of-distribution language modeling results on Pile are shown Table T6 zero-shot results and one-shot results multiple datasets shown in Table 17
The experimental results show that the base capabilities of new architecture models have still significantly improved, confirming the positive impact of our architecture modifications on base capabilities.
17

---

H=128
Vanilia BERT
FrN-Wider
FEN-Wider BERT CEA
wio Direct Pabewa MELA
Vanilla BERT
HRN-Wider BERT
HEN-Wider
BERT
BERT CEA
Validation
2.617
2,613
-2614
-2.611
-1533
-1.536
-1.58
AzXiv
3.288
-3405
-3.301
-3.997
2.315
-2.369
-2212
BookCorpus2
2,636
2,623
-2,629
-2.621
-.572
-1.583
-1.569
Books3
-3014
-3.012
11904
DM Mathematies
2.780
22865
-2.730
-2.878
2.119
2,330
-2.158
Enron Emails
3.160
3.181
-3.134
-3.122
2,100
2.1335
-2105
EuroPart
-4804
4.862
-.336
-4851
3.492
-3,74
3.420
Gichab
-3.110
3,391
3.260
-3.221
2.371
2,,46
-2.455
Gutenberg (PG-19)
-3.172
3.181
-3.779
3.174
2,090
2.122
-2.117
HockerNews
-3.223
-3,273
.3.003
-3.211
2.212
-2.247
-2.119
2.991
OpenSabtiles
-2,800
22091
22006
220063
1.399
-1.413
1,406
Weeneeh
-3.313
.3,336
.3.114
-3,111
2.122
2,,149
-2.126
PhilPPapers
4.171
4,221
44188
-4.201
2.872
2,853
2.823
-2.127
PuhMed Abstracts
-2884
-2.92
-28800
-2911
-1.719
-1.746
-1.110
PubMed Central
2.949
3.007
-2.41
-2.977
-1.442
-19989
-1933
Stack Exchange
-3.379
-3,40)
.3.347
-3.387
-2.389
-2.436
-2.374
-3.45
USPTO Backgrounds
22824
-2878
2.812
-2.833
-1.774
-1.823
-1.78
Wikipedia (en)
-2.790
-2,813
-2.780
-2.791
-1.663
-1,674
-1.41
YouruheSubstiles
-3.566
-3,600
-3583
-3.608
-2.994
-2.642
.2.563
Average
-3.211
-3.256
-3.202
-3.230
-2.188
-2.196
-2.149
Table 6:
language modeling results the development set of Pile of various BERT models (Pre-training Performance Alignment).
Model
CoLA
MRPC
SST-2
STS-B
RTR
MNLI
QNLI
QQP
Avg.
(H=128, Pro-Praining Performancess
-2.61)
Vanilla BERT
35.46
83.01
88.18
88.30
83.89
83.57
65.59
77.13
77.59
84.43
89.09
85.15
78.45
WideW BERT
34,95
79.94
85.79
87 07.84
81.85
81.76
59.20
76. 10
76.22
83.43
88.53
84.35
76.66
TEN. Wider BERT CEA
33,70
83,17
88.22
88.19
82.96
82,78
64.26
77.72
78.07
84.65
89.15
85,50
78.30
Direct Puthway
82.61
77.36
77.65
88,
(H=768, Pre-Training Performance
-1.53)
Vanilla BERT
62,10
87.33
90.70
92.43
87.97
87.74
62.04
83:43
83:78
90.06
91.08
88.02
83.89
TEN. Wider BERT
61.66
85.42
89.73
91.86
86.35
86.17
60.79
81.87
82.24
88.69
89.
86.49
82.60
FFN Wider BERT CEA
62.58
87.99
91.45
92.32
87.30
87.13
62.69
83.01
82.81
89.86
91.10
88.08
83.86
Table Fine-tuning results the development set of GLUE Benchmark of various BERT models (Pre-training Performance Alignment).
Model
BoolQ
COPA
MultiRC
WIC
ReCoRD
WSC
RTE
(H=768, Pre-Training Performancess
-1.53)
Vanilla BERT
75.44
86.01
83.88
64.83
61.91
16.24
66.77
63,70
62.82
64.90
62.04
64.41
FFN-Wide BERT
73.70
83.04
79.68
64.71
60.90
14.57
64.62
62.54
61.73
64.78
60.79
62.82
66.50
66.59
Table 8: Fine-tuning results on the development set of SuperGI.UE Benchmark of various BERT models (Pre-training Performance Alignment).
Model
HellaSwag
PIQA
WinoGrande
OpenBookQA
ARC Easy
ARC Clal.
Avg.
(H=768, Pre- Pre-Trinnig Performance2s-
-1.53)
Vanilla BERT
40.79
67.46
58.06
$6.38
53.47
36.65
$2.14
FFN Wider BERT
38.95
67.00
$5.09
57.78
51.74
39.07
51.61
BERT
Table tine-tuning results on multiple other tasks of various BERT models (Pre-training Perfor- mance Alignment).
18

---

H=128
H=768
Vanilla
FFN-Wider GPT
FEN-Wider GPT w/ CEA
Vanilla GPT
FFN-Wider GPT
FFN-Wider
GPT
GPT w/ CEA
Validation
-4.061
-4.067
-4,066
-3.197
-3.191
-3.192
ArXiv
4.855
-4927
-4893
-3.981
4.061
33,800
BookC2rpus2
-.066
-4.055
-4.01
-3.309
-3,315
-3,301
Books3
-4,999
-4.495
-4,486
3.700
3.715
-3,697
DM Mathematics
44.110
-.162
-3.999
-3542
-3,597
-3.526
Enron Emails
-4.619
-4636
-4.583
3.846
-3877
-3,811
EuroParl
-6,006
-6026
-5.987
4,853
4.855
44888
FreeLaw
-4,625
4.625
4.599
3.711
-3,226
-3,703
Github
4.979
5.030
4.899
4.135
4.189
.4.130
Gutenberg (PG-19)
-4,618
4,610
4,600
-3,909
-3,930
-3,910
HackerNews
4.738
4.711
4,049
.4.057
NIH ExPorter
-46112
4.617
4.610
3.622
-3.55
.3.636
OpenSubtitess
-3.338
-3.345
-3.316
2,85
-2,886
-2,874
OpenWehTex12
PhilPapers
-5.558
55580
-5.47
4,499
4,522
44525
Pile CC
-4.754
4.758
4,747
3,62
33979
3,962
PubMed Abstracts
-4.575
4.591
4.562
3.569
-3,99
33570
PuhMed Central
-4,583
-45977
4.521
3.760
-3.797
33776
Exchange
Ubuntu IRC
-5.649
-5.729
-5.600
4.857
-4,999
-4,997
USPTO Backgrounds
-4.460
.4.503
-4.53
33636
-3,690
-3.442
Wikipedia (en)
-4.302
-4.312
-4,295
-3.45
-3.359
-3,339
YoutubeSubtidles
-4.782
-4822
-4.775
-3,999
-4,036
-4,013
Average
-4.706
-4.728
-4.685
-3.878
-3.911
-3.882
Table 10: Out-of-distribution language modeling results on the development set of Pile of various GPT models (Pre-training Performance Alignment).
Model
Helbebwagg
PTQA
WineCirands
COPA
OpenllookQ4
ARC Easy
ARC Clui.
StoryClos
Winegrad
Avg.
25,00
$0.00
25.00
25,00
30,00
38.89
(H-66 8-Shu, Pre Prainninn Peflommoerrss- -219;
Vanilla GPT
2667
$7.28
$2.0
61.01
30.40
41.0
25.85
$5.00
$7.90
6533
PEN Wider GIP
Mi.9
$1.77
20.30
43.56
27.45
56.09
16.09
64.00
EFN Wider
34.49
45.52
(H-768, 1.Shat, Pre- Training Performanceis
-3.9%
Vanila
56.89
20.36
28.06
FFN-Waler
Table 11: Zero-shot and one-shot results on multiple datasets of various GPT models (Pre-training Performance Alignment).
19

---

H=128
H=768
FFN Wider BERT
FEN Wider BERT w/ CAA (12.5%)
FFN Wider
FEN Wider BERT w/ CAA (12.5%)
BERT
Validation
-2.613
-2.610
-1.536
-1.531
ArXiv
-3,405
-3,305
-2.369
-2220
BookCorpus2
-2,623
-2.627
-1.583
-1.578
Books3
-3,031
-3.011
-1.904
-1887
Mathematics
Enron Emails
-3.181
-3.144
-2.135
-2.955
EuroPar
4,862
4,853
-3.774
-3.472
FreeLaw
33126
-3.87
-1.52
-1938
Github
33391
33.295
2.446
2.358
Gutenherg (PG-19)
HackerNews
33.273
-3.113
-22447
2.209
NIH ExPorter
-3003
-3.005
-1.99
-17887
OpenSublitles
22091
-2.000
-1413
-1407
OpenWehTexxe
-3,336
-3,14
2.149
2.126
PhilPapers
4.221
.4195
-2853
Pile-CC
-3.248
-.222
2.127
2.097
PubMed Abstracts
-2952
-2988
-1.746
-1.227
PubMed Central
Stack Exchange
-3.40
<3.336
-2436
2.364
Ubuntu
-3,822
-3.105
-3012
USPTO Backgrounds
2,878
2.811
-1,833
.1783
Wikipedia (en)
-2813
-2.781
-1,674
-1662
YoutubeSublitles
-3.630
-3,594
-2.642
-2599
Average
-3.256
-3.203
-2.19
-2.153
Table 12: cittof-distribution language modeling results on the development set of Pile of various BERT models (Pre-training Steps Alignment & Pre-training Performance Alignment).
MRPC
STS-B
QNLI
QQP
Avg.
(R=128, Pre-Training Porformancets
-2.61,
Pre-Training Stepr 155.2k)
FFN-Wider BERT
34.95
79.94
85,79
87.84
81.85
81,76
59.20
76.10
76.22
83.43
88.53
84.35
76.66
FiN-Wider BERT w/ CAA (12.55))
35.77
83.44
88.48
87,20
84.31
84,07
59.51
77.33
78.05
84.14
89.11
85,50
78.08
(H-768, Pre-Training Performancess -1.53, Pre-Thaining Stpp
164.9k)
FFN-Wider BERT
66
85.42
89.73
91.86
86.35
86.17
60.79
81.87
82.24
88 69
89.98
86.49
82.60
FiN-Wider BERT w/ CAA (12,55)
61.72
87.13
90.88
91.06
87.71
87.58
61.37
83.38
83.12
89.29
90.83
87.71
83.48
Table 13: Fine-tuning results the development set of GLUE Benchmark of various BERT models (Pre-training Steps Alignment & Pre-training Performance Alignment).
Model
BoolQ
CB
COPA
MuttRC
WIC
ReCoRD
WSC
RTE
Avg.
(H=768, Pre-Training Performancess
-1.53, Pre- ree-Trainng Stepy=
Wider BERT
73.70
83.04
79.68
64.71
60.90
14.57
64.62
62.54
61.73
64.78
60.79
62.82
FFN-Wider BERT w/ CAA (12.5%)
75.48
83.33
81.66
65.00
62.23
16.49
66.08
62.86
62.03
65.14
61.37
63.79
Table 14: Fine-tuning results the development set of SuperGI.UE. Benchmark of various BERT models (Pre-training Steps Alignment Pre-training Performance Alignment).
Model
HellaSwag
PIQA
OpenBookQA
Easy
Avg.
(H-768, Pre-Thiining Performouee:: -1.53, Pre- Pre-Training Stops 164.9k)
FEN-WSde Wider BERT
38.95
67.00
$5.09
$7.78
51.74
39.07
$1.61
WFdr BERT C/CAA (12.5%)
40.59
67.81
$7.20
51.94
39.49
$2.45
Table 15: Fine-tuning results on multiple other tasks of various BERT models (Pre-training Steps Alignment & Pre-training Performance Alignment).
20

---

H=128
H=768
FEN Wider
FEN-Wider
FEN-Wider
GPT W/ (37.5%)
CIPT
GPT (37.55%)
Validution
-.0067
-4.066
-3.191
-3.192
ArXiv
.49927
-4,966
-4,061
4.010
BookCorpus2
-.055
-4,054
-3,315
-3,309
Books3
.4.95
-4.485
-3,715
-3,701
DM Mathematies
-4162
-4.110
-3,597
-3,540
Earon Emails
4.636
-4,598
-3,877
-3,522
EuroPad
-6.026
-4,55
44865
FreeLaw
4.625
-4.616
-3.726
-3.220
Github
-5..30
4,980
4,189
4.156
Gutenberg (PG-19)
44610
-46004
-3,930
-3,912
HackerNews
-4738
-4.229
-4001
4.055
NIH ExPorter
4,617
4,611
-3.657
-3,642
OpenSubiitles
22886
-2887
Open WehTexx2
44817
.4810
-3.755
3.964
PhilPapers
-5580
-5.557
4,522
-4511
Pile-CC
4.758
-4.754
-3,979
3,968
PubMed Abstracts
4.591
-45776
-3,593
33579
PubMed Central
4.597
-4,576
-3.797
33780
Stack Exchange
-5.331
-4233
4,210
Ubuntu IRC
-5.729
-5623
-4989
4,922
USPTO Backgrounds
-4,503
-4,668
-3,690
33668
Wikipedia (en)
.4.312
YoutubeSubtitles
-4.822
-4800
-4,036
-.0117
Average
-4.728
-4.706
-3.911
-3,893
Table 16:
language modeling results on the development set of Pile of various GPT models (Pre-training Steps Alignment & Pre-training Performance Alignment).
HallaSwag
PIUA
Windfoande
OpraBlook(4
Dhal.
Story
Winegrad
Ang.
RRnddn Perfermance)
25.00
50.00
50.00
50.00
25.00
25.00
25.00
50.00
$060
38.90
(17766 1-Shat, Prn-Thatinng Derformmort--l -1,99 Pre-Tratatey Supee: (0.90
FRN
$7.18
50.33
56.24
FFN Made W/PA
38.72
25.65
57.16
4588
Table 17: Zero-shot and one-shot results on multiple datasets of various GPT models (Pre-training Steps Alignment & Pre-tres Pre-training Performance Alignment).
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Not involving proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix C, D, E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification:
There are some restrictions that make inconvenient to publish directly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
iossiinnnoilyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pulli//uuiii wwwpc//ggue////eeeeceooc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix C, D, E and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use multiple rounds of randomization to ensure the stability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
24

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httbp:://neurippee http///urr..cccupupp///e http://...ccooocccttttcctttteeea
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not release models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "YSNlQDFzN7", "reviewer": "Reviewer_xAuN", "summary": "The paper examines the influence of architecture on the base capabilities (OOD, transfer learning, few-short learning) of large language models. The main focus is on FFN-Wider transformers and understanding why they have poorer base capabilities compared to vanilla transformers. The contibution ratio of multihead attention to pretrained language modeling is found to be a key factor affective base capabilities. Based on this observation, a Combination Enhanced Architecture is proposed and also extended to mixture of experts transformers. The analytical study is backed by experimental evaluation successfully achieving significant improvements in base capabilities of the 14B parameter MoE model.", "questions": "- Some choices of parameters such as chosing intermediate dimension to 32d instead of say 8 or 16d can be better explained. \n\n- Was the width adjustment limited to Outer-FFN?", "limitations": "The reviewer does not except any negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ While most existing study of large models has focussed on the impact of scale, this work is a significant effort in understanding the influence of architecture. \n\n+ The empirical findings are supported by a more deeper interpretation of the underlying mechanisms of these influences. \n\n+ The proposed CAA wherein the wider FFN is split into adjustable two parts - outer-FFN which stays as a transformation function, and an inner-FFN which is relocated within the MHA layer for enhancing the combination function is interesting in its own right. \n\n+ The extension to MoE transformers and the experimental demonstration with 14B parameter GPT architecture MoE model is very interesting.", "weaknesses": "- Some of the findings appear to be obvious (perhaps, in hindsight) - as the contribution ratio of the MHA layer increases, the base capabilities also improve. It would have been interesting to also find some non-intuitive influences.  \n\n- Overall the paper's analysis is much narrow in scope than the title of the paper. Nonetheless, it is interesting and useful study. \n\n- The language and presentation of the paper can be improved. It will be useful to have the grammar issues fixed in the paper before the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and valuable feedback!\n\nWe answer your questions below.\n\n***\n\n**Q1:** Some choices of parameters such as chosing intermediate dimension to 32d instead of say 8 or 16d can be better explained.\n\n**A1:** We agree with your opinion that attempting more width would enhance the persuasiveness of the work. In fact, the choice of 32d in the paper was not made with any special consideration; it was simply based on an assessment of computational resources. In future work, we will consider enriching this aspect to strengthen the persuasiveness of the work.\n\n***\n\n**Q2:** Was the width adjustment limited to Outer-FFN?\n\n**A2:** Adjusting the width ratio is mainly aimed at the original FFN in the FFN-Wider model, which will be split into Outer-FFN and Inner-FFN. The sum of their widths is equal to the original FFN, meaning an increase in the width of one will result in a decrease in the width of the other. Therefore, the width adjustment should occur simultaneously in both the Outer-FFN and the Inner-FFN.\n\n***\n\n**Other Responses:**\n\nWe sincerely apologize for any inconvenience caused by the language and expression issues in the paper. We will improve and refine the relevant expressions in the subsequent versions of the paper.\n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjlcuuNiQn", "reviewer": "Reviewer_Rvop", "summary": "This paper examines how the architecture of a transformer model influences its base capabilities, such as out-of-distribution tasks, transfer learning, and few-shot learning. Specifically, it explores the effects of replacing the feed-forward network (FFN) with a wider FFN (FFN-wide) in various parts of the architecture.\n\nInitially, the authors find that replacing the FFN (referred to as the transformation function) after the attention layer (referred to as the combination function) with FFN-wide leads to worse performance. They analyze this impact by measuring the contribution of the combination function using techniques such as mutual information and token prediction. The results indicate that performance deteriorates in most cases when the contribution of the combination function decreases.\n\nThe authors then devise multiple architectures with different width ratios of FFN-wide in the combination and transformation functions. They select the best ratio model architecture, termed CEA (Combination-Enhanced Architecture). This CEA architecture is subsequently used in a Mixture of Experts (MoE) model, demonstrating improvements over the non-CEA MoE model.", "questions": "- What does it mean for the width ratio to be 0% and 100%? Does this imply that the FFN is removed in these cases?\n\n- What does it mean that FFN-wider models with CEA are not able to beat vanilla models in some cases in Table 1 and Table 2? Isn't the vanilla model one of the ratios explored? If not, would it make more sense to explore the architectures on some other dimensions?\n\n- Please indicate whether lower or higher values are better for all the tables and figures in the paper.", "limitations": "The authors can also address the limitations of exploring only a narrow subset of architectures and discuss whether any parts of the proposed methodology can be extended to other aspects of transformer architectures. Additionally, all experiments were performed on smaller-scale models, so the results may not generalize to larger models.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an interesting problem by measuring the impact of transformer architecture on the downstream performance of the model, which is crucial for optimizing and understanding transformer models.\n\n- Attributing performance to the inner workings of a neural network, particularly the contribution of specific components, and analyzing this contribution using techniques like mutual information is both novel and insightful.\n\n- It is also interesting to see the impact on two different kinds of model like Bert and GPT and see how the optimal width for the two types of models are different. \n\n- The application of CEA to a mixture of experts models definitely demonstrates the practical benefits of this paper.", "weaknesses": "- The performance improvements for the different variations of the architectures seem minor, and the absence of standard deviations makes it difficult to assess the robustness of the results. \n\n- For both BERT and GPT, even after a thorough architecture search, the performance is at best similar to the vanilla models, which is not very promising.\n\n- Similarly for the MoE experiment, it will useful to add the standard deviation of the results. Additionally, since the training loss is lower for MoE with CEA compared to vanilla MoE, it is hard to determine whether the improvements are due to the architecture's inductive bias or just the lower training loss. Similar to the rest of the paper, please also report numbers for this experiments by keeps the same training performance level. \n\nOverall, the paper is interesting, but the performance values being so close and the lack of variance measurement do not give me much confidence in the results. \n\nMinor: In Table 1 and Table 2, the results for the proposed approach are bolded rather than the best approach for that experiment.\n\nIn most figures, the y-axis does not start at 0, which can give readers an inaccurate representation of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and valuable feedback!\n\nWe answer your questions below.\n\n***\n\n**Q1:** since the training loss is lower for MoE with CEA compared to vanilla MoE, it is hard to determine whether the improvements are due to the architecture's inductive bias or just the lower training loss ... please also report numbers for this experiments by keeps the same training performance level.\n\n**A1:** **Our MoE experiments still utilized a similar setting: ensuring that the pre-training performance of the original MoE and the improved MoE were roughly consistent.** We suspect that some results in Table 3 or Figure 1(c) may have caused the misunderstanding, so we would like to clarify them here.\n\nIn Table 3, only \"SlimPajama-CC&C4 (Loss)\" represents pre-training performance, as the models in Table 3 were pre-trained on the CC and C4 subsets of the SlimPajama dataset. Because the SlimPajama dataset achieved de-duplication across subsets, the remaining results for SlimPajama belong to OOD tests. The performance curves in Figure 1(c) are similar, only representing OOD performance.\n\nAnother concern might be that even if \"SlimPajama-CC&C4 (Loss)\" in Table 3 represents pre-training performance, the improved MoE (2.303) still seems to perform better than the original MoE (2.315). Is this advantage small enough? To illustrate the issue clearly, **we have included a PDF attachment (in the global response above)** with pre-training performance curves similar to Figure 1(c) and compared side-by-side with the original Figure 1(c). \n\n**It can be seen that:** 1) The gap in pre-training performance is negligible compared to the OOD performance gap; 2) During pre-training, there were moments when the pre-training performance of both models was identical, yet the OOD performance gap remained significant. Therefore, we believe this situation still largely aligns with our claimed setup.\n\n***\n\n**Q2:** For both BERT and GPT, even after a thorough architecture search, the performance is at best similar to the vanilla models, which is not very promising.\n\n**A2:** For the FFN-Wider model, it is indeed the case that it does not offer efficiency advantages over the vanilla model. Improving the FFN-Wider model is less beneficial than simply using a scaled-up vanilla model. In fact, we did not emphasize its practical value but rather used it as a good analytical object to conduct some meaningful analysis.\n\nThe key point is that we eventually extended the relevant analysis to the MoE model. The MoE model is a practical architecture that offers the advantage of expanding model capacity with lower computational costs compared to the vanilla model. With the same amount of computation, the MoE model performs better in pre-training, making it worthwhile to improve the MoE model.\n\n***\n\n**Q3:** What does it mean that FFN-wider models with CEA are not able to beat vanilla models in some cases in Table 1 and Table 2? Isn't the vanilla model one of the ratios explored? ...\n\n**A3:** The response to A2 should also apply to this question.\n\nAdditionally, it is necessary to clarify that the FFN-Wider model has a wider FFN, and even after adjusting width ratios, it cannot be equivalent to the vanilla model.\n\n***\n\n**Q4:** What does it mean for the width ratio to be 0% and 100%? ...\n\n**A4:** Generally, adjusting the width ratio results in the presence of two FFNs. However, when the ratio is 0% or 100%, one of the FFNs is removed, leaving only a single FFN.\n\n***\n\n**Q5:** Please indicate whether lower or higher values are better for all the tables and figures in the paper.\n\n**A5:** Our explanation is as follows:\n\n**Figure 1(b):** The orange line represents a metric, where higher values are better.\n**Figure 1(c):** Lower values are better.\n**Figure 2:** Higher values are better.\n**Figures 3 and 4:** No metrics are present.\n**Figure 6:** The blue line represents a metric, where higher values are better.\n**Tables 1 and 2:** Higher values are better.\n**Table 3:** Metrics labeled with \"Loss\" and \"PPL\" are better when lower, while metrics labeled with \"Acc.\" are better when higher.\n\nThe tables in the appendix provide detailed information, with the metrics having similar meanings.\n\n***\n\n**Q6:** The performance improvements for the different variations of the architectures seem minor, and the absence of standard deviations makes it difficult to assess the robustness of the results ...\n\n**A6:** We would like to address this from two perspectives. **On one hand,** similar to A2 above, we do not actually consider the FFN-Wider model as a replacement for the vanilla model. The FFN-Wider model does not have practical competitiveness; its performance improvement is mainly to validate our analysis. Therefore, the extent of the improvement may not be particularly critical, and our main focus is to show the performance of the MoE model. **On the other hand,** given that there were no changes in scale or data, only slight architectural modifications, the extent of this performance improvement seems reasonable. Although it has limited practical significance, it generally supports our analysis.\n\nAs for the robustness, on one hand, the results in Tables 1 and 2 are averaged performance across multiple tasks, which provides some degree of robustness. On the other hand, **we have supplemented the results with standard deviations in the PDF attachment (in the global response above)**, also indicating that the results are relatively robust.\n\nThe tables in the PDF attachment only list the results involving multiple rounds of experiments. Specifically, the results in Table 1 are from 8 rounds (H=128) and 4 rounds (H=768) of experiments, Table 2 from 10 rounds, and Table 3 from 5 rounds.\n\n***\n\n**Other Responses:**\n\nWe apologize for any inconvenience caused by the tables and figures. We will make improvements in subsequent versions to enhance the clarity and comprehensiveness of the related expressions.\n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2f5ds34zpD", "reviewer": "Reviewer_QyvJ", "summary": "The paper studies the contribution raion of FFN and MHA layer in transformers and its effect on out-of-distribution performance. It finds that the wider FFN layer will increase its contribution ratio and lower the OOD performance. Lastly, the paper proposes a new architecture that moves part of the FFN layer to MHA and shows the CEA can improve the baseline models' OOD ability.", "questions": "See weaknesses 1 and 2.\n\nOthers:\n\n- Figure 6 lacks a legend to explain the figure elements, including the blue line, orange bar, blue and orange dotted lines.", "limitations": "The authors addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper analyzes the contribution ratio with new methods that evaluate Mutual Information and token prediction accuracy of each layer's output.\n\n- The proposed CEA method has improved the MoE transformer in various tasks and datasets with the same parameter scale, showing its effectiveness in increasing OOD ability.", "weaknesses": "- The paper does not give a convincing explanation of why the study aligns pre-trained performance with different parameter scales. A larger scale model may not fully converge when it has a similar training loss to a smaller model. It is also not known whether the proposed CEA will harm in-domain performance.\n\n- The MHA module in the vanilla transformer also has linear layers for Q, K, V, and after self-attention. I think they can be seen as inner-FFNs that will transform the combination function and make it not necessary to introduce an extra inner-FFN layer.\n\n- Tables 1 and 2 show that BERT w/ CEA performs worse than vanilla in GLUE and SGLUE and GPT w/ CEA perform worse in all datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and valuable feedback!\n\nWe answer your questions below.\n\n***\n\n**Q1:** The paper does not give a convincing explanation of why the study aligns pre-trained performance with different parameter scales. A larger scale model may not fully converge when it has a similar training loss to a smaller model. It is also not known whether the proposed CEA will harm in-domain performance.\n\n**A1:** You mentioned why we adopted the alignment of pre-training performance for models with different parameter scales. We would like to explain this further. We mainly consider that comparing models with different parameter scales is often unavoidable (even outside the scope of this paper). Therefore, finding a relatively reasonable basis for cross-parameter scale comparison is necessary. **This method needs to eliminate the interference of parameter advantages, computational advantages, etc., and maximally reflect the base capability differences brought by the architecture itself.** In our paper, we explained that aligning pre-training performance might be more suitable for this goal (Section 2.2), while other methods might not be appropriate (Appendix A).\n\nYou also mentioned the potential issue of inadequate convergence in large parameter models, and you might be concerned about the inconsistency in the behavior of models during the later stages of training. While this factor cannot be entirely ruled out, we have not yet explored this topic deeply in our current analysis. **However, one thing is certain:** in our work, the comparison between large parameter models and their improved versions does not face this issue, as they have the same parameters, and there should not be significant differences in the degree of convergence.\n\nRegarding your concern about whether the new architecture would harm in-domain performance, as shown in Figure 6, most width ratios do not pose any problems. Only extreme width ratios might harm pre-training performance. Our experiments have demonstrated that choosing non-extreme width ratios can still achieve performance gains (Appendix I). For MoE models, **we have plotted pre-training performance curves similar to Figure 1(c) in the attached PDF (in the global response above).** It can be seen that the improved MoE models do not experience a degradation in pre-training performance, which essentially indicates the usability of the new architecture.\n\n***\n\n**Q2:** The MHA module in the vanilla transformer also has linear layers for Q, K, V, and after self-attention. I think they can be seen as inner-FFNs that will transform the combination function and make it not necessary to introduce an extra inner-FFN layer.\n\n**A2:** You mentioned that there are also linear layers within the MHA module, which can be considered as inner-FFN layers, making additional inner-FFN layers potentially unnecessary.\n\nWe had considered this issue as well, but we found that the linear layers in the MHA module lack non-linear transformations, which directly results in their transformation capability being inferior to that of the FFN layers. Therefore, considering the linear layers in the MHA as inner-FFN might still face the issue of the MHA contribution ratio as discussed in our paper. Ultimately, we chose to convert a part of the FFN into inner-FFN.\n\n***\n\n**Q3:** Tables 1 and 2 show that BERT w/ CEA performs worse than vanilla in GLUE and SGLUE and GPT w/ CEA perform worse in all datasets.\n\n**A3:** Your observations are correct, and they are reasonable within the context of our work.\n\nFirstly, since the models in Tables 1 and 2 are aligned based on pre-training performance, the FFN-Wider model cannot outperform the vanilla model solely based on parameter size. Therefore, it is expected that the vanilla model performs better on downstream tasks, and it is normal for the FFN-Wider model to be inferior to the vanilla model.\n\nSecondly, **the fact that FFN-Wider w/ CEA is also inferior to the vanilla model is within our expectations.** The main reason is that we used the FFN-Wider model primarily as a good subject for analysis. Improving it was only to demonstrate the effectiveness of our analysis, and ultimately, the improvements are intended to benefit the MoE model.\n\nIn fact, the FFN-Wider model does not have efficiency advantages over the vanilla model. Improving it is less practical than directly using an enlarged vanilla model. Thus, even if the FFN-Wider w/ CEA were to surpass the vanilla model, it would not have practical significance. Therefore, we ultimately focus on demonstrating practical utility through the MoE model.\n\n***\n\n**Other Responses:**\n\nWe apologize for any inconvenience caused by the figures in the paper. We will make improvements in subsequent versions to enhance the clarity and comprehensiveness of the related expressions in the paper.\n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYzpPcJU8J", "reviewer": "Reviewer_nte2", "summary": "This study investigates the influence of architecture on pre-trained language models' base capabilities. It reveals that the contribution ratio of Multi-Head Attention to pre-trained language modeling affects base capabilities. FFN-Wider Transformers reduce this ratio, leading to a decline in base capabilities. The researchers propose a Combination Enhanced Architecture (CEA) to address this issue. They also extend this to Mixture of Experts (MoE) transformers, achieving significant improvements in base capabilities.", "questions": "In general, I think the discovery of the paper is interesting, and I don't have many questions. Just some minor issues: \n\n- What is the definition of mutual information? Give some introduction.\n- How is the pre-training performance of the models?\n- How is the structure of CEA? It is not clear to me after reading Section 6.", "limitations": "The authors address it in Section 8.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper shifts the focus from the commonly studied impact of scale on pre-trained language models to the impact of architecture.\n+ The exploration of the influence of using wider FFN layer is interesting, and should impact further researches.\n+ The proposal of CEA as a solution to the decline in base capabilities shows a proactive approach to solving the identified problem.\n+ The findings are confirmed through extensive experiments, adding credibility to the claims.", "weaknesses": "- The description of CEA is not detailed enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and valuable feedback!\n\nWe answer your questions below.\n\n***\n\n**Q1:** How is the pre-training performance of the models?\n\n**A1:** For the BERT and GPT experiments, the results are primarily presented in Tables 1 and 2, with a textual description of the pre-training performance achieved by each model group. For the MoE model experiments, **we have included in the attached PDF (in the global response above) pre-training performance curves similar to Figure 1(c).** These curves allow for a comparison between the original MoE model and the improved MoE model, which we hope addresses your concerns.\n\n***\n\n**Q2:** What is the definition of mutual information? Give some introduction.\n\n**A2:** We apologize for the oversight. The absence of the definition of mutual information might cause confusion for researchers outside this field. We will follow your suggestion and add the relevant definition and explanation of mutual information.\n\n***\n\n**Q3:** How is the structure of CEA? It is not clear to me after reading Section 6.\n\n**A3:** We apologize for any confusion caused, and we would like to provide further clarification.\n\nIn Section 5, we introduce the Combination Adjustable Architecture (CAA), which is an architecture designed for analytical purposes. This architecture primarily involves splitting the original FFN in the FFN-Wider model according to a certain width ratio to obtain Outer-FFN and Inner-FFN. We then adjust the width ratio between these two components to support our arguments. At this stage, we pre-train multiple models with varying width ratios on a small scale for analysis.\n\nIn Section 6, we introduce the Combination Enhanced Architecture (CEA), which is similar to the CAA mentioned above. However, since CEA is an improved architecture rather than an analytical one, it requires a fixed width ratio to be determined before conducting large-scale pre-training experiments. For different models, we determined various width ratios and then performed large-scale pre-training and downstream experiments.\n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Here is a PDF attachment containing the figures and tables referenced in the detailed responses below.

Meta-review: This paper explores how and why the FFN-Wider architecture influences base capabilities of LLMs (out-of-distribution modeling, transfer learning, and few-shot learning). Specifically, when the pretraining performance is held constant, FFN-Wider Transformers have worse base capabilities than vanilla ones, but why? The authors identify a property of the architecture (the contribution ratio of MHA layer) that correlates with base capability, and provide further analysis and insights. The authors then propose a specific architecture (with the best ratio) which is shown to improve base capabilities in MoE models.

I recommend to **accept** this paper. Reviewers appreciated that the authors chose to study the influence of architecture (instead of scale) on LLM performance, which is often difficult due to the low interpretability of neural networks in general, but nevertheless the authors provided "novel and insightful" analysis into the topic. It is also significant and impactful that the authors found a better (than vanilla) architecture for MoE models, based on that earlier analysis.

However, reviewers also mentioned that the writing/clarity of the paper could be improved. I am optimistic about this aspect because the authors clarified reviewers' questions in the rebuttals, and I encourage the authors to revise the paper to include the clarifications and generally improve the presentation for the next revision.
Decision: Accept (poster)