PartialFormer: Modeling Part Instead of Whole for Machine Translation (2024)

Abstract: The design choices in Transformer feedforward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multihead attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer’s capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/ zhengkid/PartialFormer.

heading: 1 Introduction
text: The Transformer model (Vaswani et al., 2017) has emerged as a cornerstone in the natural language processing (NLP) domain, overshadowing convolutional neural networks (Gehring et al., 2017) and recurrent neural networks (Sutskever et al., 2014) by virtue of its minimal inductive bias, superior scalability, and proficiency in modeling sequences. Nonetheless, its substantial computational and parametric requisites pose significant challenges to its deployment and training, warranting an ongoing trend in the research community toward eliminating redundant parameters and computations (Dehghani et al., 2019; Mehta et al., 2019; Lan et al., 2020;
* Equal Contribution. † Corresponding author.
Wu et al., 2020; Mehta et al., 2021; Reid et al., 2021; Li et al., 2022a) in Transformer.
While these attempts represent significant strides in enhancing the efficiency of the Transformer architecture, they largely neglect an equally critical component: the Feed-Forward Network (FFN) that constitutes a substantial part of the Transformer’s computational and parametric footprint, due to the inherent large feature space and hidden dimension. Previous studies (Mehta et al., 2021; Wu et al., 2020; Ge et al., 2022) have simplified FFNs by naively reducing their hidden dimensions, often at the expense of expressive power. This leads to a question: Is the current formulation of lightweight FFNs truly optimal?
To answer this concern, we turn to the insights provided by Geva et al. (2021), who depicted FFNs as a collection of key-value memories, where the number of memories is equal to the number of hidden dimensions in FFNs. This finding underscores the significance of hidden dimension in FFNs. Drawing inspiration from this finding and the successful application of large hidden sizes in FFNs as evidenced by Meta’s 4B model (Tran et al., 2021)1, we hypothesize that an efficient lightweight FFN is not merely about parameter reduction. Rather, it should aim to maintain or even increase the hidden dimension while judiciously reducing the number of parameters involved.
The literature on animal cognition provides some
1They have shown enlarging the hidden size of FFNs to 16384 delivers significant BLEU improvements.
7280
clues for designing lightweight and expressive FFNs. Research on animals’ behavior has shown that group animals such as insects, fish, and some birds can emerge with some incredible abilities to deal with some complex tasks, though each individual owns poor abilities (Couzin, 2009; Conradt and Roper, 2005). This concept resonates with the AI community’s “Swarm Intelligence” paradigm (Bonabeau et al., 1999), which emphasizes the power of collective decision-making. This biological prior motivates us to integrate Swarm Intelligence principles into the FFN design process.
To this end, we propose PartialFormer, an innovative approach to Transformer architecture. At the heart of PartialFormer lies the novel concept of Partial-Level Gated Feed-Forward Networks (PGFFN). Conceived as an ensemble of streamlined FFNs operating in concert, each PG-FFN produces lower-dimensional hidden features. Despite their reduced individual dimensions, the aggregated output of these PG-FFNs either matches or surpasses the hidden dimensions of traditional, larger FFNs, as empirically substantiated in Figure 1. Moreover, we further equipped PartialFormer with a head scaling strategy tailed for efficiently scaling, and a residual-like attention calculation for stable optimization. These techniques empower PartialFormer to efficiently utilize parameters within the same parameter budget.
Our main contributions are as follows: • We introduced PG-FFNs, a method that effi-
ciently reduces parameters and computations, and integrated them into the PartialFormer architecture for high performance. Additionally, we introduced an attention calculation method for stable optimization. • We investigated the scalability of PartialFormer and proposed a head scaling strategy tailored for PartialFormer to efficient scaling. • Rigorous empirical tests across 9 machine translation tasks and 1 abstractive summarization task confirm the effectiveness and efficiency of PartialFormer on machine translation and summarization tasks.

heading: 2 Preliminary: Transformer
text: In this section, we present some prior knowledge about the Transformer. The Transformer block consists of a multi-head self-attention and a feedforward network. Let X ∈ RT×d be a T × d input matrix of T tokens. Each multi-head self-attention
component owns H heads. For simplicity, we omit layer normalization and residual connections.
Multi-Head Self-Attention MHSA aims to model the global dependency among tokens. MHSA computes as follows:
Ai = Softmax( Qi(Ki)
T
√ dk ), (1)
headi = AiVi, (2)
X = H∑
i=1
headiW O i , (3)
where Qi,Ki, Vi denote the query, key and value of i-th head, which are derived from input with three learnable matrices WQi ,W K i ,W V i ∈ Rd×dk as follows: Qi = XW Q i ,Ki = XW K i , Vi = XW V i , respectively. WOi ∈ Rdk×d is a learnable matrix. dk and d denote the head dimension and embedding dimension, respectively. Ai and headi denote the attention matrix and representation of i-th head, respectively.
Feed-Forward Network Feed-forward network is responsible for improving the expressiveness of the whole representation space by adopting an "expansion-activation-reduction" mapping strategy. It computes as follows:
X = ReLU(XW1 + b1)W2 + b2, (4)
where W1 ∈ Rd×dffn ,W2 ∈ Rdffn×d, b1 ∈ Rdffn , b2 ∈ Rd are learnable matrices and dffn denotes the hidden dimension in FFN that is usually set to 4d.

heading: 3 PartialFormer

heading: 3.1 Overall Architecture
text: Figure 2 illustrates the overall architecture of PartialFormer, encompassing both an encoder and a decoder. Although the foundational structure adheres to the design of the vanilla Transformer (Vaswani et al., 2017), there are some notable modifications.
Encoder. Different from vanilla Transformer, each encoder layer in PartialFormer consists of a unified sub-layer that integrates the PG-FFNs into the multi-head self-attention mechanism.
Decoder. Each decoder layer is composed of two types of sub-layers, both of which integrate the multi-head attention mechanism with PG-FFNs. The sub-layers differ based on the type of multihead attention mechanisms employed, specifically
whether it’s a decoder self-attention or an encoderdecoder cross-attention mechanism. Notably, this design is inspired by previous studies (Lu et al., 2019; Gulati et al., 2020), but it differs in that we employ small FFNs, known as PG-FFNs, within each attention head of both the self-attention and cross-attention modules. To reduce computation, we halved the hidden dimension of PG-FFNs. Further decoder comparisons are in Appendix C.

heading: 3.2 Partial-Level Gated FFN
text: Intuition Previous studies (Wu et al., 2020; Mehta et al., 2021; Ge et al., 2022) commonly reduced the parameters in feed-forward networks by decreasing the hidden dimension (e.g., 2048 to 256). Different from them, our key idea involves utilizing a collection of small FFNs to model smaller input features expecting them to collaboratively emerge better performance while consuming fewer parameters, akin to “Swarm Intelligence”.
In the concept of “Swarm Intelligence”, a vanilla FFN can be viewed as a single large individual, which processes the whole feature input, making it, while effective, very resource-intensive in terms of computing power and memory. Assume a vanilla FFN with mappings of 1024->4096->1024, which consumes around 8.4 million parameters. By contrast, if we utilize multiple smaller FFNs (viewed as multiple weak individuals), each of which processes a subset of the input feature and collaboratively utilizes these outputs to generate the final output, the parameter and computation consumption will be significantly fewer. For example, 8 smaller FFNs with mappings of 128->512->128, we can retain the same hidden dimension, such as 8 * 512, while using only 1.05 million parameters. This approach significantly reduces parameters while maintaining the crucial hidden dimension, as em-
phasized in Geva et al. (2021); Tran et al. (2021).
Design of PG-FFNs We have observed that the Transformer architecture inherently consists of multiple smaller subspaces, namely “heads” within the multi-head attention (MHA) mechanism. These heads act as sub-components of the original inputs and retain substantial information from the original data. Besides, the fusion mechanism in MHA enables the consolidation of the capabilities of multiple FFNs. As a result, PG-FFNs should naturally be constructed based on the MHA mechanism. More specifically, we insert multiple FFNs into the place between Eq. (2) and Eq. (3), as shown in the blue part of Figure 2(c).
While group transformation operations could be used to instantiate our idea, they are not optimal on GPUs due to their low I/O efficiency (Ma et al., 2018), causing significant inference latency. To address this, we propose sharing parameters across each FFN within different heads, thereby eliminating the need for group transformation operations. However, directly sharing weights may result in homogeneous representations across different heads, which may potentially hinder the performance (Li et al., 2018). To mitigate this, we further introduce a head-specific gated mechanism. The core idea is to use a set of diverse masks to filter the information of different heads so that the head representation will be more diverse.
Formally, given a set of head features {headi|1 ≤ i ≤ H} and diverse masks {Gi|1 ≤ i ≤ H}, the calculation of PG-FFNs is as:
headi = Gi ⊙ FFN(headi), (5) where FFN(·) is the same as Eq. (4) and Gi is generated via multiplication between the input feature of the block X and a learnable matrix WGi followed by an activation function σ(·), e.g., ReLU,
Sigmoid and Tanh, as follows: Gi = σ(XWGi ). We compared the choice of σ(·) in Table 7.

heading: 3.3 Residual-like Attention Calculation
text: Dong et al. (2021); Wang et al. (2022) have shown that the original location of FFNs plays an essential role in optimizing transformers, e.g., alleviating Token Uniformity. Therefore, it’s vital to consider the impact of altering the FFN placement. Densely residual connections are effective but typically implemented either at the feature level (e.g., DLCL (Wang et al., 2019)) or integrated into the network structure (e.g., Realformer (He et al., 2021)), which are not flexible.
To this end, we design a new variant of the residual connection integrated into the attention calculation, while also decoupling from the network architecture. Specifically, the calculation of attention maps consists of two parts: 1) AG, the global part, and 2) AL, the local part, as shown in Figure 2(c). The calculation of AL remains the same as in the vanilla Transformer, while AG is computed once by using the original embedding as input through Eq. (1) (without softmax operation). Inspired by He et al. (2021), to efficiently fuse these components, we add them together and apply a Softmax function, as follows:
Ai = Softmax(A G i +A L i ), (6)
where AGi and A L i denote the global and local attention map of i-th head. In addition to the benefit of efficient depth scaling (See Appendix G), this approach provides remarkable flexibility in combining different attention mechanisms, specifically tailored to address specific conditions. For instance, it allows for the utilization of local attention to calculate AG when dealing with small datasets (see Appendix F).

heading: 3.4 Efficient Scaling Strategy
text: Though PG-FFN offers the advantage of reducing lots of parameters when applied directly to the transformer, it also leads to marginal performance degradation (see Table 10 (a)). Thus, a crucial aspect of this study is to determine how to effectively utilize the spared parameters. In this work, we adopt a hybrid scaling strategy, which has been validated in computer vision, e.g., EfficientNet (Tan and Le, 2019). Note that our approach differs from EfficientNet, as we incorporate a combination of head scaling and depth scaling into our method.
Head Scaling As aforementioned, PartialFormer is guided by “swarm intelligence” and operates with small subspaces. Expanding the number and size of these subspaces intuitively augments PartialFormer’s capabilities. In response to this insight, we introduced a head-scaling strategy tailored specifically for PartialFormer, involving the direct addition of more heads and the expansion of their width, effectively bolstering its performance.
To achieve this objective, we decouple the relationship between the number of heads and the embedding size, specifically dk ×H ̸= d. This approach shares similarities with methods discussed in Bhojanapalli et al. (2020). However, it differs in its two-step process, which draws inspiration from the inherent redundancy observed in attention maps as discussed in Michel et al. (2019); Clark et al. (2019); Voita et al. (2019); Nguyen et al. (2022); Zheng et al. (2024). Given values for dk, d, and H , we first create intermediate values for Q and K, and then we expand the attention maps to the desired number of heads using a robust MLP network. In the case of V , we generate them directly. This approach allows for the inclusion of more heads in PartialFormer while maintaining the same parameter budget.
We demonstrate that this scaling strategy is naturally well-suited for PartialFormer (see Section 6.4). Furthermore, it can also be regarded as a variation of width scaling, offering two significant advantages: 1) enabling flexible imbalanced computation distribution in encoder-decoder architecture, and 2) preventing an excessive distribution of parameters in the embedding and output layers.

heading: 4 Experimental Setups
text: We assess PartialFormer’s performance across both machine translation and abstractive summarization tasks2. More details are given in Appendix A.
Dataset. For the machine translation task, we selected 9 datasets involving WMT’14 EnglishGerman (En-De), WMT’14 English-French (EnFr), WMT’16 English-Romanian (En-Ro), and six translation tasks from WMT’17 benchmark. We preprocessed the raw data following the standard strategy. For the abstractive summarization task, we utilized the widely-used CNN-DailyMail dataset. We followed the same preprocessing approach as described in Ott et al. (2019). We applied
2We tested PartialFormer’s performance in language modeling, with results in the Appendix.
joint byte pair encoding (BPE) (Sennrich et al., 2016) with sizes of 32K for all the tasks except the En-Ro task (20K), and CNN-DailyMail (30K).
Training & Evaluation. We trained models on GeForce RTX 3090 cards via Fairseq (Ott et al., 2019) toolkit primarily following the training strategy in Wang et al. (2019). For machine translation evaluation, we utilized multi-BLEU (Papineni et al., 2002), COMET-22 (Rei et al., 2022) and sacreBLEU (Post, 2018) scores. Following Wang et al. (2019), beam sizes were 4, 4, and 5 for En-De, EnFr, and En-Ro tasks respectively. Length_penalty of 0.6, 0.8, and 1.3 were applied to En-De, En-Fr, and En-Ro tasks respectively. For the WMT’17 benchmark, beam size and Length_penalty were set to 4 and 1, respectively. We used an ensemble of last 10 checkpoints. For abstractive summarization, we set beam size, Length_penalty, minimum length and maximum length to 4, 2.0, 55 and 140, respectively. The evaluation metric was F1-Rouge (Lin, 2004)(Rouge-1, Rouge-2 and Rouge-L).

heading: 5 Experiments

heading: 5.1 Machine Translation
text: Table 1 presents the results for the WMT’14 En-De task. N−M , d, dk, H and sBLEU denote encoderdecoder depths, embedding dimension, head dimension, number of heads and SacreBLEU, respectively. We made the following observations:
• PartialFormer achieves BLEU scores of 28.60,
29.56, and 30.09 in three different configurations, surpassing the standard Transformer by 1.17 BLEU points, 1.56 BLEU points, and 1.04 BLEU points with a similar model capacity. These observations are further supported by COMET-22 and sacreBLEU scores. • Without the head scaling strategy, PartialFormer performs slightly worse than the standard Transformer (27.15 vs. 27.43, 27.88 vs. 28.00, and 28.86 vs. 29.05) but is significantly more parameter-efficient (42M vs. 62M, 36M vs. 62M, 66M vs. 118M). This is due to our PG-FFN structure, which maintains high hidden dimensions while reducing parameter usage. • PartialFormer surpasses other multi-branch Transformers and state-of-the-art weightsharing methods like ODE Transformer (Li et al., 2022a), as well as strong baselines such as Mega (Ma et al., 2022). Notably, ODE Transformer and Mega use extra relative position encoding and require more computational resources. Moreover, while Mega and DMAN train for up to 500K updates and 220 epochs, achieving BLEU scores of 29.01 and 29.10, our strategy involves only 50K updates, leading to their sub-optimal scores of 28.11 and 27.54 under similar conditions.
Tables 2, 3, and 4 showcase results for the WMT’14 En-Fr, WMT’16 En-Ro, and WMT’17 benchmarks, respectively. Similar trends are observed in these
Model N d dk H Param BLEU COMET-22 Delight (Mehta et al., 2021) - 640 - - 53M 34.70 - Subformer (Reid et al., 2021) - - - - 48M 34.70 - ODE Transformer (RK2 γ) † (2022a) 6 1024 64 16-16 192M 35.00 82.63 Transformer 24 512 64 8-8 111M 35.00 82.11 PartialFormer 24 320 40 24-24 48M 35.30 82.52
Table 3: Results on the WMT’16 En-Ro task. †denotes re-implementation with same data and training strategy.
Model Fi←→En De←→En Lv←→En Avg. Fi→ En En→ Fi De→ En En→ De Lv→ En En→ Lv
Transformer 26.07 22.14 35.04 28.59 17.59 16.23 24.27 PartialFormer 27.48 23.35 35.60 29.91 19.65 17.37 25.56
Table 4: Results on the WMT’17 benchmark. PartialFormer has the same depth and d as the Transformer but consumes 1M fewer parameters on average.
tasks as in the En-De task.
MACs Comparison. Table 5 displayed the multiplication-addition operations (MACs), a metric for measuring neural network computations, on the En-De task. We made the following observations: 1) A deeper and narrower Transformer architecture consumes fewer computations while exhibiting superior performance (#1 vs. #2), 2) PartialFormer achieves comparable performance to the vanilla Transformer with the same width and depth, while utilizing fewer computations and parameters (#2 vs. #3), and 3) Head scaling is an efficient scaling strategy for PartialFormer to significantly improve its capacity (1.68 BLEU points) by adding 1.7B MACs and 32M parameters (#3 vs. #4).

heading: 5.2 Abstractive Summarization
text: Table 6 exhibited results on the CNN-DailyMail task. We can see that PartialFormer achieves better performance, as evidenced by higher Rough1, Rough-2, and Rough-L scores, despite having fewer parameters (37M vs. 61M). This highlights the efficiency and effectiveness of the PartialFormer architecture in this task.
Model N -M d dk H Param RG-1 RG-2 RG-L

heading: 6 Analysis

heading: 6.1 Ablation Studies
text: Table 7 presents an ablation study of PartialFormer on the WMT’14 En-De task, demonstrating the critical role of each component. Omitting any element causes performance decline, underscoring the holistic design. The PG-FFN removal (#3 vs. #4) results in a large performance drop of 2.05 BLEU points, despite a mere 16 million parameters reduction. This evidence corroborates previous findings (Dong et al., 2021) on the subpar performance of pure attention networks sans FFN, highlighting the essential role of PG-FFN in PartialFormer.
Besides, Table 7 shows the results of different PartialFormer configurations on the WMT’14 En-De task. The encoder-decoder PartialFormer achieves the highest performance, reaching 29.56 BLEU points, indicating the effectiveness of our approach in enhancing both the encoder and the decoder. Employing our concept to either the encoder or the decoder individually also improves performance, yet the encoder-decoder configuration persistently surpasses others, marking the greatest performance improvement.

heading: 6.2 Comparison of Gating Strategy
text: Table 7 (#9 and #10) presents a comparison of various activation functions used in PG-FFN. The results indicate that the default choice, ReLU activation, yields the best performance. One explanation is that the ReLU activation provides hard masks for filtering the information of different heads, compared to other activation functions. Such hard masks can make different heads more diverse.

heading: 6.3 Hyper-Parameter Analysis
text: Since the proposed method relies on multiple parameters, we conducted additional experiments and analyses with different hyper-parameters, including the number of heads, head dimensions, and embedding dimensions, to further strengthen the robustness of our findings. Table 8 presented the results on the WMT’14 En-De task. We can observe that PartialFormer demonstrates strong performance across various choices of H , dk, and d. This suggests that the superiority of PartialFormer arises from its efficient architecture design rather than hyper-parameter optimization.

heading: 6.4 Analysis of Scaling Approaches for PartialFormer
text: To disentangle the contribution of our proposed scaling method from the PartialFormer architecture, Figure 3 compares the WMT’14 En-De performance of different scaling methods. Specifically, the initial setting is the PartialFormer (N −M = 6 − 6, H = 8 − 8, d = 360). It’s important to note that our hybrid scaling initially employs depth scaling, followed by head scaling. In general, all scaling methods improve BLEU scores with the cost of more parameters, but our hybrid scaling method can further improve BLEU, by up to 2.3%, than other scaling methods, suggesting the importance of our proposed hybrid scaling.
Head scaling can also improve the vanilla Transformer, though it is not as effective as in Partial-
Former. Notably, PartialFormer attains 0.0525 BLEU per million parameters, significantly outperforming the vanilla Transformer (0.0243). This highlights the suitability of head-scaling for PartialFormer’s design, a key contribution of this paper.

heading: 6.5 Analysis on Behaviours of FFN
text: Metric. Following Zhang et al. (2022), we examine FFN behaviors across four aspects: activation neuron count (namely nact.), FFNs’ hidden dimension, activation-neuron ratio (activations divided by hidden dimension, namely Ract.), and FFN efficiency (activations divided by parameters, namely ηffn). Notably, for PartialFormer, the hidden dimension represents the concatenation of hidden dimensions from all smaller FFNs.
Results. Figure 4 (a-c) exhibits the results on the En-De test set. It is evident that PartialFormer has a lower activation ratio than the vanilla Transformer, as shown in Figure 4 (b). This indicates that PG-FFNs present lower utilization of the hidden dimension compared to the vanilla FFNs. However, our PG-FFN is parameter consumption friendly, enabling larger hidden layer dimensions with the same parameter budget (e.g., 5400 vs. 1440). Despite lower utilization of hidden dimension, it can still own more activated neurons, as depicted in Figure 4 (a). Additionally, our PG-FFN exhibits higher efficiency compared to vanilla FFNs, as shown in Figure 4 (c).

heading: 6.6 Analysis on Head Diversity
text: Metric. We select the same metric, namely Doutput, as that in Li et al. (2018) to measure the diversity among head features. In this metric, a larger value indicates a higher level of diversity.
Results. From Figure 4 (d), we can observe that PartialFormer exhibits more diverse head features compared to the vanilla Transformer. This aligns with previous study (Li et al., 2018), which demonstrates the positive impact of head feature diversity
on the Transformer model’s performance. Thus, we conclude that the insertion of FFNs into attention mechanism may be a more optimal design.

heading: 6.7 Efficiency Analysis
text: Convergence Analysis Table 9 (a) compared the convergence updates, training speed and BLEU scores of PartialFormer with other methods. We made the following observations:
• PartialFormer and ODE Transformer do not require more training updates to achieve higher performance than vanilla transformer, unlike other strong baselines. • All improved methods indeed lead to increased running latency. • PartialFormer achieve highest BLEU scores among all the comparisons. Overall, we believe PartialFormer can achieve significant performance improvements while maintaining good training efficiency.
Inference Analysis Table 9 (b) exhibits the inference efficiency on the test set of En-De task. We can see following observations: 1) Under the constraints of desired memory and performance, PartialFormer exhibits higher inference efficiency (6579 vs. 4325) when compared to the vanilla Transformer (#1 vs. #2). This revealed that PartialFormer has good practicability, and 2) In comparison to ODE Transformer, PartialFormer achieves similar inference speed and performance while significantly reducing memory consumption. This underscores PartialFormer’s superiority over weightsharing methods by effectively eliminating redun-
dant computations.

heading: 6.8 PG-FFNs vs. Vanilla Lightweight FFN
text: In this section, we further emphasized PG-FFNs’ superiority over vanilla lightweight FFN.
Settings. We replaced the Transformer’s FFNs with our PG-FFNs. In the decoder, we only integrated PG-FFNs for cross-attention, aligning with the vanilla Transformer. We set Transformer with reduced FFN hidden dimensions (384) as baseline.
Results. Table 10 (a) showcases the superior efficiency of our PG-FFNs. They outperform vanilla lightweight FFNs (26.82 vs. 26.07) with similar computational resources (40M vs. 41M, 7.7B vs. 7.7B). This is attributed to PG-FFNs’ ability to maintain a large hidden dimension while using fewer parameters and computations, setting them apart from existing lightweight FFNs.

heading: 6.9 Combination with Existing Architectures
text: We further investigated the adaptability and effectiveness of PartialFormer by applying it to three kinds of existing state-of-the-art architectures: 1) weight sharing methods (Lan et al., 2020), 2) gated linear units (Dauphin et al., 2017) and 3) deep Transformer methods (Wang et al., 2019). We utilized the ODE Transformer (Li et al., 2022a), known for its parameter efficiency. Additionally, we selected Swi-GLU (Shazeer, 2020) and DLCL (Wang et al., 2019).
Table 10 (b) shows the results. PartialFormerDLCL achieved the highest performance, outper-
forming PartialFormer by 0.32 BLEU points, while PartialFormer-GLU showed the smallest improvement with an increase of 0.11 BLEU points. We attribute this to the fact that DLCL is an architecturelevel modification addressing different issues from PartialFormer. In contrast, both GLU and ODE focus on the parameter-efficiency problem. Although ODE is also an architecture-level modification, its goal significantly overlaps with that of PartialFormer, leading to moderate performance improvements when combined. This indicates that PartialFormer already significantly enhances parameter efficiency, as adding ODE and GLU does not yield substantial performance gains.

heading: 7 Related Work
text: Lightweight Transformers Several strands of research have been dedicated to enhancing the parameter efficiency of the Transformer architecture, each taking a distinct approach to the problem at hand. The first category aims to mitigate redundancy directly through architectural innovations, employing more efficient transformation operations (Mehta et al., 2019, 2021), integrating disparate yet synergistic patterns (Wu et al., 2020), or leveraging neural architecture search techniques (So et al., 2019). Another avenue of research explores weight sharing as a means of improving parameter efficiency, exemplified by the Universal Transformer’s cross-layer parameter sharing strategy (Dehghani et al., 2019; Reid et al., 2021). Moreover, Li et al. (2022a) introduced an ordinary differential equation-inspired weight-sharing approach to achieve higher performance. Different from them, our study focus on the design of lightweight FFN.
Multi-Branch Transformer The multi-branch strategy is widely used in Transformer design. Weighted Transformer (Ahmed et al., 2017) employs a multi-branch FFN, while Multi-attentive Transformer (Fan et al., 2020), Multi-units Transformer (Yan et al., 2020), and Multi-Path Transformer (Lin et al., 2022; Li et al., 2023) extend this
concept to different components of the Transformer. Our PartialFormer can be viewed as a pure multibranch architecture based on natural subspaces.
Scaling Strategy in Transformer Deepening (Bapna et al., 2018; Wang et al., 2019; Li et al., 2020) and widening (Vaswani et al., 2017; Wu et al., 2021) Transformer have been well-acknowledged as two strategies to improve the capacity of Transformer in literature. In this work, PartialFormer adopts two alternative strategies to improve the capacity: specifically, it enhances both the number of attention heads and the dimensions of each head.

heading: 8 Conclusion
text: In this paper, we present PartialFormer, a new parameter-efficient Transformer architecture that offers an alternative approach to the design of the lightweight FFN. By employing multiple small FFNs and leveraging matrix factorization techniques, PartialFormer effectively reduces the number of parameters in the FFN. Moreover, we propose two innovative operations to further efficiently enhance the model capabilities. Experimental results across various machine translation tasks showcase the significant performance improvements achieved by PartialFormer, while maintaining comparable parameter consumption.

heading: Acknowledgments
text: This work was supported in part by the National Science Foundation of China (No.62276056), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Fundamental Research Funds for the Central Universities (Nos. N2216016 and N2316002), the Yunnan Fundamental Research Projects (No. 202401BC070021), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009).
Limitations
Despite the potential advantages of Partialformer in terms of parameter utilization and performance
within a limited parameter budget, it is important to note that the existing conclusions regarding its effectiveness have not been thoroughly examined in the context of large-scale datasets and a higher number of parameters. Further research is needed to validate the claims and assess the scalability of Partialformer in more challenging scenarios.

heading: A Detailed Setups of Experiments
text: A.1 Dataset
Table 11 displays the statistics of all the 9 translation task.
A.2 Training Details
Table 12 and 13 exhibits the training details on all translation tasks.
B Implementation of Previous State-of-the-art Methods
The accuracy of fairseq-based translation results can vary due to tokenization methods and other factors. To address fairness concerns, we reimplemented three state-of-the-art approaches in our codebase. To ensure absolute fairness, we employed the identical training strategy and data usage as in our PartialFormer model.
Data. The dataset is sourced from Google’s open release, featuring BPE operations totaling 32K.
Training Strategy. Our training strategy is the same as that of Wang et al. (2019), where 0.002 learning rate, 16000 warmup steps, pre-norm, relu_dropout=0.1, attention dropout=0.1, 4096 tokens per GPUs (8 GPUs) and update the parameters every 2 steps.

heading: C Ablation on Design of Decoder
text: The design of the Decoder is a crucial component of the Transformer architecture due to its direct association with decoding. We evaluated three configurations: 1) Integrating PG-FFNs into both the decoder’s self-attention and cross-attention, while halving the hidden dimension, 2) Incorporating PGFFNs solely into the decoder’s cross-attention, and 3) Incorporating PG-FFNs solely into the decoder’s self-attention.
Table 14 exhibited the results on the WMT’14 En-De task. Our observations are as follows: 1) The first configuration yields the best performance, aligning with the insights from Gulati et al. (2020); Lu et al. (2019), 2) Using a single PG-FFN in each layer also delivers commendable results with a score of 29.21, and 3) Excluding PG-FFNs from the decoder’s cross-attention results in erratic training, which is expected since there are no FFNs to handle the cross-attention features.

heading: D Metric Definition
text: D.1 Measurement of Head Diversity Following Li et al. (2018), we measure the head diversity as follows:
Doutput = exp(− 1
H2
H∑
i=1
H∑
j=1
|Oi ·Oj | ∥Oi∥∥Oj∥) (7)
During evaluation, we calculate the metric on all samples and average the values to obtain the final result.

heading: E More Comparison with Previous Lightweight Transformer
text: Table 15 presents a comprehensive comparison of previous lightweight Transformer models on the En-De task’s test set, with a specific focus on operating within a smaller parameter budget. The

heading: AG AL Param BLEU
text: results prominently showcase the outstanding performance of PartialFormer, even when faced with constraints on model capacity. This outcome further emphasizes the superior capabilities of PartialFormer in scenarios with limited resources.

heading: F PartialFormer with Different AG for Small Dataset
text: Table 16 showcases the results of PartialFormer on the WMT’16 En-Ro task, a small-scale translation dataset, specifically when AG is calculated using
local attention (Shaw et al., 2018). Notably, these results reveal that by adopting such an approach, PartialFormer achieves an impressive BLEU score of 35.76. We hope this can shed lights on the area of model integration.

heading: G Analysis on Token Uniformity
text: Following (Dong et al., 2021; Wang et al., 2022), we measure the token uniformity among token representations. We use pearson correlation to compute it.
From Figure 5, we can observe that PartialFormer owns a lower token uniformity among token representations than the vanilla Transformer, revealing that PartialFormer can benefit from depth scaling efficiently (Dong et al., 2021; Wang et al., 2022).

heading: H Preliminary Experiments on Language Modeling
text: We also evaluate the effectiveness of PartialFormer on the language modeling task.
Dataset. For the language modeling task, we utilized the WikiText-103 dataset for evaluation. The training set comprises 103 million words from 28,000 articles, while the validation and test sets contain 218,000 and 246,000 words, respectively. We followed the data acquisition and preprocessing instructions from Fairseq (Ott et al., 2019).
Training & Evaluation. The training and evaluation settings adhere to the standard guidelines for language modeling in PyTorch (Ott et al., 2019). We trained all models over 286,000 updates.
Results. Table 17 exhibited results on the WikiText-103 task. PartialFormer surpasses the Adaptive Input model (Baevski and Auli, 2019) with a lower test perplexity of 19.87 compared to 21.11. Remarkably, PartialFormer achieves this with slightly fewer parameters (143M vs. 147M), demonstrating its efficiency and effectiveness as a
language model for WikiText-103. We will present more comprehensive experiments in the future.

heading: Future Work
text: We will present more comprehensive experiments in the future