MSCFFN: A New FFN with Multi-Space Cross to Accelerate Transformer
Authors: Tang Dongge
Qing Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: New FFN structure
MSCFFN
Multi-Space Cross method
Accelerate Transformers

Abstract: Transformer models have achieved impressive success in various natural language processing tasks. But it is also limited used in some areas and the heavy computation complexity is one of the main limitations. Many model structures have been proposed to reduce the computation complexity and some are really effective. The previous research can be divided into two categories. One is to use more effective training and inference strategies and the other is focused on how to replace the standard self-attention mechanism with linear attention method. Differently, we revisit the design in Transformer and find that the feed forward network (FFN) is also computationally expensive, especially when the hidden dimension is large. In this paper, we propose a new FFN structure, named MSCFFN, which splits the large matrix space to several small space to reduce the computation complexity and uses the Multi-Space Cross method to ensure the accurate result. To the best of our knowledge, this is the first time to redesign FFN to accelerate Transformers. We experimentally validate the effectiveness of the proposed method on the Long-Range Arena benchmark. And the results show MSCFFN can achieve a faster speed with a similar or even better accuracy.

MSCFFN: A New FFN with Multi-Space Cross to Accelerate Transformer
Dongge Tang Beihang University, Beijing, China Du Xiaoman Financial, Beijing, China tangdongge@buaa. edu.cn
Qing Yang Du Xiaoman Financial, Beijing, China
yangaingeduxiiammnn com
Abstract
The effectiveness of transformer comes from the multi-head self-attention mechanism (MHSA), fully connccted feed-forward network (FFN) and the deep layers. The MHSA helps model to learn the relation between tokens in different position of a sequence, the FFN helps enlarge the network space and enhance the non-linear capability, and the deep layers are helpful to build a more complex sentence relationship. However, these operations also cause the high computation complexity, and hard to use for user with low computation re- source. To tackle this essential obstacle, there has been much prior literature on improving the effi- ciency of Transformers. The main research consists of two parts, one is to use more effective strategies. For example, Pyramid- PyRT BERT (Huang al., 2022) uses the core-set based token selection method reduce complexity, and PCEE-BERT (Zhang al., 2022) accelerates BERT using patient and confi- dent early exiting method. The other is focused on how to reduce the computation complexity of the standard self-attention, which is quadratic com- plexity correlated with sequence length, to linear attention method. Such as Performer (Choroman- ski al., 2020), Longformer (Beltagy et al., 2020), Cosformer (Qin et al., 2022), Flowformer (Wu et al., 2022) and so on.
Transformer models have achieved impressive success in various natural language process- ing tasks. But it is also limited used in some areas and the heavy computation complexity is one of the main limitations. Many model structures have been proposed to reduce the computation complexity and some are really effective. The previous research can be divided into two categories. One to use more effec- tive training and inference strategies and the other is focused on how to replace the stan- dard self-attention mechanism with linear atten- tion method. Differently, we revisit the design in Transformer and find that the feed forward network (FFN) also computationally expen- sive, especially when the hidden dimension is large. In this paper, we propose a new FFN structure, named MSCFFN, which splits the large matrix space to several small space to re- duce the computation complexity and uses the Multi-Space Cross method to ensure the accu- rate result. To the best of our knowlee inowledge, this is the first time to redesign FFN to accelerate Transformers. We experimentally validate the effectiveness of the proposed method on the Long-Range Arena benchmark. And the re- sults show MSCFFN can achieve a faster speed with similar or even better accuracy. Our codes are available at sttps://gtt c henriTang/Transe waataaan//rraaffrrrrsrs.. Speed.
Although these methods have been proven be very effective in reducing computation com- plexity, they almost don't 't discuss the influence of hidden dimension to computation complexity. fact, transformer models are also quadratic com- plexity correlated with hidden dimension. When the hidden dimension is large, it is computation- ally prohibitive to train or inference (Zhang and Cai, 2022). Actually, to achieve better results, Bert xlarge uses 2048 of hidden dimension and Albert xxlarge (Lan et al., 2019) uses 4096 of hidden di- mension, which really computationally expen- sive for a common researcher. For the transformer models, the main unit with quadratic complexity of
1 Introduction
Since Transformer models (Vaswani et al., 2017) were proposed, especially the Bert (Devlin et al., 2018) with pre-trained language models, they have achieved state-of-the art results in various domains of natural language processing, such machine translation (Liu al., 2020; Bao al., 2021), text classification (Xiong et al., 2021; Yang et al., 2020), natural language inference (Jiang and Marneffe, 2019; Guo et al., 2019), question answering (Yang et al., 2019; ; Heo et al., 2022) and so son.

---

hidden dimension (d) is FFN which has the com- plexity of ((8d²) in vanilla transformer, because the dimension of inner-layer is usually set 1d for a better result. Therefore, in this paper, we focus on how to reduce the computation complexity of FFN.
elombiev al., 2021) uses grouped transforma- tions to reduce the density of fully connected layers. Sparse mixtures of experts (sparse MoEs) reduce the computation cost through conditional compu- tation which only activates a subset the overall model, such as (Zhang et al., 2021; Fedus et al., 2022).
According to the characteristic of quadratic com- plexity related to hidden dimension, if we shrink the hidden dimension to d, the computation com- plexity will be reduced to n2 of the original. There- for, we firstly split the original large matrix space to several matrix spaces, and we apply the different weight parameters for each small matrix to con- duct the fordrd network separately. Obvi- ously, the representation ability of n small matrices with de which is the space of weight parameters is no much less than it of d², and it is easy to cause the effect decline. In order to solve this problem, we propose a method of space cross mechanism, and we name the final model MSCFFN. Through the cross of different small space, we can enlarge the matrix space and obtain a stronger representation capability. Furthermore, benefitting from the in- formation from different representation subspaces, a larger coefficient of inner-layer and the deeper network of FFN, MSCFFN can achieve a better accuracy with a faster speed. We conduct experi- mental study on the Long-Range Arena benchmark (Tay et al., 2020), and the results demonstrate that our MSCFFN is effective in speeding up and per- forms better than baselines.
Linear attention method: This method is resolve the quadratic complexity of vanilla self- attention mechanism and try to reduce the complex- ity of On²2) to (nnoggn)) or even O(n). For ex- ample, Leeasly-se hashing (I.SH) attention (Kitaev et al., 2020) applies a multi-round hashing scheme to compute dot- dot-product attention. Long- former utilizes a sliding windows to learn the token relation in a local window. Performer introduces an unbiased linear estimation of the attention matri- ces, Cosformer proposes a linear attention by using the kernel methods with a non-linear re-weighting scheme.
3 Method
3.1 Preliminaries of vanilla Transformers In this paper, only talk about the encoder struc- ture, but our model is universal for all FFN. The Transformers is composed of stack of N identical layers, and each layer contains MHSA and FFN. Let us use X, E Rixd and X E Rlxd to repre- sent the input of MHSA and FFN respectively, in which / is the sequence length and d is the hidden dimension.
2 Related work
For the MHSA, the input is firstly linearly pro- jected to Q E Raxd K E Raad V Ruxd by the parameter matrices WQ E Raxdd WK Raxd WV Raxd,
As mentioned in section I, The most common meth- ods to reduce the computation complexity are as follows:
(Q, V) = (X, x,W*, X,WV) (1)
Effective strategies: with this method, they don't change the structure of MHSA and FFN, but change the strategies of layers to train or tokens train. For example, (Dai et al., 2020) proposed Junnei--Trastoonn which gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. (Huang et al., 2022) proposed Pyramid- Beramid-BERTT which used a core- set based token selection method to successively shorten sequence length and void expensive compu- tation resource. PCEE-BERY (Zhang et al., 2022) was proposed by Zhang al., and it achieves the goal of speedup through exiting early when enough numbers of consecutive intermediate layers are con- fident about their prediction. GroupBERT (Ch-
And then the attention can be computed as:
QKT Vdk
Altention(Q... Q.K,V) ) = softmax
V (2)
where de is the dimension of Q and K. After that, the attention will be linearly transformed to the output by the parameter matrices WO Raxd And residual connection with layer normalization will be added. The output is also the input of FFN:
X = LN LN (Attenttto (Q, K, K,V' + X.) (3)
For the FFN, two linear transformations with a

---

ReLU activation will be implemented as:
Norm/Ad))
FFN(X;) = mm.000, X + 5 by2 (4)
Oy Lineer/Conca)
where Wi E Rdxd; and W E Rdlxd are weight parameters. by1 E Rd and by2 R are bias parameters. After that residual connection with layer normalization will be added.
En
G
G
Ox = LN(FFN(X,) + Xÿ)
(5)
From the above equations, we calculate and obtain the computation complexity of MHSA in each layer is approximately equal (4ld2 + d), the computation complexity of FFN in each layer is approximately equal to (88d2) and the total computation complexity in each layer is ap- proximately equal to O(12dd + 2/2d),
4
d). The ratio of FFN to MHSA equals 7=7. Of course, when >>> d, the complexity of FFN is not worth con- sidering. But in many cases, this condition is true, and even << d, such as the daily dialogue system, real-time machine translation and so on. Therefore, optimizing the structure of FFN is really meaningful for reducing the computation complex- ity. What's more, for the improved linear attention mechanism, the FFN occupies a large proportion no matter how long the sequence is. Our MSCFFN be combined with linear attention and speedup the transformer models further.
Linear
4d
Xy
Figure I: General framework of MSCFFN (here
we set m to 6 to achieve a better result. We have to admit that the sum of subspace of R x md is smaller than the original space of Rdxd the cross method of subspace is proposed improve the range of representation space. Firstly, The input L divided into pairs. Secondly, we use non-linear method to one of pairs. Finally, the element-wise product is applied to one with the other in pairs.
3.2 MSCFEN Model
this part, we present the proposed MSCFFN model detail.
C Relu(Lzi O L2s
(8)
The structure of MSCFFN is shown in Figure 1. First of all, we transfer linearly the input of FFN to new space by the parameter W/3 E Rdxd and bys Ied so that we can split it into n subspaces casily.
After that linear transformation is used to project from high dimension low dimension. And we concat the results to map linearly the original dimension of Xy.
CLi = C,Wos + bbi O, = Concat(Cr,)Wc bet (10)
(9)
I Xx + by3 = Concal I2...., ..., In] (6)
where 1 C x n is the th matrix in subspace. And then we use n different parameters to trans- former the I; to larger space.
where Wos = and WeL R$ xd Finally, residual connection followed by a layer normal- ization applied as the original FFN.
L = I,W + br
3.3 Theoretical Analysis for MSCFFN
In this equation, WT: Ra x md and by: R are the parameters in which m indicates the coeffi- cient of lnner-layer. To make the network easier to learn, the inner-layer usually has a bigger dimen- sion and in vanilla FFN m is set to 4. In our work,
Effectiveness of speed: Through equations above, can calculate the theoretical computation complexity of MSCFFN which equals O((1.5 2m )dd). Usually, we set m to and n to 12, so the theoretical computation complexity is (22.ldd)

---

Model
ListOps
Text
Retrieval
Image
Pathfinder
Avg
Performer
18.01
65.40
53.82
42.77
77.05
51.41
Longformer
35.63
62.85
56.89
42.22
69.71
53,46
Transformer
36.37
64.27
57.46
42.44
71.40
54.39
Flowformer
38.70
64.29
62.24
43.20
73.95
56.48
Performer+MSCFFNN
19.67
66.21
56.82
42.77
76.83
52.46 (+1.05)
Longformer+-MSCFNN
36.70
63.61
57.59
41.26
70.25
53.88 (+0.42)
Transformer+MSCFFNN
37.25
64.51
60.32
43.61
71.58
55,45 (+1.06)
Flowformer+MSCFFN
38.65
64.74
61.98
43.96
74.26
56.72 (+0.24)
Table 1:
Results on the Long-Range Arena.
Model Flowformer+-MCCHHN
ListOps
Text
Retrieval
Image
Pathfinder
Avg
38.65
64.74
61.98
43.96
74.26
56.72
Flowformer++1FN.or di
36.75
63.32
60.47
42.12
71.48
54.83 (-1.89)
Flowformer+FFNN
37.59
64.16
61.18
42.91
72.96
55.76 (-0.96)
Table 2:
Comparing multi-space cross method with FFN and FFN MoEE The toNtow indicates the vanilla FPN with low dimension of inner-layer. The FFN MoE indicates the method of sparse mixture of experts from Switch Transformer (Fedus et al., 2022) used to FFN.
Model
Steps per second
4 Experiments
Sequence Length
128
1K
4K
Transformer
20.21
3.45
In this section, we testify the effectiveness and generality of our proposed model MSCFFN the dataset of Long-Range Arena (LRA).
Longformer
12.30
2.44
Performer
25.48
4.17
1.18
Flowformer
14.29
4.50
1.16
LRA is a benchmark which contains five dif- ferent tasks: Long sequence ListOps (Nangia and Bowman, 2018), Byte-level text classification (Maas al., 2011), document retrieval (Radev et al., 2013), image classification sequence of pixels (Krizhevsky et al., 2009), and Pathfinder (Linsley et al., 2018). Their length of sequences ranges from 1K to 16K. LRA a suitable bench- mark for testing the quality of efficient transformer variants, especially for the ability of long input sequence. We chose this dataset to verify that MSCFFN is general for different tasks and is effi- cient in different sequence length. We utilize di- rectly the official method of data processing, exper- iment protocol and training parameters' to ensure the fair comparison. And we use MSCFFN directly to replace the original FFN in four base models. The experiment is conducted on 4 V100 GPUs. To ensure result reliability, we conducted five sepa- rate experiments for each task with different seed values, averaging them for the final reported result.
Transformer+
29.09
4.55
Longformer+
20.38
2.70
Performer+
41.22
6.67
1.85
Flowformer+ 24.93
7.14
1.82
Table 3: Efficiency analysis with the hidden dimen- sion 768, layers 6, number of heads 12, inner-layer dimension 3072, and batch size 32 on byte-level text classification benchmark. "+" represents MSCFFN. "." indicates out-of-memory situation.
Compared with the original FFN whose complex- ity is O(8Id), MSCFFN can save more than 60 percent of the computing resources.
Ability of model representation: Multi-subspaae help the model to learn different representations actively. The cross mechanism of subspace and the larger coefficient of inner-layer are beneficial enlarge the matrice space and obtain stronger representation capability. What's more, MSCFFN has a deeper network which 5 compared with in original FFN, and this motivates the model to learn more complex representations.
The experimental results are shown Table And we can see that they all achieve a better per- formance in average score than baselines. The
httpp::///t thub. com com/google-researe long long-ranger range-nngaaaaaa arena

---

MSCFFN brings 1.06 on Transformer and 1.05 on Performer averaged promotion respectively. There- fore MSCFFN has more competitive performance over the original FFN.
equals zero. Therefore, although it happens with low possibility, constraint should be added avoid it. What's more, we believe that there are more efficient methods to conduct the cross mecha- nism. We leave all this for future work and we will refine it more perfect theoretically and experimen- tally.
To further prove the effectiveness of our multi- space cross method, we conduct ablation studies whose results shown in Table 2. The FFNtow-ds indicates the vanilla FFN with low dimension of inner-layer, and we set it to vddd to ensure that the computation complexity is close to that of MSCFNN, The FFNS - indicates the method of sparse mixture of experts from Switch Trans- former (Fedus et al., 2022) used to FFN, meanwhile we set numbert of experts to and the hidden di- mension of single expert to vēd. d. The experimental 2
Ethical Considerations
All the datasets our experiments are already pub- lic by other researches and can be acquired easily from network. And they don't contain real per- sonal information and will not cause any harm. To simplify the research of others, we will release the code later.
results show that directly reducing the dimension FFN for faster speed is harmful to the effect, even if sparse moe are used.
References
We also conduct experiment in efficiency anal- ysis, which is shown in Table 3. To make the re- sults more general, we use the model parameters of common big models and test it over larger range of length (128-> 4K). The results indicate that MSCFFN can speedup all these models both in short and long sequence. The MSCFFN can achieve at most 1.7x speedup on Flowformer and 1.6x speedup on Performer in Table 3.
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. arXiv preprint 1rXi:22105.447.1
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXi::2004.05//3
Ivan Chelombiev, Daniel Justus, Douglas Orr, Anastasia Dietrich, Frithjof Gressmann, Alexandros Koliousis, and Carlo Luschi. 2021. Groupbert: Enhanced trans- former architecture with efficient grouped structures. arXiv preprint 106.v:2106.0582.
5 Conclusions
In this paper, we propose a new FFN structure named MSCFFN, which can replace the original FFN in Transformer models and speedup model training and inference matter how long short the input sequence is. In MSCFFN, we reduce the computation complexity through splitting the ma- trix space to several small space. Benefiting from multi-space cross mechanism, a larger coefficient of inner-layer and a deeper network, MSCFFN can achieve comparable or even better performances than the vanilla FFN. The experiment results on five tasks of LRA demonstrate the effectiveness and generality of our proposed model MSCFFN. Furthermore, the proposed model is probably not only used for FFN in transformer models, but also for other model structures which contain FFN.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Aftoz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint 2xXiv:2009.1479
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020.
Filtering out sequential redundancy for efficient language processing. Ad- vances in neural information processing systems, 13:4271-4288
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint 1rxiv:1810.04805.
William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parame- ter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):2232- 5270.
Limitations
We have note that the cross mechanism of sub- space by element-wise product will fail to smaller space in a small probability. For example, two spaces are orthogonal, the wles product
Maosheng Guo. Yu Zhang, and Ting Liu. 2019, Gaus- sian transformer: a lightweight approach for natural language inference. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 6489-6498

---

Yu-Jung Heo. Eun-Sol Kim, Woo Suk Choi, and Byoung-Take Zhang. 2022. Hypergraph trans- former; seaklissu multi-hop reasoning for knowledge-basee visual question answering. arXiv preprinl arXiv:2204. 10444888..
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sehastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. arXiv preprint 20Xi::001..040000
Xin Huang. Ashish Khetan, Rene Bidart, and Zohar Karnin. 2022. Pyramid-bert: Reducing complexity via successive core-set based token selection, arXiv preprint 2203 21..22223..888.. 14380.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Nanjiang Jiang and
de Marneffe. 2019. Evaluating bert for natural language inference: A case study on the commitmentbank. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- tional joint conference on natural language process- ing (EMNLP-IJCNIPP.
Haixu Wu, Jialong Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long, 2022. Flowformer: Linearizing transformers with conservation flows. arXiv preprint arXiv:2202.0622.
Yijin Xiong. Yukun Feng, Hao Wu, Hidetaka Kami- gaito, and Manabu Okumura. 2021. Fusing label embedding into bert: An efficient improvement for text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 1743-1750.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint (tt::000000445!
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learn- ing multiple layers of features from tiny images.
Linyi Yang, Eoin M Kenny, Tin Lok James Ng, Yi Yang, Barry Smyth, and Ruihai Dong. 2020. Generating plausible counterfactual explanations for deep trans- formers in financial text classification. arXiv preprint 20Xi:2010.12212..
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint 1909:/990.11999
Wei Yang. Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019, End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01711
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. 2018. Learn- ing long-range spatial dependencies with horizontal gated recurrent units. Advances in neural informa- tion processing systems, 31.
Yizhe Zhang and Deng Cai. 2022. Linearizing trans- former with key-value memory bank. arXiv preprint arXiv:2203.1 12644.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Zhen Zhang. Wei Zhu, Jinfan Zhang. Peng Wang, Rize Jin, and Tae-Sun Chung. 2022. Peee-bert: Acceler- ating bert inference via patient and confident early exiting. In Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 327-338.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Moefication: Transformer feed-forward layers are mixtures of ex- perts, arXiv preprint 1rx0v:2110.01786.
Andrew Maas, Raymond E Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Nikita Nangia and Samuel Bowman. 2018 Listops: A diagnostic dataset for latent tree learning. arXiv preprint 1004v:880400000
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yun- shen Wei, Baohong Lx, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022. cosformer: Rethinking soft- max attention. arXiv prepriat 088.77.20...8711
Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. 2013. The acl anthology network corpus. Language Resources and Evaluation, 47(4):119-94


Reviews:
[{"review_id": "mvMgK9AJ45", "reviewer": "Reviewer_wC4i", "summary": "", "questions": "Question A: What will be the impact of the proposed approach on the overall parameter of the model? \nQuestion B: why is a non-linear function applied only to one of the pairs? why can't we apply to both pairs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your insightful comments. \n1.\tFor Question A: For the number of parameters for different models, the FFN has a parameter value of $8d^2$, whereas our MSCFFN has a parameter value of $2.25d^2$. MSCFFN has a smaller number of parameters and actual GPU memory usage compared to FFN, giving it an advantage in model size.\n2.\tFor Question B: The primary reason for non-linear functions only to one of the pairs is to introduce a controlled form of non-linearity while maintaining the ability to pass through information unaltered. If non-linear functions were applied to both parts, the information might undergo extensive transformations in both branches, leading to a loss of the original input's identity.\n3.\tWe apologize for the typos and grammatical errors. Thanks for bringing these issues to our attention in detail. We will carefully review the whole paper to ensure that the overall writing meets the high standards of the EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors factorized the huge matrix and proposed multi-space cross mechanisms to optimize the feed-forward block of the transformer. The experiment show improvement in the performance when compared with the baseline.", "reasons_to_reject": "None", "questions_to_authors": "Question A: What will be the impact of the proposed approach on the overall parameter of the model? \nQuestion B: why is a non-linear function applied only to one of the pairs? why can't we apply to both pairs?", "paper_topic_and_main_contributions": "The authors tried to optimize the feed-forward network by splitting the huge matrix into a smaller matrix and used the multi-space cross method to improve the performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "It would be great if the authors do one additional round of proof reading. There are a few minor things like Early exiting -> early exiting.\n"}, "extra_text": {}}, {"review_id": "S0dKHj7ZDc", "reviewer": "Reviewer_iZ74", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. \n1.\tWe acknowledge your raised concerns regarding the juxtaposition of our paper with the existing body of work. The constraints imposed by temporal limitations necessitated the evaluation of merely four foundational models. It is important to highlight that our methodology distinctly diverges from the paradigms proposed by MoEﬁcation and GroupBERT. Rest assured, we are committed to not only referencing the pertinent sources but also incorporating the comparative outcomes of our experiments within the definitive iteration of our manuscript.\n2.\tWe apologize for the typos and grammatical errors. Thanks for bringing these issues to our attention in detail. We will carefully review the whole paper to ensure that the overall writing meets the high standards of the EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed MSCFFN can save more than 60 percent of the computation resources without undercutting model performances.", "reasons_to_reject": "The paper fails to compare to related work. Even though the goal of the proposed model is to reduce computational cost of transformers, the method itself is a method to reduce FFN, and the paper fails to introduce related work and compare to them. Please see related work in \"Missing References\" section. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to reduce Transformer's computational cost by reducing the computational complexity of feed forward layers, specifically, it splits the large matrix space to several small spaces and uses the Multi-Space Cross method to prevent performance decline. The paper evaluates the accuracy and efficiency on the Long-Range Arena benchmark and shows the proposed MSCFFN can achieve better results with a faster speed. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There should be a space between \"(\" and its preceding word (line 041 and 042)."}, "extra_text": {"Missing_References": "1. Zhang, Zhengyan, et al. \"Moefication: Transformer feed-forward layers are mixtures of experts.\" arXiv preprint arXiv:2110.01786 (2021).\n2. Chelombiev, Ivan, et al. \"Groupbert: Enhanced transformer architecture with efficient grouped structures.\" arXiv preprint arXiv:2106.05822 (2021).\n\nPlease also check the related work section in the above papers."}}, {"review_id": "KS2xS29fMO", "reviewer": "Reviewer_i4EW", "summary": "", "questions": "In Table 2, is it number of training steps or testing steps per second? \nHow were those hyper-parameters m,n selected?  ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your insightful comments. \n1.\tIn Tabel 2, it is number of training steps. And we set m to 6 and n to 12 as mentioned in Section 3.3, line 234.\n2.\tFor the question of “contribution looks limite”. Our MSCFFN can replace the original FFN in Transformer models and speedup model training and inference no matter how long or short the input sequence is. And It is easily to combine with other speedup methods of linear attention to achieve faster Acceleration, such as flowformer、cosformer, which can reduce the quadratic complexity of self-attention layer related to sequence length to linear complexity.  Regrettably, as of now, there exists no method to transition the quadratic complexity of the FFN into a linear complexity structure while preserving performance integrity. Nevertheless, our forthcoming endeavors will be dedicated to meticulously exploring avenues for mitigating the quadratic complexity associated with the FFN, striving for optimal reductions without compromising efficacy. This commitment underscores the focus of our future research.\n3.\tFor the question of “cross method of subspace”. Firstly, Due to computational complexity, we pair subspaces, but not all possible pairs. We also experimented with all possible pairs, it exhibited superior results. But to balance the effect and computation, we finally choose to pair the subspaces. Secondly, we apply ReLU to one of them is to introduce a controlled form of non-linearity while maintaining the ability to pass through information unaltered. If non-linear functions were applied to both parts, the information might undergo extensive transformations in both branches, leading to a loss of the original input's identity. Thirdly, we did a comparative experiment without a multi-space crossover mechanism, and the average score would drop by 1. The multi-space intersection mechanism in our MSCFFN is indeed important.. \n4.\tFinally, thanks for your suggestions, we can add more experimental results in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed MSCFFN contains the cross method of subspace, which aims to keep the representation learning power while reducing the space dimension and number of parameters. Given that FFN is a widely used structure (as Transformers are), it can inspire the community to try it to reduce the computation complexity or further improve the method. ", "reasons_to_reject": "First, the contribution looks limited to me -- 60% of computations are saved theoretically, but the complexity is still in the same order -- quadratic. With the GPU acceleration, the practical speed up might be even less.\n\nSecond, the cross method of subspace is not intuitive to me. Why randomly pair the subspaces but not consider all possible pairs? Why on;y apply ReLU to one of them but not the other? Also, the author has increased the layers from 2 to 5, making it a deeper network. I feel some reasonable baselines might be needed to demonstrate the real power of MSCFFN. For example, what if we just use d/4 as the dimension but use 4 layers of them as the FFN (this also saves complexity)? \n\nThird, the key for this kind of substitution is to achieve the same performance. I think more benchmarks are needed in addition to LRA.", "questions_to_authors": "In Table 2, is it number of training steps or testing steps per second? \nHow were those hyper-parameters m,n selected?  ", "paper_topic_and_main_contributions": "This paper proposed a new structure called MSCFFN to replace the feed forward network (FFN) structure in Transformers. The MSCFFN has lower computation complexity and achieved similar or higher performance than FFN in the LRA benchmark (5 tasks).\n\nThe key idea is to split the d-dimension vector into n subspaces with each being d/n dimension, then apply elementwise product to pairs of subspaces to enlarge the representation power. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The presentation is overall great and clear."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to reduce the cost of MLPs found in Transformer architectures by decomposing the fully connected layers into smaller parallel components. Results demonstrate improved performance/faster runtime. Reviewers mostly agree on the importance of this research direction and find the proposed method promising. However they point out the lack of comparison with related work (sparse-MoEs) and ablations; which impacts the soundness of the claims made in the paper.
Decision: Accept-Findings