LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing
Year: 2023

Abstract:
Transformer has become an important technique for natural language processing tasks with great success. However, it usually requires huge storage space and computational cost, making it difficult to be deployed on resourceconstrained edge devices. To compress and accelerate Transformer, we propose LightFormer, which adopts a low-rank factorization initialized by SVD-based weight transfer and parameter sharing. The SVD-based weight transfer can effectively utilize the well-trained Transformer parameter knowledge to speed up the model convergence, and effectively alleviate the lowrank bottleneck problem combined with parameter sharing. We validate this method on machine translation, text summarization, and text classification tasks. Experiments show that on IWSLT’14 De-En and WMT’14 En-De, LightFormer achieves similar performance to the baseline Transformer with 3.8× and 1.8× fewer parameters, and achieves 2.3× speedup and 1.5× speedup respectively, generally outperforming recent light-weight Transformers.

heading: 1 Introduction
text: Transformer (Vaswani et al., 2017) has been widely used and achieved state-of-the-art results in natural language processing tasks such as machine translation, text summarization, and text classification. To improve the performance of networks, we usually stack deeper transformer blocks, or increase the dimension of the hidden layers (Mehta et al., 2020; Li et al., 2020). However, this will bring a huge amount of parameters and computations to the model, exceeding the capabilities of many edge devices. Common methods of model compression mainly include pruning (Han et al., 2015), quantization (Gong et al., 2014), knowledge distillation (Hinton et al., 2015), low-rank factorization (Sainath et al., 2013), and weight sharing (Lan et al., 2019).
∗Corresponding Author
Deploying Transformer networks on edge devices is challenging. Edge devices are limited in terms of computing power, storage resources, and so on. We generally think of model compression as having three metrics: compression ratio, model performance, and inference speed (FLOPs) (Thakker et al., 2020). Low-rank factorization can greatly reduce the parameters and computations of the model, and we believe it has the potential to be applicable to end-to-size scenarios.
Low-rank factorization (Kuchaiev and Ginsburg, 2017; Noach and Goldberg, 2020; Hsu et al., 2022) is a widely used model compression technique. Low-rank factorization can reduce the parameters and improve the inference speed of the network. This method can be applied to any linear layer, which decomposes the weight matrix of the linear layer into many small low-rank matrices. However, when compressing a model with a high compression ratio, the rank of the matrix factorization needs to be set very small, which will reduce the expression ability and affect the model performance (Thakker et al., 2020). This phenomenon is called the low-rank bottleneck problem.
To address this problem of low-rank matrix factorization, we first propose the SVD-based Weight Transfer to initialize the weights of low-rank matrix factorization. We perform singular value factorization (SVD) on the original trained Transformer weight matrices, which can obtain the optimal lowrank approximation of the weight matrix, and then the obtained small weight matrices are used as the initialization weights of low-rank factorization. Compared with randomly initialized low-rank factorization, SVD-based weight transfer for the initialization of low-rank factorization can effectively utilize the parameter knowledge of the well-trained Transformer, and accelerate the convergence of the model, which can improve the performance of lowrank matrix factorization to alleviate its low-rank bottleneck problem.
10323
Weight sharing is also a parameter-efficient model compression technology (Lan et al., 2019; Reid et al., 2021; Dabre and Fujita, 2019; Takase and Kiyono, 2021), which reduces the parameters by reusing the parameters of the model. To further solve the low-rank bottleneck problem and compress the model under the premise of ensuring model performance, we adopt a group-based cross-layer weight sharing mechanism based on the above method. It compresses the model by sharing the parameters of Transformer layers within a group. Achieving a certain compression ratio, parameter sharing can make the low-rank factorization take a larger rank, which can further lessen the low-rank bottleneck.
To sum up, we propose a lightweight Transformer based on SVD weight transfer and parameter sharing, called LightFormer. LightFormer can greatly reduce parameters under the premise of ensuring performance and speeding up the inference speed of the model to be more friendly to the end-to-end scenarios. We verified our method on machine translation, text summarization, and classification tasks. Experimental results show that our LightFormer outperforms recent light-weight transformers.
The main contributions of our work can be summarized as follows:
• We first propose a novel low-rank matrix factorization initialized by SVD-based weight transfer, which can effectively utilize the parameters of the trained network to accelerate the convergence of low-rank factorization networks compared with random initialization.
• We propose a lightweight Transformer (LightFormer) using low-rank factorization based on SVD initialization and group-based parameter sharing, which can significantly reduce the parameters on the premise of ensuring model performance and accelerate inference speed.
• On IWSLT’14 De-En, WMT14’En-De, and WMT’16 En-Ro datasets of translation, LightFormer achieves comparable BLEU scores to the baseline Transformer with 3.8×, 1.8×, and 3.1× fewer parameters respectively. On dataset GigaWord of text summarization, LightFormer achieves better performance than baseline Transformer using 2.9× fewer parameters.

heading: 2 Preliminaries

heading: 2.1 Low Rank Factorization
text: Low-rank matrix factorization (LMF) is a common and effective method to compress the deep neural networks. To compress the fully-connected layers (Figure 1), the weight matrix Wm×n is decomposed into the product of two smaller matrices, as shown in Figure 1:
Wm×n ≈ Um×r × Vr×n (1)
where r is the rank of low-rank factorization. The original matrix W requires mn parameters, and for the decomposed small matrices U and V , the parameters is r(m+ n).

heading: 2.2 SVD
text: For any matrix mathematical formula: Am×n, singular value factorization (SVD) is to decompose the matrix into the following form:
Am×n = Um×mΣm×nV Tn×n (2)
where each column of U and V is called the left and right singular vectors of A respectively. Σ is a non-negative definite real diagonal matrix.
We can choose the largest r singular values and the corresponding left and right singular vectors to approximate the original matrix:
Am×n ≈ Um×rΣr×rV Tr×n (3)
where r is the rank of truncated SVD.

heading: 3 Methodology
text: In this section, we first introduce the motivation of our method. Secondly, we introduce SVD-based weight transfer with parameter sharing. Finally, we introduce the framework of our LightFormer.

heading: 3.1 Motivation
text: Limitations of LMF based on Random Initialization. The process of compressing the network using low-rank factorization can be divided into two steps: initializing a low-rank factorization network structure, and then training this network from scratch. The advantage of this method is that the implementation process is simple. However, this approach has the following limitations:
(1) Hard to Converge. The low-rank factorization based on random initialization is difficult to converge. The experimental results show that the loss of the low-rank factorization with random initialization can not be reduced to that of the original Transformer. Furthermore, the low-rank factorization based on random initialization is hard to achieve the performance of the original Transformer.
(2) Low-rank Bottleneck. Low-rank matrix factorization (LMF) will encounter a low-rank bottleneck problem when achieving a high compression ratio, leading to a significant drop in the performance of the model (Thakker et al., 2020). In table 1, as the rank decreases, the parameters of the model are also reduced, but the performance of LMF Transformer also drops significantly.

heading: 3.2 SVD-based Weight Transfer and Weight Sharing
text: Many studies (Glorot and Bengio, 2010; Liu et al., 2020) have shown that the initialization of the network is very important for the optimization of the model. Lin et al. (2021) shows that knowledge in the trained network parameters are important.
SVD-based Weight Transfer. Based on the motivation above, we proposed a novel initialization strategy for low-rank factorization, named SVD-
based weight transfer, which aims to retain the parameter knowledge of the original network. Unlike the factorization method of learning from scratch (random initialization), SVD-based weight transfer use the weight of a fine-training network to initialize a low-rank factorization network via SVD.
Intuitively, the expressibility of the decomposed network is positively related to the top k% singular value of the well-trained weight matrices. For example, using SVD to compress a picture, the more singular values retained, the clearer the restored image will be. Therefore, SVD-based weight transfer can not only reduce parameters and operations but also retain the performance of the original network.
Group-based Weight Sharing. To further compress the parameters and alleviate the low-rank bottleneck problem, we combine the group-based cross-layer parameter sharing with the low-rank factorization. We can divide the L layers into N groups of size M , and each size-M group shares parameters.
Weight sharing alleviates the low-rank bottleneck. For m × n fully connected layers, and the total layers is L, the rank of the low-rank factorization is r, and the number of groups for parameter sharing is N , then the parameter numbers of the low-rank network can be calculated as follows:
P = L(m+ n)r (4)
After combined parameter sharing, the parameter numbers are given by:
P ′ = N(m+ n)r (5)
Under the same parameter settings, the rank of the low-rank factorization combined with parameter sharing is defined as r′:
r′ = L
N r (6)
where LN > 1, r ′ > r. Low-rank factorization with parameter sharing can make the rank set larger under the same scale parameters.

heading: 3.3 LightFormer
text: In this section, we propose a light-weight Transformer, named LightFormer, as shown in Figure 2, using SVD-based weight transfer and weight sharing. We first introduce the low-rank matrix factorization Transformer (LMF Transformer). Then we
initialize the LMF Transformer using the method based on SVD weight transfer. Finally, we combine SVD weight transfer with weight sharing to achieve greater parameter efficiency, getting our LightFormer.
Low-rank Layers. Based on low-rank matrix factorization, we propose low-rank layers to replace the original fully-connected layers in the Transformer for achieving model compression and acceleration, which can be defined as follows:
W = UV (7)
where W ∈ Rm×n, U ∈ Rm×r, and V ∈ Rr×n. LMF Transformer. Transformer includes word embeddings, self-attention, and feed-forward networks (FFN), which are composed of fullyconnected layers. LMF Transformer replaces fullyconnected layers of these sublayers with low-rank layers.
(1) LMF Embedding is defined as follows:
WE = WaWb (8)
where Wa ∈ Rd×r, Wb ∈ Rr×V , and r is the rank of the low rank matrix factorization.
(2) LMF Self-Attention is defined as follows:
A = XUQVQ(XUKVK)
T
√ dk
f(Q,K, V ) = softmax(A)XUV VV
(9)
where f is the LMF self-attention function, UQ ∈ Rd×r, VQ ∈ Rr×d, UK ∈ Rd×r, VK ∈
Rr×d, UV ∈ Rd×r, VV ∈ Rr×d. d is the dimension of hidden size, and r is the rank of low-rank factorization.
(3) LMF Feed-Forward Network (FFN) can be defined as follows:
g(X) = ReLU(XU1V1 + b1)U2V2 + b2 (10)
where U1 ∈ Rd×r, V1 ∈ Rr×dff , U2 ∈ Rdff×r, V2 ∈ Rr×d, b1 ∈ Rdff and b2 ∈ Rd. r is the rank of low-rank factorization and dff is the dimension of FFN.
SVD-based Weight Transfer. Different from random initialization, we use SVD-based weight transfer to initialize LMF Transformer. The implementation process of SVD-based weight transfer is as follows:
(1) Training baseline Transformer. Training the original Transformer as the network for singular value factorization (SVD).
(2) Setting the rank of SVD. We set the rank of SVD according to the proportion of singular values to be retained, the rank determines the model parameter compression rate.
(3) SVD-based Weight Transfer. As shown in Figure 2, we perform singular value factorization on parameter matrices of Embedding, SelfAttention, and FFN layers of the trained Transformer, which can be written as:
Wm×n ≈ Um×rΣr×rV Tr×n = Um×rV ′r×n (11)
where V ′r×n = Σr×rV T r×n.
We use the small weight matrices as the initial value of the corresponding layers of the LMF Transformer. The rank in SVD should be consistent with the rank of the matrix factorization in the previous step.
Combining with Group-based Weight Sharing. We use group-based parameter sharing for the LMF Transformer. We share the network parameters within a group of the model’s Encoder layers. Each group consists of several contiguous LMF Transformer layers. We share all parameters across layers, including feed-forward network (FFN) and selfattention, as shown in Figure 2.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: Machine translation: We conduct experiments on three machine translation datasets: IWSLT’14 DeEn, WMT’16 En-Ro and WMT’14 En-De, which have been widely used for machine translation. The IWSLT’14 De-En dataset consists of about 160K/7K/7K sentence pairs for training, validation, and testing respectively. It has a joint byte pair encoding (BPE) (Sennrich et al., 2016) vocabulary of about 10K tokens, which is the same setup as Liu et al. (2020). For WMT’14 En-De, we train the model on WMT’16 training data with 4.5M sentence pairs, validate on newstest2013, and test on newstest2014, the same as Wu et al.. The WMT’16 En-Ro dataset consists of 600K/2K/2K sentence pairs for training, validation, and testing respectively. It has a joint BPE vocabulary of about 35K tokens, which is the same setup as Mehta et al. (2020). For evaluation, we use beam search decoding in three tasks. For De-En and En-Ro, the beam size is 5. For En-De, the beam size is 4 and length penalty 0.6. The performance was measured by case-sensitive tokenized BLEU (Papineni et al., 2002) for all translation tasks. The evaluation setting is the same as Mehta et al. (2020).
Text summarization: We evaluate on the GigaWord dataset, which consists of a total of 3.8M article-title pairs in English. We take the article as the encoder input and title as the decoder input. We use the F1 score of ROUGE-1, ROUGE-2 and ROUGE-L as the evaluation metric1 on the GigaWord testset. We use beam search with a beam size of 5 for inference.
1https://github.com/pltrdy/files2rouge
Text classification: We validate our method on four text classification tasks. CR (Hu and Liu, 2004): Customer reviews composed of positive or negative product reviews; MR (Pang and Lee, 2004): Movie reviews divided into positive and negative categories; SUBJ: Subjectivity dataset where the target is to classify a text as being subjective or objective; MPQA (Wiebe et al., 2005): Opinion polarity detection subtask.

heading: 4.2 Architecture
text: Deep Encoder and Shallow Decoder for Sequence Modeling. The vanilla Transformer (Vaswani et al., 2017) adopts 6 encoder layers and 6 decoder layers. Besides the 6-6 setting, we choose a deep encoder shallow decoder setting that assigns 18 encoder layers and 3 decoder layers. Assigning more layers on encoders than decoders is beneficial for inference speed while maintaining its performance (Li et al., 2021; Kasai et al., 2021).
Transformer Encoder for Text Classification. For text classification, we use the Transformer encoders as the baseline. The number of encoder layers L = 6, and the dimension of model d = 512. And word embeddings are initialized by GloVe (Pennington et al., 2014).

heading: 4.3 Experimental Setup
text: Baselines and Implementations. We compare our method with Transformer (Vaswani et al., 2017) and recent light-weight Transformers including Lite Transformer (Wu et al.), Hardware-Aware Transformers (HAT) (Wang et al., 2020a), DelighT (Mehta et al., 2020), and Subformer (Reid et al., 2021). The implementation of all models use Faiseq Library (Ott et al., 2019). We reproduce the results of baselines following the setting from their papers or download the trained models from their official GitHub.2 3 4
Speed Measures. We do not use FLOPs as a speed metric because Wang et al. (2020a) found that FLOPs does not reflect the measured latency in autoregressive Transformer. The inference speed metric we used is tokens/s, which means the number of tokens translated per second. We sample 50 sentences of an average output length to test inference speed. We run these samples 10 times
2https://github.com/mit-han-lab/lite-transformer 3https://github.com/mit-han-lab/hardware-aware-
transformers 4https://github.com/sacmehta/delight
and remove 10 % of the fastest and slowest results, and average the rest 80 % results. We test the speed on 1 core Intel Xeon E5-2678 v3 @ 2.50GHz CPU. We evaluate the inference speed with the batch size of 1 to simulate the inference of edge devices.

heading: 4.4 Experimental Results
text: In Table 2 and 3, we first compare the results between our method with previous light-weight Transformers (Wang et al., 2020a; Wu et al.; Mehta et al.,
2020) in the setting of Transformer Base (Vaswani et al., 2017) on IWSLT’14 De-En, WMT16’ En-Ro, and WMT14’ En-De tasks. Under the similar or even better performance, LightFormer compresses Transformer 2.9 ∼ 4.4× parameters, and accelerates Transformer 1.5 ∼ 2.3× on Intel CPU, which generally outperforms recent light-weight Transformers in compression ratio, inference speed, and performance.
Table 4 and Table 5 show the experimental results of text classification and text summarization tasks. As shown in Table 4, LightFormer achieves comparable performance to the baseline Transformer, while the number of parameters is only
about 1/4 of the baseline model. On text classification datasets, LightFormer outperforms Transformer by 0.5 higher average accuracy, while the parameters are only 0.5M.
To further verify the effectiveness of our method, we compare our LightFormer with Transformer, LMF Transformer, and other lightweight Transformers on different model scales on IWSLT’14 De-En, WMT’16 En-Ro, and WMT’14 En-De. As shown in Table 3, LightFormer is consistently better than LMF Transformer and recent light-weight Transformers, which shows that our method can effectively alleviate the low-rank bottleneck problem.
Furthermore, compared to the LMF Transformer with randomly initialized weights, our method converges faster during training and has a even lower loss on De-En, En-Ro, and En-De train and valid datasets, as shown in Figure 4. This shows that using the parameter knowledge of the teacher model can indeed improve the convergence speed of the LMF Transformer model and improve the performance of the model, which also demonstrates the effectiveness of our proposed SVD-based weight transfer method.

heading: 4.5 Ablation Study
text: Ablation on methods. In Table 6, we show the ablation study of how the vanilla Transformer evolve into our LightFormer and evaluate the impact of
low-rank matrix factorization and weight sharing. Weight sharing (r3) can reduce model parameters while maintaining model performance compared to the baseline Transformer (r1,r2). Using lowrank matrix factorization (r4) can greatly reduce parameters while model performance drops a little. The performance of low-rank factorization with weight sharing (r5) is better than only matrix factorization (r4). Based on the low-rank factorization model with weight sharing, the SVD-based parameter transfer method (r6) can further improve the performance of the model and achieve a similar performance to the original Transformer.
Ablation on Deep to Shallow Setting. In previous experiments, we adopt 18 encoder layers and 3 decoder layers besides the 6-6 setting. As shown in Table 7, we assign different encoder and decoder layers settings to the experiment on IWSLT’14 De-En. According to the experimental results, we found that the 18-3 mode is a better trade-off between compression ratio and model performance. The deep and shallow setting has a speed advantage over the original 6-6 setting.
Ablation on Weight Shared Groups. For weight sharing, the number of independent layers (shared groups) is an important hyperparameter. In Table 8, we perform ablation experiments on IWSLT’14 De-En with different group numbers of weight sharing. According to the experimental results, we can see that 6 independent layers of LightFormer 18-3 is a good setting on De-En.

heading: 5 Related work
text: Generally, model compression methods mainly include low-rank factorization (Thakker et al., 2020;
Hsu et al., 2022), parameter sharing (Lan et al., 2019; Reid et al., 2021; Dehghani et al.), knowledge distillation (Sun et al., 2019; Wang et al., 2020b; Jiao et al., 2020), pruning (Cui et al., 2019; Hou et al., 2020), and quantization (Zafrir et al., 2019; Dettmers et al., 2022). In this paper, our focus is on low-rank factorization and wight sharing.
Low-rank factorization is a widely used technique in model compression (Kuchaiev and Ginsburg, 2017; Lan et al., 2019). The goal of low-rank matrix factorization is to decompose a large matrix into two small matrices of low rank. Therefore, the parameters and calculations of the model will be reduced. However, there is a low-rank bottleneck problem in low-rank matrix factorization.
Weight sharing. Dabre and Fujita (2019) shares the weights across all Transformer layers for machine translation with a small performance drop. Universal Transformer (Dehghani et al.) shares the weights across all layers, allowing for recurrent computation with a dynamic halting mechanism. ALBERT (Lan et al., 2019) uses weight sharing to reduce the parameters of BERT. Although weight sharing can greatly reduce the number of model parameters, it cannot improve the inference speed of the model.

heading: 6 Conclusion
text: In this paper, we propose a lightweight Transformer (LightFormer) based on SVD weight Transfer and parameter sharing, which can guarantee the performance of the model but with fewer operations and parameters. Compared with the randomly initialized low-rank matrix factorization, our method can effectively alleviate the low-rank bottleneck problem and speed up the convergence of the model. We validate our method on three machine translation tasks. Experimental results show that our method is consistently better than recent light-weight Transformers.
Limitations
Our experiments are mainly on traditional datasets and do not fully demonstrate the effectiveness of the method in end-to-end scenarios. In order to solve the low-rank bottleneck problem, this paper proposes a method of SVD weight transfer, but this method is limited to matrix factorization and does not apply this method to more general lowrank factorization, such as tensor factorization. We leave it as future work.

heading: Acknowledgements
text: This work is supported in part by the Natural Science Foundation of China (grant No.62276188 and No.61876129), TJU-Wenge joint laboratory funding, and MindSpore.

heading: A Experiment on Raspberry Pi
text: In this section, we conducted our experiments on the end-side device (Raspberry Pi) to verify the effectiveness of our method. We compare LightFormer with Transformer on three machine translation datasets.
As shown in Table 9, our method has achieved significant compression performance on three datasets. Achieving similar BLEU scores on three tasks, LightFormer can compress the Transformer parameters 2.8 ∼ 4.8×, and speedup it 1.4 ∼ 2.5× on Raspberry Pi ARM CPU. Therefore, our proposed LightFormer is an end-to-side friendly lightweight Transformer model.

heading: B Low-rank factorization Based on SVD
text: Parameter Initialization
The previous low-rank factorization methods applied to model compression can be mainly divided into two categories, one is random initialization of parameters, and then optimizes through downstream tasks to update parameters; the other is to decompose and compress trained network parameters using SVD (Hsu et al., 2022).
Different from the above methods, the method proposed in this paper can effectively utilize the trained parameter knowledge. We first train a network model with a deep architecture as the baseline model and then perform SVD on the trained model to obtain a low-rank matrix as the initial value of the low-rank network parameters. Finally, the lowrank factorization network is trained to converge in a downstream task. SVD parameter transfer is essentially an initialization method for low-rank factorization.

heading: C Experiment Details
text: Setting of SVD rank: The rank in the singular value factorization determines the ratio of retaining the original network weight information. The larger the rank, the more original weight information is retained, and the model performance is better.
Suppose the dimensions of the input and output of the fully connected layer are m and n respectively, we reserve the largest k singular values for matrix compression, where k < m,n, and the parameter compression ratio is p, then the rank k of SVD should satisfy the following formula:
p = mn
(m+ n)k
k =
⌊ mn
p(m+ n)
⌋ ; k ∈ Z+
(12)
Therefore, the selection of rank depends mainly on the parameter compression rate. In this paper, we set the rank based on the parameter compression rate. Of course, the rank selection is also a valuable research work, and our future work may involve this aspect.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The last section.
7 A2. Did you discuss any potential risks of your work? The work of this paper is mainly to compress the Transformers model, which does not involve potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The code in this article is implemented based on an open source library, and the dataset is also open source

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset processing steps used in this paper are consistent with the baseline model, and are mainly processed according to the data preprocessing of the baseline model.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The experimental results in this paper are obtained by taking the average value of multiple experiments.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Because we follow the default hyperparameters of the baseline model
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? The paper does not involve human annotators.
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? The dataset used in this paper is open source
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? The dataset used in this paper is open source
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The dataset used in this paper is open source
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? The dataset used in this paper is open source
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? The dataset used in this paper is open source

heading: Future Work
text: Of course, the rank selection is also a valuable research work, and our future work may involve this aspect. ACL 2023 Responsible NLP Checklist