TranSFormer: Slow-Fast Transformer for Machine Translation
Year: 2023

Abstract:
Learning multiscale Transformer models has been evidenced as a viable approach to augmenting machine translation systems. Prior research has primarily focused on treating subwords as basic units in developing such systems. However, the incorporation of finegrained character-level features into multiscale Transformer has not yet been explored. In this work, we present a Slow-Fast two-stream learning model, referred to as TranSFormer, which utilizes a “slow” branch to deal with subword sequences and a “fast” branch to deal with longer character sequences. This model is efficient since the fast branch is very lightweight by reducing the model width, and yet provides useful fine-grained features for the slow branch. Our TranSFormer shows consistent BLEU improvements (larger than 1 BLEU point) on several machine translation benchmarks.

heading: 1 Introduction
text: Transformer (Vaswani et al., 2017) has demonstrated strong performance across a range of natural language processing (NLP) tasks. Recently, learning multiscale Transformer models has been evidenced as a promising approach to improving standard Transformer. Previous research on this line can be broadly categorized into two streams: one learns local fine-grained features by using a fixed-length window (Yang et al., 2019; Hao et al., 2019; Guo et al., 2020), linguistic-inspired local patterns (Li et al., 2022b), and a hybrid approach that combines convolution and self-attention models (Gulati et al., 2020) or run in parallel (Zhao et al., 2019); the other learns sequence representations by considering multiple subword segmentation/merging schemas (Wu et al., 2020).
Despite the attractiveness of these approaches, previous work is based on an assumption that sub-
∗The work was done when the first author was an intern at Microsoft Research Asia.
†Corresponding author.
words are the basic units in sequence modeling, and therefore ignores smaller, more fine-grained character-level features. In fact, the benefits of using characters have long been appreciated, and character-based models have been discussed in several sub-fields of NLP, such as language modeling (Xue et al., 2022) and machine translation (Lee et al., 2017; Li et al., 2021; Gao et al., 2020). But there are still important problems one needs to address in multi-scale Transformer. The first of these is the computational challenge of dealing with long sequences. For example, when we represent an English text as a character sequence, the length of this sequence is in general 5× longer than that of the subword sequence. We therefore need to consider this length difference in model design. The second problem is that, from a multiscale learning perspective, learning text representations with features at different levels is not just making use of the syntactic hierarchy of language. To better model the problem, we need some mechanism to describe the interactions among these different linguistic units.
In this study, we aim to exploit the potential of character-level representations in multiscale sequence models while maintaining computational
6883
efficiency. Drawing inspiration from the SlowFast convolutional models in video classification (Feichtenhofer et al., 2019), we propose the SlowFast Transformer (TranSFormer) model, which utilizes a fast, thin branch to learn fine-grained character-level features and a slow, wide branch to capture correlations among subword features. A cross-granularity attention layer is placed between the self-attention and feedforward sublayers to make exchanges of cross-granularity information. This enables the slow branch to be aware of fine-grained features while providing optimized high-level representations of the input sequence to the fast branch.
We also make use of character-to-word boundary information to model the interactions among neighboring characters in a word. Additionally, we develop a boundary-wise positional encoding method to better encode the positional information within words for the fast branch. Through a series of extensive experiments on the WMT’14 English-German, WMT’17 Chinese-English and WMT’16 EnglishRomanian tasks, we demonstrate that TranSFormer yields consistent performance gains while having a negligible increase in the number of parameters and computational cost. As a bonus, our TranSFormer is robust to errors caused by suboptimal tokenization or subword segmentation.

heading: 2 Related Work
text: Multiscale Transformer Learning multiscale Transformer is a promising way to acquire for further improvements in the machine translation task. A feasible way is to model global and local patterns to enhance Transformer models (Shaw et al., 2018; Yang et al., 2018, 2019; Zhao et al., 2019). These work mainly modeled the localness within a fixed window size upon subword input features. Apart from these, Wu et al. (2018) partitioned the input sequence according to phrase-level prior knowledge, and build attention mechanism upon phrases. Similarly, Hao et al. (2019) proposed a multi-granularity self-attention mechanism, designed to allocate different attention heads to phrases of varying hierarchical structures. Perhaps the most related work to ours is UMST (Li et al., 2022b). They re-defined the sub-word, word and phrase scales specific to sequence generation, and modeling the correlations among scales. However, more fine-grained character-level scale is not explored in the previous work due to the serve challenge for encoding long
character sequences.
Character-level NMT Fully character-level neural machine translation originates from recurrent machine translation system in Lee et al. (2017). They built a fully character-level encoder-decoder model, and utilize convolution layers to integrate information among nearby characters. Cherry et al. (2018) show the potential of character-level models which can outperform subword-level models under fully optimization. This contributes to their greater flexibility in processing and segmenting the input and output sequences, though modeling such long sequences is time-consuming. More recently, several studies analyze the benefits of character-level systems in multilingual translation scenarios (Gao et al., 2020), low-resource translation and translating to typologically diverse languages (Li et al., 2021). But these methods all simply view characters as basic units in language hierarchy, and it is still rare to see the effective use of multi-scale learning on character-based language features.
Multi-Branch Transformer The utilization of multi-branch architectures has been extensively studied in Transformer models. Early efforts in this area include the Weighted Transformer (Ahmed et al., 2017), which replaced the vanilla self-attention by multiple self-attention branches. Subsequently, the Multi-attentive Transformer (Fan et al., 2020) and Multi-Unit Transformer (Yan et al., 2020) have advanced this design schema by incorporating branch-dropout and switching noise inputs, respectively. Additionally, Wu et al. (2020) investigated the potential advantages of utilizing dual cross-attention mechanisms to simultaneously attend to both Sentencepiece (Kudo and Richardson, 2018) and subword (Sennrich et al., 2016). In this work, we take a forward step to exploit the potential of character features. We argue that a lightweight branch is sufficient to encode useful fine-grained features, an aspect that has not been previously investigated.

heading: 3 Method
text: The proposed TranSFormer follows a encoderdecoder paradigm (see Figure 1) which involves two encoder branches operating at different input granularities. The original subword encoder, which has a large model capacity for fully learning correlations among input individuals, is defined as the slow branch. The other branch, designed to handle
character-level representations using a thin encoder to efficiently capture correlations among characters, is referred to as the fast branch. Our goal is to use the fast branch to learn a fine-grained but less precise representation to complement the slow branch. In the following sections, we will elaborate the core design of Slow branch, Fast branch and the cross-granularity attention, respectively.

heading: 3.1 The Slow Branch for Subwords
text: We use the standard Transformer as the slow branch due to its strong ability to model global interactions among input sequences. The input of the slow branch is the mixture of subwords and words since some high-frequency words have not been further divided into subwords. Following the suggestions in Li et al. (2022b), we adopt a graph convolutional network to model the inner correlations among words through the adjacency matrix As. To this end, the Slow branch then encodes the enhanced representation via the selfattention mechanism, SAN = Softmax(Q·K
T √ dk ) ·V , where Q, K, V are obtained through three independent projection matrix, such as Wq, Wk, Wv. A point-wise feed-forward network is followed, FFN = max(xW1+ b1, 0)W2+ b2, where W1 and W2 are transformation matrices and b1 and b2 are bias matrices. To bridge the gap between two granularities, we sandwich a new sublayer between the self-attention and the feed-forward network, to accomplish the feature interaction between the slow and fast branches. A straightforward idea is to employ a cross-attention similar with encoder-decoder
attention in the decoder side. We will discuss more details in the Section 3.3.

heading: 3.2 The Fast Branch for Characters
text: To enhance the efficiency of modeling long character-level inputs, we propose the use of a fast branch with a tiny hidden size. The hidden size is a critical factor in the computation of the selfattention network (Vaswani et al., 2017), and by reducing it, we can achieve faster computation. To the best of our knowledge, this is the first attempt to design multiscale Transformer models that considers character-level features, as the long input sequence has previously hindered such exploration.
While the fast branch may not be as powerful as the slow branch, it is still effective in learning fine-grained features. Our initial experiments have yielded two notable findings: 1) a slow branch with hidden size of 32 is sufficient for transferring fine-grained knowledge to the slow branch, and 2) cross-granularity fusion is crucial for the slow branch, while removing the reversed fusion in the fast branch has only a moderate effect on performance. We would ablate this settings in the Section 4.2. To further improve the modeling ability, we introduce several techniques as follows:
Char Boundary Information The use of wordboundary information has been shown to effectively reduce the redundant correlations among subwords, as demonstrated in (Li et al., 2022b). This leads to the consideration of character-level modeling, which poses a more challenging problem
due to the greater number of characters typically present within a word in comparison to subwords. The statistical analysis in Figure 2c further evidences it that a significant proportion of words contain more than 5 characters, while a much smaller number are divided into subwords. Thus, model may be unable to discern the distinction between the same character that belongs to the same word and that of distinct words.
To address this issue, we propose the use of a character-level graph convolution network (GCN) to learn local, fine-grained features while also allowing each character to be aware of its proximity to other characters. GCN(Kipf and Welling, 2017) is a suitable choice for this task as it aggregates feature information from the neighbors of each node to encapsulate the hidden representation of that node. The computation can be described as:
GCNFast = σ(D̃ − 1 2 f Ãf D̃ − 1 2 f · xW g f ), (1)
Ãf = Af + IL denotes the adjacency matrix of the undirected graph with self-connections. Here IL denotes the identity matrix. D̃f is the degree matrix of the adjacency matrix Ãf . W gf is a linear transformation which is a trainable parameter. The character-level encoder architecture is illustrated in Figure 3.
Boundary-wised Positional Encoding To further enhance the relative positional representation among characters, we design a boundary-wised positional encoding (PE) method. Our intuition is to provide each character with the ability to recognize which characters belong to the same word. Thus we restrict the relative window within each word, as illustrated in Figure 4. Vanilla relative positional encoding (Shaw et al., 2018) models the correlations in a fixed window size 2k + 1. Here we set k = 3, positions exceed k would be masked. Differently, the proposed boundary-wised PE is utilized to enhance the inner relative positional information among characters within each word. In our preliminary experiments, boundary-wised PE is helpful for stable training.

heading: 3.3 Cross-Granularity Fusion
text: As depicted in Figure 3, the computation of the two branches in our TranSFormer architecture is separated, with each branch operating independently of the other’s representation. To facilitate communication between the branches, we propose the utilization of a cross-granularity information fusion method within each encoder block. This method can be implemented through various options. Given the lengths of the slow and fast branches as Ls and Lf , and the hidden sizes as Hs and Hf , respectively, the goal is to seamlessly
integrate cross-scale information within each encoder block between the two branches. In the field of machine translation, it is straightforward to employ cross-attention mechanisms, such as encoderdecoder attention (Vaswani et al., 2017) or contextaware cross-attention (Voita et al., 2018), to capture correlations between the representations.
Our default strategy is to employ a crossgranularity attention mechanism (namely CGA) sandwiched between the self-attention and feedforward network. The architecture is plotted in Figure 3. xf and xs denote the representation of the fast and slow branches, respectively. The challenge remains here is the mismatched feature shape between xs and xf Here, we take the Fast branch as an instance, we first normalize xs via x̂f = LN(xs). LN(·) denotes the layer normalization for stable optimization. Then x̂f is fed into CGA of the fast branch, the formulation is as follows:
ATTNf = Softmax( xfW q f · (x̂fW kf )T√
dkf
),
CGA = ATTNf · x̂fW vf , (2)
where the query is derived from the residual output of SAN in the Fast branch via xs ·W qf . The key and value are derived from the Slow branch via x̂fW kf and x̂fW vf , respectively. It is worthy to note that, the shape of W kf and W v f ∈ RHs×Hf , to reduce the hidden size. Detailed transformation could be found in the left part of Figure 3.
It is important to note that our proposed method of cross-granularity fusion is bidirectional, as opposed to the lateral connections used in the SlowFast (Feichtenhofer et al., 2019). Other alternative methods would be discussed in Section 4.3.

heading: 3.4 Interactions Between Encoder and Decoder
text: In vanilla Transformer, the key and value of the encoder-decoder attention on the decoder side derives from the encoder output, however, there are two branches in our TranSFormer (See Figure 1). It is worthy to investigate how to effectively leverage the multi-granularity representations. Our default strategy is to regard the fast branch as an auxiliary to provide fine-grained features for the slow branch, thus only the output of the slow branch is exposed to the decoder. Besides this, there are also several feasible options. For example, we can fuse the outputs of two branches as the final encoder output, or building a double-branch encoder-decoder attention to attend two branches independently. We compares this options in our experiments.

heading: 4 Experiments

heading: 4.1 Experimental Setups
text: Datasets The present study examines the performance of our proposed TranSFormer on several machine translation datasets: the WMT’14 EnglishGerman (En-De), WMT’16 English-Romanian (En-Ro) and WMT’17 Chinese-English (Zh-En) datasets. The En-De dataset comprises approximately 4.5 million tokenized sentence pairs, which were preprocessed following the same procedure as in Ott et al. (2018) to yield a high-quality bilingual training dataset. For validation, we use the newstest2016 set, while the newstest2014 set served as the test data. The En-Ro dataset consists of 610K bilingual sentence pairs, and we adopt the same preprocessing scripts as in Lee et al. (2018); Kasai et al. (2020), using a joint source and target BPE factorization with a vocabulary size of 40K. The newsdev2016 set is used for validation, while the newstest2016 set served as the test set. For the ZhEn task, we collect all the available parallel data for the WMT17 Chinese-English translation task, consisting 15.8M sentence pairs from the UN Parallel Corpus, 9M sentence pairs from the CWMT Corpus and about 332K sentence pairs from the News Commentary corpus. After carefully data filtering setups in Hassan et al. (2018), there are left 18M bilingual pairs. newsdev2017 and newstest2017 are served as the validation and test sets, respectively.
Setups For the machine translation task, we mainly evaluate the proposed TranSFormer on base and big configurations. The hidden size of slow
branch is 512/1024 for base and big, respectively. And the filter size in FFN is 2048/4096. In our default setting, a width of 32 slow branch is enough to learn fine-grained features, and the corresponding filter size is set to 128. We both employ residual dropout, attention dropout and activation dropout. All values are 0.1, except the residual dropout 0.3 for big counterparts.
Training and Evaluations The codebase is developed upon Fairseq (Ott et al., 2019). All experiments are conducted on 8 Tesla V100 GPUs. We use Adam (Kingma and Ba, 2015) optimizer with (0.9, 0.997), and the default learning rate schedule with 0.002 max value, 16, 000 warmup steps. For machine translation tasks, BLEU scores are computed by mult-bleu.perl, and we also provide the SacreBLEU1 for En-De. The beam size is 4 for En-De and 8 for Zh-en, and the length penalty is 0.6 and 1.3, respectively.
1BLEU+case.mixed+numrefs.1+smooth.exp+ tok.13a+version.1.2.12
Results of En-De The results of the WMT EnDe task under both base and big configurations are summarized in Table 1. As evidenced by the results, our TranSFormer model demonstrates significant improvements in BLEU when compared to the Slow only model, with gains of up to 1.16/1.05 BLEU scores under the base/big configurations. Conversely, the Fast only baseline, which has a hidden size of 32, only attains a BLEU score of 17.90, leading to a considerable performance gap due to its limited capacity. However, it still contributes up to a 1.14 BLEU-point benefit to the Slow branch, indicating that the fine-grained correlations modeled by the Fast branch are complementary. Additionally, we present the results of prior works, which have employed both character-level and subword-level systems, and categorize them in terms of various aspects. TranSFormer can beat or on par with prior works with less parameters. This indicates the investigation of character-level mutliscale models is meaningful. Note that TranSFormer is computationally efficient, only requiring additional 15% training cost and negligible inference latency. And TranSFormer can also benefit from
advanced design, e.g., another 0.74 improvement with ODE method (Li et al., 2022a).

heading: 4.2 Results
text: Results of Zh-En The WMT’17 Zh-En task poses a significant challenge due to the linguistic differences between Chinese and English. Additionally, the Chinese language owns less characters per word than English. Table 2 shows the results of our comparison of the TranSFormer with prior works. We observe TranSFormer yields a 1 BLEU point improvements than the subword-level systems. Our TranSFormer model demonstrates superior performance compared to previous work that models local patterns, while maintaining efficient computational requirements. We will exploit whether TranSFormer can gain more benefits when incorporating these techniques on the slow branch.
Results of En-Ro Furthermore, our empirical evaluations of the proposed TranSFormer architecture on the smaller WMT En-Ro dataset also demonstrate consistent improvements in BLEU scores as a result of the utilization of interactions among granularities. Notably, the TranSFormer model even outperforms the ODE Transformer (Li et al., 2022a), an advanced variant that leverages the advantages of high-order ordinary differential equations (ODE) solutions, by a substantial margin while incurring less computational cost.

heading: 4.3 Analysis
text: This section provides ablation studies of TranSFormer in terms of several core techniques.
Effect of width on Fast branch We first aim to explore TranSFormer under various widths of the Fast branch, including 16, 32, 64, 128 and 256. Results in Table 4a show that even a hidden size of 16 can provide helpful fine-grained features for the slow branch, and yielding almost 1 BLEU-point gains by bringing modest parameters. Empirically, a hidden of 32 and 64 deliver the best performance on base (En-De) and big (Zh-En) configurations, respectively. Further increasing the hidden layer dimension of the model results in no more gains, while requiring more computational cost.
Fusion methods between branches In addition to our proposed fusion method CGA, there are several alternative techniques that can be considered. The most straightfoward one is to transform the hidden with a linear projection and then use a standard cross-attention. It delivers similar performance but consumes more parameters. Another option is to downsample the character-level representation from Lf to Ls, and then concatenate (namely DS + Concat) or sum (DS + Sum) the two representations. Although both of these methods have been found to outperform the Slow only baseline, they have not been found to be on par with CGA method. This may be due to the fact that downsampling may impede optimization due to the low compression ratio of text compared with images.
Various interaction methods In Table 4c, we present a summary of various promising options for interactions between the encoder and decoder. Empirical results indicate that utilizing the Slow
branch as the output yields the highest performance. This can be attributed to the Fast branch’s ability to provide fine-grained features and low-level semantic knowledge as auxiliary information to the Slow branch. Additionally, while utilizing the Fast branch as the encoder output results in inferior performance compared to the baseline, it still yields a significant improvement over the Slow only baseline (17.90). This highlights the effectiveness of the TranSFormer model in leveraging interactions between different granularities. Furthermore, we also evaluated a two-stream approach in the decoder, in which one stream attends to the Slow branch and the other attends to the Fast branch, with a gated mechanism being used to fuse the features. However, this method was not sufficient to further improve performance. We attribute this to the negative interactions brought by the Fast branch, increasing the optimization difficulty.
Effect of various input granularities To ascertain whether the observed performance gains can be attributed to the complementary information provided by fine-grained character-level representations, we replaced the input of the fast branch with subword-level sequences, identical to that of the slow branch. The results presented in Table 4d demonstrate a degradation of up to 1 BLEU point. This can be attributed to the lack of distinct or complementary features provided by the fast branch and the limited capacity of the model in fully optimizing subword-level features. This observation further supports the hypothesis that the Slow-Fast design can learn complementary features for each granularity. Furthermore, we found that the TranS-
Former architecture with sentencepiece (Kudo and Richardson, 2018) as the fast branch input can also benefit from the two-branch design, due to the different segmentations. Additionally, our TranSFormer is a general design that can work well with a character-level fast branch and a sentencepiecelevel slow branch, yielding a BLEU score of 28.60, even slightly better than the subword-level one.
Ablations on fast branch designs It is hard to directly learn the tedious character sequence. The proposed character-boundary injection serves as a crucial component in addressing this challenge. Without this injection, the TranSFormer model suffers from a significant decrease in BLEU (#3). Furthermore, the situation is exacerbated when the boundary is replaced with a randomly initialized one (#4), emphasizing the importance of the proposed character-boundary injection. Also, both removing the boundary-wised positional encoding (#5) or replacing the vanilla attention by linear attention (Wang et al., 2020) (#6) lead to modest BLEU degradation. While, there is no significant impact when using unidirectional CGA (# 2, from Fast to Slow).
Ablations on interactions in different levels Our default configuration permits the model to allocate interactions at each encoder layer. It is beneficial to determine how interaction frequency impacts the performance. Table 6 compares various interaction frequencies at different levels, including exclusively at the final encoder block, the bottom three blocks, the top three blocks and every two blocks. The experiments were conducted on
WMT En-De. It is evident that the default configuration delivers optimal performance. Interactions conducted at the top three blocks demonstrate superior results compared to those at the bottom three blocks. Furthermore, performing fusion solely at the last encoder block proves insufficient for the model to learn multiscale interactions effectively.
BLEU v.s. Depth and BPE mergings Figure 5 plots the performance against model depths and BPE merging operations. The proposed TranSFormer architecture demonstrates consistent performance improvements as a result of increased encoder depth. Furthermore, an empirical evaluation of the TranSFormer against various byte-pair encoding (BPE) operations (Sennrich et al., 2016), on the slow branch of the model yields a statistically significant average gain of 1.1 BLEU scores over the Slow only baseline.
Low resource setting and morphological evaluation Li et al. (2021) has shown that characterlevel systems are better at handling morphological phenomena and show strong performance in low-resource scenarios than subword-level systems. Consequently, we evaluate how TranSFormer behaves at these scenarios. For the low-resource setting, we randomly select subsets of 50K, 500K, and 1000K from the WMT En-De training corpus. TranSFormer achieves respective BLEU scores of 11.87, 22.75, and 25.30, while the character-only and subword-only Transformers yield approximate scores of 10.50/7.00, 20.50/22.00, and 22.50/23.50. This empirical evidence demonstrates that TranSFormer effectively amalgamates the benefits of
both character-level and subword-level features. Moreover, Figure 6 plots the performance on MorphEval(Burlot and Yvon, 2017) benchmark. TranSFormer behaves better than subword solely in terms of Negation, Past, P-Gender and P-Number metrics.
Comparisons in Efficiency Table 8 compares the FLOPs between baseline and our TranSFormer both in base and big configurations. Due to the light computation cost of the fast branch, TranSFormer only brings additional 0.3G/1.1G FLOPS in base/big configurations, respectively. Note that the bulk of the additional computational cost is associated with the upsampling/downsampling operations within the cross-granularity attention mechanism. This process aligns the hidden size between the two representations.

heading: 5 Conclusions
text: In this work, we comprehensively leverage the potential of character-level features in multiscale sequence models while preserving high computational efficiency. To accomplish this, we propose a Slow-Fast Transformer architecture consisting of two branches in the encoder. The slow branch, akin to the vanilla Transformer, handles subwordlevel features, while the fast branch captures finegrained correlations among characters. By leveraging the complementary features provided by the fast branch, our TranSFormer demonstrates consistent improvements in BLEU scores on three widelyused machine translation benchmarks. Further indepth analyses demonstrate the effectiveness of the TranSFormer and its potential as a universal multiscale learning framework.

heading: Acknowledgments
text: This work was supported in part by the National Science Foundation of China (No. 62276056), the National Key R&D Program of China, the China HTRD Center Project (No. 2020AAA0107904), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Yunnan Provincial Major Science and Technology Special Plan Projects (No. 202103AA080015), the Fundamental Research Funds for the Central Universities (Nos. N2216016, N2216001, and N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No. B16009).
Limitations
The proposed TranSFormer architecture employs a two-branch design, which separately encodes character-level and subword-level features. Our original design of the proposed cross-granularity attention is to acknowledge the correlation between subwords and characters that belong to the same word. For example, a cross-granularity Gaussian distribution to let subwords pay more attention to the corresponding characters. However, the variability of word boundary information across sentences presents a challenge in effectively batching them and achieving high computational efficiency. This is an area of ongoing research, and will be the focus of future work. On the other hand, our current evaluation of the TranSFormer architecture is limited to machine translation tasks. It is worth exploring the potential of TranSFormer in optimizing character sequences on natural language understanding tasks and other sequence generation tasks, such as abstractive summarization. These tasks are more challenging in terms of encoding longer sequences, but we believe that TranSFormer can serve as a versatile backbone. We aim to verify its effectiveness on these tasks in the future.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitation
3 A2. Did you discuss any potential risks of your work? 3

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract
7 A4. Have you used AI writing assistants when working on this paper? 7
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: This is an area of ongoing research, and will be the focus of future work. On the other hand, our current evaluation of the TranSFormer architecture is limited to machine translation tasks. It is worth exploring the potential of TranSFormer in optimizing character sequences on natural language understanding tasks and other sequence generation tasks, such as abstractive summarization. These tasks are more challenging in terms of encoding longer sequences, but we believe that TranSFormer can serve as a versatile backbone. We aim to verify its effectiveness on these tasks in the future