Enhancing Neural Machine Translation with Semantic Units
Authors: Langlin Huang
Shuhao Gu
Zhang Zhuocheng
Yang Feng
Venue: EMNLP 2023 Findings (2023)
Keywords: machine translation
TL;DR: We propose to mix the embeddings of semantically unified tokens to construct a semantic-level sentence representation and leverage it to enhance Machine Translation. We also propose a fast but effective phrase extraction tool.

Abstract: Conventional neural machine translation (NMT) models typically use subwords and words as the basic units for model input and comprehension. However, complete words and phrases composed of several tokens are often the fundamental units for expressing semantics, referred to as semantic units. To address this issue, we propose a method Semantic Units for Machine Translation (SU4MT) which models the integral meanings of semantic units within a sentence, and then leverages them to provide a new perspective for understanding the sentence. Specifically, we first propose Word Pair Encoding (WPE), a phrase extraction method to help identify the boundaries of semantic units. Next, we design an Attentive Semantic Fusion (ASF) layer to integrate the semantics of multiple subwords into a single vector: the semantic unit representation. Lastly, the semantic-unit-level sentence representation is concatenated to the token-level one, and they are combined as the input of encoder. Experimental results demonstrate that our method effectively models and leverages semantic-unit-level information and outperforms the strong baselines.

Enhancing Neural Machine Translation with Semantic Units
Langlin Huang, Shuhao Gu, Zhuocheng Zhang, Yang Feng"
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences ac ac.icttt.c.....
(huanglanglln221,, zhangzhuocheng26z, Fengyang)@ict.cn
Abstract
24 A
Source
Provido cover for doctors on the pic @ ket line-
Conventional neural machine translation (NMT) models typically use subwords and words the basic units for model input and comprehension. However, complete words and phrases composed of several tokens are often the fundamental units for expressing semantics, referred to as semantic units. To address this issue, we propose a method Semantic Units for Machine Translation (SU4MT) which models the integral meanings semantic units within a sentence, and then leverages them to provide a new perspective for understanding the sentence. Specifically, we first propose Word Pair Encoding (WPE), a phrase extraction method to help identify the boundaries of semantic units. Next, we design an Attentive Semantic Fusion (ASF) layer integrate the semantics of multiple subwords into a single vector: the semantic unit representation. Lastly, the semantic-unit-levll sentence representation is concatenated to the token-level one, and they are combined the input of encoder. Experimental results demonstrate that our method effectively models and leverages semantic-unit-leve information and outper- forms the strong baselines. The code is available at https://gituub.co//ctttlt//////
40.68
*
24 # Hh (picket line)
-E f/ 18 (on) (doctons) (provide) (cover)
+1
* S
Target
(for)
4
4b-k (pickett
#Ky 49 lime)
IR A
4E
48-1"
Transformer
SU4MT
(picket line) (on)
Figure 1: An example of how integrating semantic units helps reduce translation errors. Instead of translating each word separately. SU4MT learns a unified meaning of picket line" to translate
whole.
semantic units in natural language. For words com- posed of multiple subwords, their complete mean- ing expressed when they are combined. For some phrases, their overall meanings differ from those of any individual constituent word. Consequently, other tokens attend to each part of semantic unit rather than its integral representation, which hin- ders the models from effectively understanding the whole sentence.
Recent studies have been exploring effective ways to leverage phrase information. Some works focus on finding phrase alignments between source and target sentences (Lample et al., 2018; Huang et al., 2018). Others focus utilizing the source sentence phrase representations (Xu et al., 2020; Hao et al., 2019; Li al., 2022a, 2023). However, these approaches often rely on time-consuming parsing tools to extract phrases. For the learning of phrase representations, averaging token represen- tations is commonly used (Fang and Feng, 2022: Ma et al., 2022). While this method is simple, it fails to effectively capture the overall semanties of the phrases, thereby impacting the model perfor- mance. Some researchers proposed effectively learn phrase representations (Yamada al., 2020: Li al., 2022b) using BERT (Devlin et al., 2019). but this introduces a large number of additional parameters.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015b; Gehring et al., 2017: Vaswani al., 2017; Zhang et al., 2023) has achieved significant success and has been continu- ously attracting significant attention. Currently, the mainstream language models use tokens as mean- ingful basic units, which are typically words or sub- words, and in a few cases, characters (Xue et al., 2022). However, these tokens are not necessarily
Corresponding author.
'This paper is done when Shuhao Gu studied in Institute of Computing Technology, and now he is working at Beijing Academy of Artificial Intelligence.

---

Therefore, we propose method for extract- ing semantic units, learning the representations of them, and incorporating semantic-level representa- tions into Neural Machine Translation to enhance translation quality. In this paper, semantic unit means a contiguous sequence of tokens that collec- tively form a unified meaning. In other words, semantic unit can be combination of subwords, words, or both of them.
of word embedding layer and stack of 6 en- coder/decoder layers. The word embedding layer maps source/target side natural language texts a/y to their vectored representation Temb/ Yemb- An en- coder layer consists of a self-attention module and non-linear feed-forward module, which outputs the contextualized sentence representation of x. A decoder layer is similar to the encoder layer but has additional cross-attention module between self-attention and feed-forward modules. The input of a decoder layer Jemb attends to itself and then cross-attends to to integrate source sentence in- formation and then generate the translation y. The generation is usually auto-regressive, and a cross- entropy loss is applied to train a transformer model:
For the extraction of semantic units, rela- tively easy to identify semantic units composed of subwords, which simplifies the problem to extract- ing phrases that represent unified meanings within sentences. Generally, words that frequently co- occurent in corpus are more likely to be a phrase. Therefore, we propose Word Pair Encoding (WPE), method to extract phrases based on the relative co-occurrence frequencies of words.
LCE = Σ log Hogplyile (yrr)
(1)
For learning semantic unit representations, we propose an Attentive Semantic Fusion (ASF) layer, exploiting the property of attention mech- anism (Bahdanau et al., 2015b) that the length of the output vector is identical with the length of the query vector. Therefore, it is easy to obtain fixed- length semantic unit representation by controlling the query vector. To achieve this, pooling opera- tions are performed on the input vectors and the result is used as the query vector. Specifically, a combination of max-pooling, min-pooling, and av- erage pooling is employed to preserve the original semantics much as possible.
An attention module projects a query and set of key-value pairs into an output, where the query, key, value, and output are vectors of the same dimen- sion, and the query vector determines the output's length. The output is the weighted sum of values, where the weight is determined by dot product of query and key. For training stability, scaling factor 3 is applied to the weight. The attention Va operation can be concluded as equation (2)
QKT vd V (2)
Attention V) softmart
After obtaining the semantic unit representation, we propose a parameter-effieient method uti- lize The token-level sentence representation and semantic-unit-lewe sentence representation are concatenated together as the input to the encoder layers, with separate positional encoding applied. This allows the model to fully exploit both levels of semantic information. Experimental results demon- strate this approach is simple but effective.
Specifically, in the transformer model, the self- attention module uses same input vector as query and key-value pairs, but the cross-attention module in the decoder uses target side hidden states as query and encoder output x as key-value pairs.
2.2 Byte Pair Encoding (BPE)
NLP tasks face out-of-vocabulan problem which has largely been alleviated by BPE (Sennrich et al., 2016). BPE is a method that cuts words into sub-words based on statistical information from the training corpus.
2 Background
2.1 Transformer Neural Machine Translation model
There are three levels of texts in BPE: character level, sub-word level, and word level. BPE does not learn how to turn word into sub-word, but learns how to combine character-level units into sub-words. Firstly, BPE splits every word into separate characters, which are basic units. Then, BPE merges adjacent two basic units with the high- est concurrent frequency. The combination of two units forms a new sub-word, which also basic
Neural machine translation task is to translate a source sentence into its corresponding target sen- tence y. Our method is based on the Transformer neural machine translation model (Vaswani al., 2017), which is composed of an encoder and a decoder to process source and target sentences re- spectively. Both the encoder and decoder consist

---

Translation
P
Decoder
Feed Forward
Encoder Layer
x.V
concat (O1, O2, 03)
Attentive Fusion
Query
Key
Value
(mncat(Tann T Tmaz; T
Attntive Semantic Fusion
min, max, avg pooling
Taken Embedding
[Tg Ti,1111... Tit8
Tom re@@@e@@@ @ tic@ @ ent in regards toihisiprize
Figure 2: This picture illustrates overview of SU4MT. The left part illustrates the whole model structure. We only modify the embedding layer of the encoder and keep others unchanged. The circles denote token-level representations and circles representations. The right part illustrates the detailed structure of the Attentive Semantic Fusion (ASF) layer. The input is contiguous tokens that constitute a semantic unit and the output of it is a single semantic representation.
same color show they can form a semantic unit. The squares denote semantic-level
unit and it joins the next merge iteration other units do. Merge operation usually iterates 10k-40k times and BPE code table is used to record this process, These operations are called BPE learning, which is conducted on the training set only.
token-level representation is provided to the coder layers to supplement detailed information. We also propose Word Pair Encoding (WPE), an offline method effectively extract phrase spans that indicate the boundaries semantic units.
Applying BPE is to follow the BPE code table and conduct merge operations on training, valida- tion, and test sets. Finally, special sign "@ @" is added sub-words that do not end word. For example, the word "training" may become two sub- words "train @ @" and "ing".
3.1 Learning Semantic Unit Representations
In our proposed method, a semantic unit refers to contiguous sequence of tokens that collec- tively form a unified meaning. This means not only phrases but also words composed of subwords can be considered semantic units.
3 Method
We extract token representations' features through pooling operations. Then, we utilize the attention mechanism to integrate the semantics of the entire semantic unit and map it to fixed-length representation.
In this section, present methods in detail. As depicted in Figure the proposed method in- troduces an additional Attentive Semantic Fusion (ASF) module between the Encoder Layers and Token Embedding layer of the conventional Trans- former model, and only a small number of extra parameters are added. The ASF takes the repre- sentations of several tokens that form semantic unit input, and it integrates the information from each token to output single vector representing the united semantic representation. The sentence representations of both token-level and semantic- unit-level are then concatenated as the input of the first one of the encoder layers. Note that the
The semantic unit representations are obtained through the Attentive Semantic Fusion (ASF) layer, illustrated on the right side of Figure 2. It takes series of token representations Ti & [T,, Ti. Ti+a] input, which form phrase of length & + 1. To constrain the output size as fixed number, we leverage a characteristic of attention mechanism (Bahdanau al., 2015a) that the size attention output is determined by its query vector. Inspired by Xu et al. (2020), who

---

are printed in different colors, and the tokens of the same color can form a semantic unit. When SU4MT takes sentence as input, the token em- bedding' maps these tokens into token-level repre- sentations, denoted by colored circles Figure 2.
Remove WPE Signs
striking <irikingg.... 1ike0 between British long juméé per and...
Apply BPE
The striking likee eness between British Long #ee see jume8 per and...
One copy of the token-level sentence represen- tation is directly prepared be half of the next layer's input after being added with positional en- coding. The other copy goes through the ASF layer and is transformed into the tence representation, denoted by colored squares in Figure 2. It is then added with positional encoding, the position count starting from zero. Subsequently, the two levels of sentence representations are catenated the complete input of encoder layers. In practice, normalization layer is applied to en- hance the model stability.
Apply WPE
The striking likeness between British long #5s jumper and...
sen-
BPE Word Table
WPE Phrase Table Long jumper
lik
j um per
Raw Text
The striking likeness between British long jumper and...
Since the two levels of sentence representations are respectively added with positional encodings, they essentially represent two separate and com- plete sentences. This is like generating transla- tions based on two different perspectives of same natural language sentence, allowing the model capture either the overall information or the fine- grained details as needed. To ensure the validity of concatenating the two levels of sentence repre- sentations, we conducted preliminary experiments. When duplicate of the source sentence is concate- nated to the original one as input of the conven- tional Transformer model, the translation quality does deteriorate. This ensures the rationality of concatenating the two levels of sentence represen- tations.
Figure example of how to get the span positions of semantic units in a subword-level sentence by applying WPE. The boxes are colored differently to distinguish different processing stages. Tokens that form seman- tic unit are highlighted with background color. The two highlighted strings in the uppermost box are the extracted semantic units.
have demonstrated max pooling and average pool- ing are beneficial features for learning phrase rep- resentations, we propose leverage the concatena- tion of min pooling, max pooling, and average pool- ing to preserve more original information, yielding Tpoot = concat(Tmin: Tmaz, Taog), which serves as the query vector for the attentive fusion layer. The attentive fusion layer uses Tiiitk as key and value vector and outputs the fused representation of length 3: O = concat O2, Oz). The output is then transposed be vector with triple-sized embed_dinn dim but length of 1. This is followed by a downsampling feed-forward layer acquire the single-token-sizee semantic representation P:.
3.3 Extracting Semantic Spans
The semantic units encompass both words com- prised of subwords and phrases. The former can be easily identified through the BPE mark "@ @". However, identifying phrases from sentences re- quires more effort. Previous research often relied on parsing tools to extract phrases, which can be time-consuming In our model, the syntactic struc- ture information is not needed, and we only have to concern about identifying phrases. Therefore, we propose Word Pair Encoding (WPE) as fast method for phrase extraction. Since our model only utilizes the phrase boundary information as an aux- iliary guidance, it does not require extremely high accuracy. As a result, WPE strikes a good balance The left part of Figure 2 illustrates how the whole model works. The tokens in the input sentence
In practical applications, we also consider the semantic units that are composed of only one token phrases: a generalized form of phrases with length of 1. Therefore, all tokens processed by ASF be transformed into semantic representa- tions. By doing so, representations of all semantic units are the same vector space.
3.2 Integrating Semantic Units
'Token embedding is the same as word embedding in Transformer: We use this to distinguish tokens and words.

---

Base
Big
Method
Param.
BLEU
Param.
BLEU
Transformer" (Vaswani al., 2017)
65M
27.3
213M
28. .
MG-SA" (Hao et al., 2019)T
89.9M
28.28
271.5M
29.01
LD" (Xu et al., 2020)
173.0M
28.67
29.60
Proto-TF" (Yin et al., 2022)*
28.19
Transformer (Vaswani et al., 2017)
63.10M
28. 28.16((2.29)
209.90M
29.56(28.59)
UMST (Li al., 2022a)
68.65M
29.24 (28.191))))))
237.98M
30.70
SU4MT
66.69M
29.80 (28.58)
232.99M
30.89 (29.71)
Table 1: This table shows experimental results En->De translation task. The proposed SU4MT approach is compared with related works that involve phrases spans. The sign denotes the results come from the original papers, and the sign denotes the approaches use external parsing tools. Besides Multi-BLEU, we report SacreBLEU in brackets. The results show that approach outperforms other methods w/ or w/o prior syntactic knowledge. The 1 sign denotes our approach significantly surpasses Transformer (p < 0.01) and UMST (p < 0.05).
between quality and speed, making it suitable for the following stages of our approach.
is extract semantic units matter whether the integrant parts are words or subwords. To sum up. WPE applied to extract the position of semantic units, without changing the text.
Inspired by byte pair encoding (BPE) (Sennrich et al., 2016), we propose a similar but higher-level algorithm, word pair encoding (WPE), to extract phrases. There are two major differences between WPE and BPE: the basic unit and the merge rule. The basic units in BPE are characters and pre- merged sub-words as discussed in section 2.2. Analogously, basic units in WPE are words and pre-merged sub-phrases. Besides, the merge opera- tion does not paste basic units directly but adds a special sign "#$&" to identify word boundaries.
In practice, there are some frequent long segments in the training corpus, bringing non- negligible noise to WPE. So clip phrases with more than tokens to avoid this problem.
Experiments
In this section, we conduct experiments on three datasets of different scales to validate the effective- ness and generality of the SU4MT approach.
The merge rule means how to determine which two basic units should be merged in the next step. In BPE, the criterion is co-current frequency. In WPE, however, the most frequent co-current word pairs mostly consist of punctuations and stopwords. To effectively identify phrases, we change the crite- rion into a score function as shown in equation (3).
4.1 Data
To explore model performance on small, middle, and large-size training corpus, test SU4MT and related systems on WMT14 English-to- German (En-DDe), WMT16 English-to-Romanian (En RRo), and WMT17 English-to-Chinese (En ZZh) translation tasks.
count(wl, w:2) score /coun((wl) count(w2)
(3)
Datasets
w1 and w2 represent two adjacent basic units, and 8 controllable threshold to filter out noises. Notably, our WPE is orthogonal to BPE and we provide a strategy to apply both of them, as illustrated in Figure 3. Firstly, BPE word table and WPE phrase table are learned separately on raw text X. Then, WPE is applied to yield Xw. denoted by the red boxes. Next, we apply BPE to Xw and obtain XWB, B. denoted by the purple boxes. Finally, the special WPE signs are moved out and result in sub-word level sentences XB, which is the same as applying BPE on raw text. The important thing
For the En-+De task, we use the script "prepare- 4 provided by Fairseq (Ott et al., 2019) to download all the data. The training cor- pus consists of approximately 4.5M sentence pairs. We remove noisy data from the training corpus by filtering out (1) sentences with lengths less than 3 or more than 250, (2) sentences with words com- posed of more than 25 characters, (3) sentence pairs whose ratio of source sentence length over target sentence length is larger than 2 or less than 0.5. After cleaning, the training corpus contains

---

Base
Big
Method
Param.
BLEU
Param.
BLEU
Transformer (Vaswani et al., 2017)
60.92M
(34.2()4.23)
209.9M
34.34(31.17)
UMST (Li et al., 2022a)
60.83M
31.66 (34.50)
222.32M
(4.33
SU4MT
66.69M
(4.87(34.711 f
237.20M
34.73
Table 2:
Experimental results on small-sized WMT16 En-*Ro dataset. The sign denotes the approach uses external parsing tool. The ft sign and f denote the approaches significantly surpass the Transformer system, measured by SacreBLEU(p < 0.05 and < 0.1 respectively).
4.02M sentence pairs. We use newstes12013 as the validation set and use newstes22014 as the test set. For the En >Zh task, we collect the corpora pro- vided by WMT17 (Bojar et al., 2017). The train- ing corpus consists of approximately 25.1M sen- tence pairs, with notable number of noisy data. Therefore, clean the corpus by filtering out non- print characters and duplicated null lines, leaving 20. IM sentence pairs. We use newsdev2017 and newstesr2017 as validation and test sets.
and batch size to 32k. Adam optimizer (Kingma and Ba, 2015) is applied with / = (0.9, 0.98) and 1e - 8. The dropout rate adjusted according to the scales of model parameters and training data. For base-setting models, we set dropout 0.1. For large-setting models, we set dropout = 0.3 for En-+Ro and En- tasks, and set dropoul- = 0.1 for En-*Zh task.
For SU4MT, a pretrain-finetune strategy is ap- plied for training stability. Specifically, we initial- ize our method by training a Transformer model for approximately half of the total convergence steps. At this point, the token embedding parameters have reached relatively stable state, which facilitates the convergence of the ASF layer. The En task, however, is an exception. We train SU4MT models from scratch because only takes few steps for them to converge.
For the En-Ro task, the training corpus con- sists of 610K sentence pairs, and newsdev2016 and newstes:2016 are served as validation and test sets.
Data Preparation
We use mosesdecoder (Kochn et al., 2007) for the tokenization of English, German, and Romanian texts, while the jieba tokenizer is employed for Chinese texts. For the En-De and En-Ro datasets, we apply joint BPE with 32,768 merges and use shared vocabulary with vocabulary size of 32,768. As for the En-Zh dataset, since English and Chi- nese cannot share the same vocabulary, we perform 32,000 BPE merges separately texts in both lan- guages. The vocabulary sizes are not limited to certain number, and they are about 34K for English and about 52K for Chinese.
We reproduce UMST (Li et al., 2022a) according to the settings described in the paper. Note that the authors have corrected that they applied 20k BPE merge operations in the W En-Ro task, and follow this setting in our re-implementation.
Inference Stage
For all experiments. we average the last check- points as the final model and inference with it. Note that, we save model checkpoints for every epoch for En-+Ro and En tasks and every 5000 steps for En Bnn>Zhh task.
In our approach, the proposed WPE method is used to extract the boundaries of semantic units. For all three tasks, we employ the same settings: the 5 value of 100 and 10,000 WPE merges. Finally, the resulting spans longer than 6 are removed.
Evaluation Stage
We report the results with three statistical metrics and two model-based metrics. Due to space limita- tion, we exhibit two mainstream metrics here and display the complete evaluation in Appendix A. For En En-+Ro and En-+De tasks, we report Multi-BLBU for comparison with previous works and Sacre- BLEU as more objective and fair comparison for future works. For the En- task, we discard Multi-BLEr because it is sensitive to the word
4.2 Implementation Details
Training Stage
In SU4MT and the bascline method, the hyperpa- rameters related to the training data scale and learn ing strategies remain unchanged across all tasks and model scales. We set the learning rate to 7c
"https://githeee http:/u...ooo///sj/eeeeeee

---

Method Base Setting
SacreBLEU
ChrF
Param.
Encoder Input
SacreBLEU
Multi-BLEU
teeancse
28.58
29.80
Transformer
35.17
31.18
88.19M
semanticx2
28.28
29.15
SU4MT
35.54
31.40
94.27M
token x
28.14
29.29
Big Setting
semantic only
28.35
29.51
Transformer 35.73
31.77
265.07M
token only
27.54
28.64
SU4MT
36.32
32.12
288.15M
Table 4: In this set of experiments, encoder input is varied discover the effectiveness of concatenating sentence representations of both levels. Experiments are conducted on En Dn task and the strategy of iden- tifying semantic units WPE 10k BPE.
Table 3: Experimental results on En-*Zh translation task are reported with SacreBLEU and ChrF. This is because the tokenization of Chinese is highly dependent the segmentation tool, leading to the unreliability of Multi-BLEU as metric.
representations with only one type, but duplicate and concatenate it the original sentence eliminate the influence of doubled sentence length. As demonstrated by the experimental results, us- ing only one perspective of sentence representation leads to a decrease in translation quality, indicating that diverse sentence representations complement each other and provide more comprehensive seman- tic information.
segmentation of Chinese sentences. Instead, we report SacreBLEU and ChrF (Popovié, 2015). For modellbased metries, COMET (Rei et al., 2022)² and BLEURT (Sellam et al., 2020; Pu et al., 2021)* are leveraged.
4.3 Main Results
We report the experiment results tasks in Ta- ble1, 2, 3. Of all the comparing systems, MG-
Furthermore, we remove the duplicated por- tion to investigate the impact of adding redundant information on translation. Surprisingly, when doubling the token-level sentence representation, SacreBLEU increases by 0.6, but when doubling the semantic-unitlleell sentence representation, SacreBLEEU even exhibits a slight decline. We spec- ulate that repeating the information-spane token- level sentence representation allows the model to learn detailed semantic information and to some extent learn overall semantic representation sepa- rately from the two identical representations. How- ever, semantic-unit-level sentence representation is semantically dense. It's already easy for the model to understand the sentence and duplicating may introduce some noises. In conclusion, using both perspectives of semantic representation together as input maximizes the provision of clear, comprehen- sive semantic and detailed linguistic features.
SH (Hao et al., 2019), Proto-TF (Yin et al., 2022) and UMST (Li et al., 2022a) uses external tools to obtain syntactic information, which is time- consuming. Notably,
approach and LD (Xu al., 2020) can work without any syntactic parsing tools and yield prominent improvement too. Apart from Multi-BLEU, we also report SacreBLEU in brackets. Results Table demonstrate that our approach outperforms other systems in both base and big model settings and significantly surpasses the systems without extra knowledge.
To make the evaluation of translation perfor- mance more convineing, we report detailed evalua- tion scores on five metrics, which are displayed in Appendix A.
5 Analysis
5.1 Ablation Study
To further understand the impact of providing dif- ferent forms of source sentence representations to the encoder layer on model performance, we con- duct three sets of ablation experiments based on the default settings of ASF, as shown in Table 4. We replace one of the two perspectives of sentence
5.2 Influence of Granularity
We also discuss the criterion of how to consider phrase as a semantic unit. We change the WPE merge steps to control the granularities. In Table 5, "BPE" indicates we treat all bpe-split subwords semantic units. "RANDOM" means randomly select consecutive tokens as semantic units until the ratio of these tokens is the same as that of our default setting (approximately 36%). The results
"nttps://huggingee cotp/beaeeeee..
wmt22-conet-da
"https://storee googleeppis.com/bluutttos--
BLEURT-20.

---

Granularity
SacreBLEU
Multi-BLEU 34.71
WPE 5k +BPE
34.62
WPE 10k+BPE
34.71
34.87
Transformer
UMST
29
WPE 15k+BPE
34.54
34.60
SU4MT
31.6%
WPE 10k
34.32
34.45
Group Numbert (())
27.1%
28
Average Sacreli SacreBLEn
BPE
34.31
34.46
RANDOM
34.35
34.48
27
Table 5: In the table, we regard different contiguous tokens semantic units to demonstrate the effectiveness of our method in extracting and leveraging semantic units. Experiments are conducted on En->Ro task. "5k", "10k", and "15k" denote the number of WPE merges, "BPE" indicates subwords that form word are treated as semantic units.
15
26
10-
7.2%
-25
4.0%
24
0
1-2
3-4
5-8
>8
Span Number in Sentence
demonstrate it is essential to integrate semantic meanings of subwords.
Figure 4: En-De test set divided into 5 groups by the number of spans within sentence. Lines exhibit the average SacreBLEU scores of sentences among each group. Bars show the number of sentences in each group. In most scenarios, SU4MT significantly outperforms the contrast methods.
5.3 Performance on Different Span Number To explore the effectiveness of our method in mod- cling semantic units, we divide the test set of the En-De task into five groups based on the num- ber of semantic-unit spans contained in the source sentences (extracted using WPE 10k+BPE). We cal- culate the average translation quality our method and the comparison systems in each group. The results are presented in Figure 4, where the bar graph represents the proportion of sentences in each group to the total number of sentences in the test set, and the line graph shows the variation in average BLC When the span count is O, SU4MT degrades the scenario of doubling the token-level sentence representation, thus resulting only a marginal improvement compared to the baseline. However, when there are spans present in the sentences, SU4MT significantly outperforms the baseline and consistently surpasses the compar- ison system (Li et al., 2022a) in almost all groups. This demonstrates that SU4MT effectively mod- els semantic units and leverages them to improve translation quality.
proposed by RWTH (Vilar et al., 2006), which con- sists of 508 sentence pairs and word alignments. The alignments are labeled either (sure) or P (possible). We train SU4MT and base- line systems on WMT14 En-De dataset and eval- uate them on the RWTH alignment dataset. The recall rate of target-side words that correspond to source-side semantic units is reported the metric. i.e., the proportion of correctly translated semantic units. The results presented in Table 6.
Method Transformer
recall(S)
SacreBLEU recall(P&S)
68.14
65.99
25.27
UMST
69.18
66.96
25.96
SU4MT
69.29
67.24
26.58
Table 6:
Our approach presents more accurate trans- lation on words that align with semantic units. In "re- call(S)", only the words with "sure" alignment are calcu- lated, while in "recall(P&S)". we calculate words with both "sure" and "possible" alignment.
5.4 Effectiveness of Modeling Semantic Units
To evaluate the efficacy of the proposed approach in modeling the integral semantics of semantic units, we calculate the translation accuracy target-side words that correspond to the source-side seman- tic units. For this purpose, we leverage the Gold Alignment for German-Engling Corpus dataset
Related Works
6.1 Multiword Expression
Multiword Expression (MWE) is notion similar semantic units that proposed in our work. Several works have proposed definitions of MWE (Carpuat and Diab, 2010; Calzolari et al., 2002; Sag et al., 2002; Baldwin and Kim, 2010). Although MWE
"https://www/ 16, informatik aecch-aachencco de/ goldAlignmen www.lliggmenniinee...mmm

---

a hot research topic, leveraging it in neural machine translation is hindered by its intrinsic char- acteristics like discontinuity (Constant et al., 2017), e.g., the expression "as as .......
7 Conclusions
In this work, we introduce the concept of semantic units and effectively utilize them to enhance neu- ral machine translation. Firstly, we propose Word Pair Encoding (WPE), a phrase extraction method based on the relative co-occurrence of words, to ef- ficiently extract phrase-level semantic units. Next, propose Attentive Semantic Fusion (ASF), phrase information fusion method based atten- tion mechanism. Finally, we employ simple but effective approach leverage both token-level and Gemantic-unit-les sentence representations. Ex- perimental results demonstrate our method signifi- cantly outperforms similar methods.
Despite the large overlap between these two no- tions, we claim the distinctions of semantic units. For empirical usage, we require semantic units to be strictly consecutive tokens. Subword tokens also considered part of semantic unit.
6.2 Extraction of Phrases
From the perspective of extracting semantic phrases, some of these methods require addi- tional parsing tool to extract semantic units and their syntactic relations, which may cost days in the data preprocessing stage. Phrase extraction has been explored by various methods, such as the rule-based method (Ahonen et al., 1998), the alignment-base based method ( (Venugopal et al., 2003; Vogel, 2005; Neubig et al., 201 1), the syntax-based method (Eriguchi et al., 2016; Bastings et al., 2017), and the neural network-bass -ased method (Mingote al., 2019).
Limitations
Our approach aims to model the semantic units within sentence. However, there still few sentences that contain semantic units as Figure 4 shows. In this scenario, SU4MT brings less im- provement in translation performance. Therefore, the advantage of our approach may be concealed in extreme test sets. Admittedly, SU4MT requires more FLOPs than Transformer does, which is an in- evitable cost of leveraging extra information. There are also potential risks in our work. Limited by model performance, reliable translations, which may cause misunder- standing in cross-culture scenarios.
6.3 Translation with Phrases
model could generate
Several works have discussed leveraging semantic phrasal information for neural machine trans- lation. Some of them take advantage of the span information of phrases but do not model phrase rep- resentation. This kind of method modifies attention matrices, cither conducting matrix transformation to integrate semantic units (Li et al., 2022a) or ap- plying attention masks to enhance attention inside a semantic span (Slobodkin et al., 2022). Other works model united representation of semantic units. Xu et al. (2020) effectively learns phrase representations and achieves great improvement over the baseline system. However, complicated model structure designed to leverage semantic information, resulting in heavy extra parameters. MG-SA (Hao et al., 2019) models phrase represen- tations but requires them to contain syntactic in- formation instead of semantics, which narrows the effectiveness of modeling phrases. Proto-TF (Yin al., 2022) uses semantic categorization to en- hance the semantic representation of tokens ob- tain a more accurate sentence representation, but it fails to extend the method into phrasal semantic representations.
Acknowledgements
We thank the anonymous reviewers and chairs for their insightful comments.
References
Helena Ahonen, Oskari Heinonen, Mika Klemettinen. and A Inkeri Verkamo. 1998. Applying data mining techniques for descriptive phrase extraction digital document collections. In Proceedings IEEE Interna- tional Forum Rexearch and Technology Advances in Digital Libraries-ADI. '98-, pages 2-11. IEEE.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Ben- gio. 2015a. Neural machine translation by jointly learning to aligu and translate. 3rd International Con- ference Learning Representations. ICLR 2015 Conference date: 07-05-2015 Through 09-05-2015.
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Ben- gio. 2015b. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Truck Proceedings.

---

Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. Handbook of natural language process- ing, 2:267-292.
machine translation. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics.
Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima' an. 2017. Graph con- volutional encoders for syntax- aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing. EMNLP 2017. Copenhagen, Denmark, Septem- ber 9-11, 2017, pages 1957-1967. Association for Computational Linguistics.
Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-level universal visual repre- sentations. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5687-5698 Dublin, Ireland. Association for Computational Linguistics.
Ondicj Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthis Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz. Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation 1)). In Proceedings of the Sec- ond Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169-214, Copenhagen, Denmark. Association for Computational Linguis- tics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learn- ing, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252. PMLR.
Jie Hao, Xing Wang, Shuminng Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019. Multi- granularity self-attention for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, SMNLL-IJCNLL 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 887-897. Association for Com- putational Linguistics.
Nicoletta Calzolari, Charles J Fillmore, Ralph Gr- ishman, Nancy Ide, Alessandro Lenci, Catherine MacLeod, and Antonio Zampolli. 2002. Towards best practice for multiword expressions in compu- tational lexicons. In LREC, volume 2, pages 1934- 1940.
Po- Ro-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018 Towards neural phrase- based machine translation. In 6th International Con- ference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net.
Marine Carpuat and Mona Diab. 2010. Task- based evaluation of multiword expressions: a pilot study in statistical machine translation. In Human Lan- guage Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguisties, pages 242-245, Los Angeles, California, Association for Computational Linguistics.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709.
Kyunghyun Cho. Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP 2014, pages
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mathieu Constant, Gülşen Eryigit, Johanna Monti, Lon- neke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Multiword Expression Processing: A Survey. Computational Linguistics, 43(4):837-892.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico. Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondiej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long und Short Pupers), ), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare Aurelio Ranzato. 2018. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Treeence attentional neural

---

5039-5049, Brussels, Belgium. Association for Com- putational Linguistics.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C' Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Muchine Translation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Bei Li, Yi Jing, Xu Tan, Zhen Xing. Tong Xiao, and Jingbo Zhu. 2023. TranSFormer: Slow-fast trans- former for machine translation. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 6883-6896, Toronto, Canada. Association for Computational Linguistics.
Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for nip. In Compu- national Linguistics and Intelligent Text Processing, pages 1-15, Berlin, Heidelberg, Springer Berlin Hei- delberg.
Bei Li, Tong Zheng. Yi Jing, Chengho Jiao, Tong Xiao, and Jingbo Zhu. 2022a. Learning multiscale trans- former models for sequence generation. In Interna- tional Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1225-132411 PMLR.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. In Proceedings of ACL.
Jiacheng Li, Jingho Shang, and Julian McAuley. 2022b. UCTopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 6159-6169, Dublin, Ireland. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, ACL 2016, August 7-12, 2016, Berlin, Ger- many, Volume 1: Long Pupers. The Association for Computer Linguisties.
Tingting Ma. Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin- Yew Lin. 2022. Decomposed meta- learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1584-1596, Dublin, Ire- land. Association for Computational Linguistics.
Aviv Slobodkin, Leshem Choshen, and Omri Abend. 2022. Semantics-aware attention improves neural machine translation. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 28-43, Seattle, Washington. Association for Computational Linguistics.
Victoria Mingote, Antonio Miguel, Alfonso Ortega, and
Eduardo Lleida. 2019. Supervector extraction for encoding speaker and phrase information with neu- ral networks for text-dependent speaker verification. Applied Sciences, 9(6))
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Se- quence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Graham Neubig. Taro Watanabe, Eicchroo Sumita, Shin- suke Mori, and Tatsuya Kawahara. 2011. An unsu- pervised model for joint phrase alignment and extrac- tion. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies. pages 632-641, Port- land, Oregon, USA. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Ashish Venugopal, Stephan Vogel, and Alex Waibel. 2003. Efffcttive phrase translation extraction from alignment models. In Proceedings of the 41st Annual Meeting of the Association for Computational Lin- guistics, pages 319-326, Sapporo, Japan. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
David Vilar, Maja Popovic, and Hermann Ney. 2006. AER: do we need to "improve" our alignments? Proceedings of the Third International Workshop on Spoken Language Translation: Papers, Kyoto, Japan.
Stephan Vogel. 2005. PESA: Phrase pair extraction as sentence splitting. In Proceedings of Machine Trans- lation Summit X: Papers, pages 251-221 1-1225, Phuket. Thailand.
Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for mt. In Proceedings of EMNLP.

---

Hongfei Xu, Josef van Genabith, Deyi Xiong, Qiuhui Liu, and Jingyi Zhang. 2020. Learning source phrase representations for neural machine translation. In Proceedings of the 58th Anmual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 386-396. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang. Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards Token- Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291-306.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLPP) pages 6442-6454, On- line. Association for Computational Linguisties.
Yongiin Yin, Yafu Li, Fandong Meng, Jie Zhou, and Yue Zhang. 2022. Categorizing semantic represen- tations for neural machine translation. In Proceed- ings of the 29th International Conference Com- putational Linguistics, pages 9227-5239, Gyeongju. Republic of Korea, International Committee on Com- putational Linguisties.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhen- grui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023. Bayling: Bridging cross-lingual align- ment and instruction following through interactive translation for large language models, arXiv preprint 209668.006.0068.

---

A More Evaluation Metrics
To better evaluate our model and baseline systems, and to provide a complete benchmark for future research on machine translation, we report the results on five metrics. "Multi-BLEU", "SacreBLEU", and "ChrF" are statistical metrics; "COMET" and "BLEURT" are model-based metrics.
Method
Parameters
Multi-BLEU SacreBLEU ChrF
COMET BLEURT
Base setting
Transformer
65M
28.16
27.29
57.68
83.11
71.74
UMST
28.65M
29.24
28.19
57.75
83.20
71.75
SU4MT
66.69M
29.80
28.58
57.84
83.28
72.05
Large setting
Transformer
213M
28.56
28.59
57.25
83.76
72.63
UMST
237.98M
30.70
29.56
58.76
84.25
73.32
SU4MT
232.99M
30.89
29.71
58.69
84.44
73.61
Table Experimental results on En translation task. This complete metric version of Table 1.
Method
Parameters
Multi-BLEUU
SacreBLEU
ChrF
COMET
BLEURT
Base setting
Transformer
60.92M
34.42
34.23
60.14
81.08
71.97
UMST
60.83M
34.66
34.50
60.38
81.49
72.36
SU4MT
66.69M
34.87
34.71
60.59
81.71
72.78
Large setting
Transformer
209.90M
34.34
34.17
60.33
81.52
72.56
UMST
222.32M
31.58
31.15
60.53
81.93
72.88
SU4MT
237.20M
34.73
34.53
60.29
81.76
72.61
Table 8: Experimental results En-Roo translation task. This complete-mente metric version of Table 2.
Method
Parameters
Multi-BLEEU SacreBLEU
ChrF
COMET
BLEURT
Base setting
Transformer
88.49M
21.40
35.17
31.18
82.96
63.24
UMST
121.6/M
21.37
35.52
31.33
83.01
63.26
SU4MT
94.27M
21.51
35.54
31.40
83.29
63.31
Large setting
Transformer
265.07M
22.12
35.73
31.77
83.53
63.71
UMST
343.94M
22.29
36.54
32.31
83.68
64.21
SU4MT
288.15M
21.96
36.32
32.12
83.78
64.31
Table 9: Experimental results on En-tZh translation task. This complete-metric version of Table 3.

---

B Scaling up Training corpus
We apply our approach to an even larger dataset: WMT14
the training corpus of which consists of approximately 35M sentence pairs. We apply joint BPE with 40,000 merges and use shared vocabulary with size 40K. The hyper-parameters of WPE are = 100 and 10,000 merges. We save model checkpoints for every 5,000 steps and average the last 5 checkpoints for inference.
Method
Parameters
Multi-BLEU
SacreBLEU
ChrF
COMET BLEURT
Base setting
Transformer
64.62M
40.60
37.38
63.62
84.03
70.00
SU4MT
70.39M
41.09
37.82
63.98
84.49
70.74
Large setting
Transformer
217.32M
41.46
38.29
64.27
85.09 85.54
71.43
SU4MT
210.10M
42.00
38.90
64.69
72.29
Table 10: Experimental results on En-Frr translation task.
English-Targeted Experiment
Besides experimenting with English as the source language, we further implement our approach on English-targeted translation datasets. Specifically, leverage the same datasets mentioned in 4.1, but put English the target side. Table11 shows the translation performance five metrics.
Method
Multi-BLEU
SacreBLEU
ChrF
COMET
BLEURT
Transformer
32.60
31.65
57.96
82.94
71.04
De-En
SU4MT
33.07
32.02
58.31
83.21
71.37
Transformer
33.83
33.67
59.90
79.45
67.09
Ro-Lin
SU4MT
34.44
34.21
60.28
80.00
67.66
Transformer
24.40
23.86
52.97
81.04
66.88
Zh-En
SU4MT
24.97
24.41
53.40
81.22
67.03
Table 11: Results of English-targee targeted experiments, which are conducted on WMT14 De-tE dataset, WMT16 En dataset, and WMT17 -En dataset. All models in base scale setting.


Reviews:
[{"review_id": "WhO3JAxnSb", "reviewer": "Reviewer_PSur", "summary": "", "questions": "- Whether the proposed methods work in a larger-scale data or using a larger-scale model? do you have any ideas about the performance tredency when model or data scales up.\n- Any relateness to the Multiword Expression, see https://direct.mit.edu/coli/article/43/4/837/1581/Multiword-Expression-Processing-A-Survey?\n- Any impact on the decoding time?  Especially if semantic units could make decoded sequences shorter. \n- Are there any other ways to implement the similar mechanism like semantic units? I am not an expert in translation, but I felt like there should be some similar mechanism like semantic units -- sorry, this is a little bit subjective since it seems using   semantic units is straightforward.", "limitations": "", "rating": 2, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we have to express great gratitude to you for carefully reviewing the paper and providing the term **Multiword Expression(MWE)**. We believe its notion -*MWEs consist of several words (in the conventionally understood sense) but behave as single words to some extent*- largely overlaps with the notion of *semantic unit* in our work, and will add MWE in our Background section and illustrate the relationship between MWE and semantic unit.\n\nNow, we will answer your questions as follows:\n\n1. **Comparative Analysis: Where does baseline fail where ASF succeeds?**: \n\nThis kind of situation happens when some of the component words in MWE have their own meaning. \n\nTake a part of a sentence in WMT17 EN-ZH testset for example: \n\n*the striking lik@@ eness between British long jum@@ per Gre@@ g Ru@@ ther@@ ford and popular actor Neil Patrick Har@@ ris .*\n\nThe output of baseline system is:\n\n英国 长衣 选手 格雷格·鲁瑟福德 和 著名 演员 尼尔·帕特里克 · 哈里斯 之间 惊人 的 相似 。\n\nThe output of out model is:\n\n英国 跳远 运动员 格雷格·鲁瑟福德 ( Greg Rutherford ) 和 著名 演员 尼尔·帕特里克·哈里斯 ( Neil Patrick Harris ) 之间 的 惊人 相似性 。\n\nHere, *long jumper* means a kind of athlete, whereas *jumper* also means a kind of clothes. Our model treats *long jum@@ per* as a whole and correctly translates it into \"跳远\", while the baseline system fails to understand it and translates it into \"长衣\", meaning long clothing.\n\n2. **Performance when model or data scales up**: \n\nWe are sorry that with limited computing resources, we are not able to conduct sufficient experiments on larger model scales currently, nevertheless, we will leave this for future work. Intuitively, a conventional Transformer will still meet the discussed problem as model or data scales up, since it doesn't explicitly model semantic units. Therefore, the proposed ASF approach will still be effective, so far demonstrated by Base and big settings.\n\nAlthough we were not able to scale up the model size, we experimented on a larger dataset: the WMT14 En-Fr dataset, with 36M sentences. The results are as follows:\n\n| **dataset** | **method** | **model scale** | **multibleu** | **sacrebleu** | **chrf** | **comet** | **bleurt** |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| En-Fr | Transformer | base | 40.60| 37.38 | 63.62 | 84.03 | 70.00  |\n|  | Ours | base | **40.96** | **37.61** | **63.79** | **84.47** | **70.72**  |\n\n3. **Any relatedness to Multiword Expression?**: \n\nThanks for providing us with this survey, which we find quite helpful. The answer to your question is definitely yes, and the major notions of them are the same: several words/tokens behave like a single word to some extent. \n\nThe difference mainly lies in the “domain of definition”: MWE describes the phenomenon in natural languages and every atom part is a word, whereas the semantic unit is defined on the model level and the atom part can be a subword. Another difference is semantic units have to consist of contiguous words, yet MWEs don’t have to. In future work, we may extend semantic units to token-level MWEs and benefit the research about MWEs.\n\nWe will carefully revise our paper to introduce MWE in the background section and illustrate its relatedness to our work.\n\n4. **Impact on decoding time? Can decoded sequences be made shorter?**:\n\nWe are surprised that we have a similar idea, but for future work. In this model, the decoding time is slightly increased because the source sentence is prolonged. Empirically, we test the decoding speed of our model and the baseline system, by testing on the WMT14 En-De testset 5 times and calculating the average speed. This experiment is conducted on 1 Tesla T4 GPU. Our full model generates about 3377 tokens/s, and the baseline system generates about 3390 tokens/s.\n\nYour idea about shortening the decoded sequence is intriguing and may provide help for LLMs. Here is how we want to extend our work for this objective. Since we have verified the effectiveness of our ASF module, integrating the meaning of several tokens, we can further train a reverse module to decode the integrated representation into several tokens. In this case, Decoder takes in a long text and transfers it into a shorter “hidden text”, which can reduce the computing time, and dilates the “hidden text” into longer output. Therefore, we consider our work a first step toward this goal, which is quite meaningful.\n\n5. **Other ways to implement similar mechanism like semantic units?**:\n\nSorry, but we are a bit confused by the word “implement”. Please forgive our potential misunderstanding. Do you mean how to find the subwords that constitute a semantic unit? There are some parsing-based tools like *spaCy* or *Stanford Parser*, but they are either limited in extracting noun chunks or too slow. Or do you mean other mechanisms like MWE? We consider MWE a good mechanism to achieve our motivation, but the problem still lies in how to extract MWEs from texts. Just as you said, semantic units and WPE are straightforward. We will continue to study to find an elegant solution.\n\nThanks again for your patience in reviewing and for providing the MWE mechanism. It lightens the way to possibly combine our work with existing linguistic phenomena in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clarity: The concept of integrating multiple subwords into a single vector to encapsulate the meaning is explained lucidly.\n\n- Experimental Results: It's commendable that the authors have provided experimental proof of the model's superiority over existing baselines. This gives weight to their proposal and ensures that it's not merely theoretical.\n\n", "reasons_to_reject": "- Comparative Analysis: While the paper mentions outperformance over strong baselines, a more detailed comparative analysis would be beneficial. For instance, where does the baseline fail where AttnSF succeeds in translation?", "questions_to_authors": "- Whether the proposed methods work in a larger-scale data or using a larger-scale model? do you have any ideas about the performance tredency when model or data scales up.\n- Any relateness to the Multiword Expression, see https://direct.mit.edu/coli/article/43/4/837/1581/Multiword-Expression-Processing-A-Survey?\n- Any impact on the decoding time?  Especially if semantic units could make decoded sequences shorter. \n- Are there any other ways to implement the similar mechanism like semantic units? I am not an expert in translation, but I felt like there should be some similar mechanism like semantic units -- sorry, this is a little bit subjective since it seems using   semantic units is straightforward.", "paper_topic_and_main_contributions": "The authors present an innovative method named \"Attentive Semantic Fusion (AttnSF)\" to address a recognized issue in Neural Machine Translation (NMT) – the potential overlooking of semantic connections between subwords. The paper suggests that by fusing the semantics of multiple subwords into a unified vector, the model can more adeptly comprehend the input sentence. A dual-layer architecture that utilizes both token-level and semantic-unit-level sentence information is proposed.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2if7ow1x98", "reviewer": "Reviewer_V8Rx", "summary": "", "questions": "Question A: While the innovative neural architectures introduced are intriguing, how do they compare in terms of computational efficiency and training time against more conventional NMT models?\nQuestion B: Could you elaborate on the choices behind the specific datasets and languages used in the experiments? How do you anticipate the proposed methods to generalize to low-resource languages or more diverse linguistic scenarios?\nQuestion C: In the comparative analyses, were any other state-of-the-art techniques or architectures considered but not included in the paper? If so, could you provide insights into their performance relative to the proposed methods?\nQuestion D: While the paper focuses on enhancing the performance and robustness of NMT systems, were there any observed scenarios or edge cases where the proposed techniques might not be optimal or could potentially falter?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your detailed review comments, which are comprehensive and very helpful.\n\nWe will answer your questions as follows:\n\n1. **Potential Overemphasis on Novelty**: \n\nFirstly, it is true that simpler models can achieve similar results, but what we want is not only improving translation quality. Specifically, we noticed a kind of language phenomenon that some words combine to form an integral meaning, which is similar to the notion of \"Multiwords Expressions\" mentioned by Reviewer PSur. Therefore, we propose a method to extract these semantic units and calculate their semantic representations, and this is our major motivation. Secondly, our approach does provide a computationally simple setting (see section 5.1, the \"semantic only\") which has similar computation complexity with the baseline system but outperforms it.\n\n2. **Generalizability Concerns**: \n\nWe understand your concern because all of our three experiments are translation tasks from English to other languages. So, we provide results on extra experiments:\n\n| **dataset** | **method** | **model scale** | **multibleu** | **sacrebleu** | **chrf** | **comet** | **bleurt** |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| De-En | Transformer| base | 32.60 | 31.65 | 57.96 | 82.94 | 71.04  |\n|  | Ours| base | **33.07** | **32.02** | **58.31** | **83.21** | **71.37**  |\n| Ro-En | Transformer| base | 33.83 | 33.67 | 59.90 | 79.45 | 67.09  |\n|  | Ours | base | **34.44** | **34.21** | **60.28** | **80.00** | **67.66**  |\n| Zh-En | Transformer| base | 24.40 | 23.86 | 52.97 | 81.04 | 66.88  |\n|  | Ours | base | **24.97** | **24.41** | **53.40** | **81.22** | **67.03** |\n\nTo demonstrate generalizability in more languages, as well as to demonstrate the effectiveness of our model on an even larger dataset, we conducted an experiment on the WMT14 En-Fr dataset, with 36M sentences. The results are as follows:\n\n| **dataset** | **method** | **model scale** | **multibleu** | **sacrebleu** | **chrf** | **comet** | **bleurt** |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| En-Fr | Transformer | base | 40.60| 37.38 | 63.62 | 84.03 | 70.00  |\n|  | Ours | base | **40.96** | **37.61** | **63.79** | **84.47** | **70.72**  |\n\n\nWe hope this can answer your concerns.\n\n3. **Lack of Comparative Analysis**: \n\nAs far as we know, UMST is the SOTA method among non-contemporary related works. We are not sure which work you mean. Please let us know if we missed any strongly related works and we will supplement a comparative analysis soon.\n\n4. **Implementation Details and Reproducibility**: \n\nWe agree that implementation details are very important and thus provide about 1 page to introduce details in our experiments for better reproducibility. Please understand there's no more space to showcase everything. Could you please specify the details you want to know so that we can reply accordingly? Anyway, we will open source all of the codes and scripts to reproduce the results after the anonymity period.\n\n5. **Potential Overfitting to Specific Scenarios**: \n\nWe conducted experiments on small-scale dataset WMT16 En-Ro(0.6M), medium-scale dataset WMT14 En-De(4.5M), and large-scale dataset WMT17 En-Zh(25.1M), which are all commonly used in Machine Translation research. Therefore, we do not think the judgment \"overly tailored to specific datasets or linguistic scenarios\" is convincing.\n\n6. **Insufficient Discussion on Limitations**: \n\nThanks for your kind reminder and we are aware of the extra computational complexity problem. You may check the rebuttal on his/her review for detailed analysis. We will include this in our limitations in the future version.\n\n7. **Question A**: \n\nTheoretically and empirically, our model is about half the speed of the conventional Transformer. Here follows the analysis:\n\n①Theoretically, computational complexity for *attention* is $L^2 * D$, and that for *FFN* is about $L * D^2$, where $L$ means length and $D$ means model dimension. In translation tasks, $L$ is usually less than 50 in most cases while $D$ is 512 for base-scale models. Therefore, we can roughly get D >> L, and the computational cost of doubled source length is: $(2L)^2 * D + 2L * D^2$ =$2L * D * (2L + D)$ ≈ $2L * D * D$, about twice of baseline system: $L * D^2$.\n\n②Empirically, we compare the training time of the baseline model and our model in the same environment. On 4 NVIDIA 2080Ti GPU, train both models on the WMT16 En-Ro dataset, baseline system updates 3.15 steps/s on average, and our model updates 1.58 steps/s on average.\n\nAlthough our model is computationally complex, it significantly improved translation quality by facilitating phrasal semantic information. Also, our \"semantic only\" setting model is less complex than the baseline system, but performs better. Moreover, we will try to diminish the gap between the \"semantic only\" model and the full model in future work.\n\n8. **Question B**: \n\nWe have claimed why we use these datasets in *Answer 5*. We do not expect our model to generalize well in low-resource languages since no special modification is made towards low-resource scenarios. Again, our primary motivation is to find and model phrases and subword-composed words. Therefore, our model may perform much better when there are more slangs in the corpus. \n\n9. **Question C**: \n\nAs mentioned in *Answer 3*, among all the non-contemporary related works, we could find no other strong SOTA better than UMST or perform similarly well. \n\n10. **Question D**: \n\nOf course, every method will meet scenarios that it might not be optimal, and so will our model. As mentioned in the limitations section, if there's no semantic unit in the source sentence, our model will fall back into simply doubling the source sentence, and the performance may falter.\n\nThanks again for your hard work in reviewing!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty in Approaches: The paper introduces innovative neural architectures and methodologies tailored for Neural Machine Translation. These advancements push the boundaries of what's achievable in the realm of NMT, offering fresh perspectives and tools for researchers and practitioners alike.\n\n2. Comprehensive Experiments: The depth and breadth of the experiments conducted validate the efficacy of the proposed methods. The experiments cover multiple datasets, languages, and domains, ensuring the findings are robust and widely applicable.\n\n3. Significant Performance Improvements: The research showcases marked improvements in translation quality over standard NMT systems. Such advancements are crucial as the demand for high-quality machine translations grows, especially in professional and academic settings.\n\n4. Balancing Quality with Efficiency: Beyond just enhancing translation quality, the paper emphasizes maintaining computational efficiency. This dual focus ensures that the proposed methods are not just theoretically sound but also practically deployable.\n\n5. Beneficial for Diverse Linguistic Scenarios: The techniques presented show promise in handling diverse linguistic nuances and contextual intricacies, making them especially valuable for complex translation tasks and low-resource languages.\n", "reasons_to_reject": "1. Potential Overemphasis on Novelty: While the paper introduces new neural architectures and methodologies, there might be concerns regarding the actual necessity of such complexities. It's crucial to determine if simpler models, with proper tuning, could achieve similar results, thereby questioning the real-world applicability of the proposed advancements.\n\n2. Generalizability Concerns: The paper, although comprehensive in its experiments, might still leave questions about the generalizability of its findings. It's essential to understand how the proposed methods fare across a broader spectrum of languages, especially those not covered in the study.\n\n3. Lack of Comparative Analysis: While the paper showcases the strengths of the proposed methods, there might be a perceived lack of in-depth comparative analysis with other state-of-the-art techniques. Such comparisons could provide clearer insights into the actual advancements made.\n\n4. Implementation Details and Reproducibility: There might be concerns regarding the clarity of implementation details, making reproducibility challenging. A clearer exposition of the methodologies, hyperparameters, and training procedures would be beneficial for the broader community.\n\n5. Potential Overfitting to Specific Scenarios: The techniques, while promising, might run the risk of being overly tailored to specific datasets or linguistic scenarios, thereby compromising their broad applicability.\n\n6. Insufficient Discussion on Limitations: Every methodology has its limitations, and a more thorough discussion on potential pitfalls or scenarios where the proposed techniques might not be optimal could provide a more balanced perspective.\n", "questions_to_authors": "Question A: While the innovative neural architectures introduced are intriguing, how do they compare in terms of computational efficiency and training time against more conventional NMT models?\nQuestion B: Could you elaborate on the choices behind the specific datasets and languages used in the experiments? How do you anticipate the proposed methods to generalize to low-resource languages or more diverse linguistic scenarios?\nQuestion C: In the comparative analyses, were any other state-of-the-art techniques or architectures considered but not included in the paper? If so, could you provide insights into their performance relative to the proposed methods?\nQuestion D: While the paper focuses on enhancing the performance and robustness of NMT systems, were there any observed scenarios or edge cases where the proposed techniques might not be optimal or could potentially falter?\n", "paper_topic_and_main_contributions": "This paper is centered around the theme of enhancing the performance and robustness of Neural Machine Translation (NMT) systems. With the ever-growing complexity and demands of translation tasks, there's a pressing need to not only improve the accuracy of translations but also to ensure that the models generalize well across diverse linguistic scenarios. In essence, the paper presents a comprehensive exploration into the realm of Neural Machine Translation, offering both theoretical insights and practical solutions. The proposed methods promise to elevate the standards of NMT, making translations more accurate, context-aware, and efficient.\nMain Contributions:\n1. Advanced Modeling Techniques: The paper introduces innovative neural architectures and methodologies designed explicitly for NMT. These techniques aim to capture deeper contextual information and linguistic nuances, leading to more accurate translations.\n2. Computationally-aided Linguistic Analysis: Through a series of experiments, the paper sheds light on how different linguistic factors influence translation quality. This analysis is pivotal in understanding where typical models falter and how the proposed techniques make a difference.\n3. NLP Engineering Experiments: The research conducts extensive experiments across multiple datasets, languages, and domains. These experiments validate the efficacy of the proposed methods and showcase their superiority over standard NMT approaches.\n4. Approaches for Data- and Compute Efficiency: The paper touches upon methods that not only enhance translation quality but also ensure that computational efficiency is not compromised. This balance is crucial in real-world applications where both accuracy and speed are paramount.\n5. Publicly Available Software and Pre-trained Models: While it would need explicit confirmation within the paper's content, the depth and significance of the research suggest a potential for releasing tools, frameworks, or pre-trained models, making it easier for the community to leverage the findings.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are too many colors in Figure 1 and Figure 2 without any meaning which might be confusing for readers. "}, "extra_text": {}}, {"review_id": "pz0gH6HIYS", "reviewer": "Reviewer_8dZL", "summary": "", "questions": "1. In Line 490, it seems that this paper is not complete.\n\n2. Can you provide the results for COMET(https://github.com/Unbabel/COMET) and BLEURT(https://github.com/google-research/bleurt)? These two metrics may better align with human preferences.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to express our sincere gratitude for your hard work in reviewing and your appreciation for our work.\n\nNow, we will answer your questions as follows:\n\n1. **\"semantic representations\" are not semantics**: \n\nThanks for asking this good question which can bring more insights to our method and we design an analysis experiment to answer this question. We extract semantic-level representations from the output of the ASF module and calculate the cosine similarities between them and the representations of all single-tokens. For each semantic unit, we collect the top-k closest tokens to examine if they have similar semantics to the semantic unit. We experimented on the model trained on WMT14-ende dataset and present some results in the table below:\n| **Semantic units** | **Top-5 closest tokens** |\n|:---:|:---:|\n| United States | States, **USA**, United, staaten, **US** |\n| as well | well, **also**, in, **auch**, selbst |\n| a lot of | lot, **much**, Schadenser@@, **many**, **lots** |\n| aston@@ ished | **surprised**, **surprise**, regret, odi@@, Venice |\n\nThis table helps to understand the \"semantics\" of the semantic units. Not only nouns and adverb phrases are close to their semantic meanings, but subwords (e.g., \"aston@@ ished\") also recovered to the original meaning. \n\nFor more results, we draw a figure with the dimensionality-reduced representations of semantic units and tokens. We will add this experiment and the results in the next version.\n\n2. **Incomplete sentence**: \n\nWe are sorry that we forgot to delete the word \"Besides,\". We will fix it and carefully examine the whole presentation of the paper.\n\n3. **Evaluate results with COMET and BLEURT**:\n\n Sure, the results are shown below, where \"asf\" indicates our model and \"umst\" is the strong SOTA:\n\n| **dataset** | **method** | **model scale** | **multibleu** | **sacrebleu** | **chrf** | **comet** | **bleurt** |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| En-De| Transformer  | base | 28.46 | 27.29 | 57.68 | 83.11 | 71.74  |\n|  |  | big | 29.56 | 28.59 | 57.25 | 83.76 | 72.63  |\n|  | UMST | base | 29.24 | 28.19 | 57.75 | 83.20 | 71.75  |\n|  |  | big | 30.70 | 29.56 | **58.76** | 84.25 | 73.32  |\n|  | Ours| base | **29.80** | **28.58** | **57.84** | **83.28** | **72.05**  |\n|  |  | big | **30.89** | **29.71** | 58.69 | **84.44** | **73.61**  |\n| En-Ro | Transformer | base | 34.42 | 34.23 | 60.14 | 81.08 | 71.97  |\n|  |  | big | 34.34 | 34.17 | 60.33 | 81.52 | 72.56  |\n|  | UMST  | base | 34.66 | 34.50 | 60.38 | 81.49 | 72.36  |\n|  |  | big | 34.58 | 34.45 | **60.53** | **81.93** | **72.88**  |\n|  | Ours| base | **34.87** | **34.71** | **60.59** | **81.71** | **72.78**  |\n|  |  | big | **34.73** | **34.53** | 60.29 | 81.76 | 72.61  |\n| En-Zh| Transformer | base | 21.40 | 35.17 | 31.18 | 82.96 | 63.24  |\n|  |  | big | 22.12 | 35.73 | 31.77 | 83.53 | 63.71  |\n|  | UMST  | base | 21.37 | 35.52 | 31.33 | 83.04 | 63.26  |\n|  |  | big | **22.29** | **36.54** | **32.31** | 83.68 | 64.21  |\n|  | Ours | base | **21.51** | **35.54** | **31.40** | **83.29** | **63.31**  |\n|  |  | big | 21.96 | 36.32 | 32.12 | **83.78** | **64.31** |\n\n(Note: multibleu in Chinese is relatively less consistent with sacrebleu, because the sentence length can vary a lot after Chinese tokenization or after detokenization.)\n\nWe will add the above results in our next version and hopefully set a comprehensive benchmark for these MT datasets.\n\nThank you again for your precious questions and valuable suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Inspired by byte pair encoding (BPE), the authors adopt word pair encoding (WPE) to extract Semantic Spans. Moreover, WPE is orthogonal to BPE and does not introduce additional noise.\n\n2. Table 4 shows that \"Semantic Embedding Only\" has already improved the translation performance, demonstrating the effectiveness of the proposed phrase information.", "reasons_to_reject": "1. The term \"semantic\" as utilized in the paper, is misleading.  I can not find any evidence to support the claim that these representations are ''semantics''.\n\n", "questions_to_authors": "1. In Line 490, it seems that this paper is not complete.\n\n2. Can you provide the results for COMET(https://github.com/Unbabel/COMET) and BLEURT(https://github.com/google-research/bleurt)? These two metrics may better align with human preferences.", "paper_topic_and_main_contributions": "This paper presents a novel approach to augment translation performance by leveraging phrase information. The authors introduce the concept of Word Pair Encoding, which facilitates the extraction of phrases, and propose the integration of an Attentive Semantic Fusion (ASF) module to incorporate the extracted phrase information into the translation process effectively. The generated semantic embeddings from the ASF module and token embeddings are conjoined in the encoder. Experimental results on En-De, En-Ro, and En-Zh language pairs demonstrate the performance gains from phrase information in translation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c9J5I2zDoP", "reviewer": "Reviewer_mJLU", "summary": "", "questions": "A. In figure 5, why is UMST much better with 0 spans?\n\nB. Why equation (3) exactly instead of e.g. PMI/PPMI?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we want to express our sincere gratitude for your patient review, valuable questions, and kind suggestions on the paper presentation.\nNow, we will answer your questions as follows:\n1. **Computational costs**: \n\nAdmittedly, in the worst cases the source sentence can double in length. The overall computational cost is about 2 times of the baseline system and we will analyse this:\n\n①Theoretically, computational complexity for *attention* is $L^2 * D$, and that for *FFN* is about $L * D^2$, where $L$ means source sentence length and $D$ means model dimension. In translation tasks, $L$ is usually less than 50 in most cases while $D$ is 512 for base-scale models. Therefore, we can roughly get D >> L, and the computational cost of doubled source length is $(2L)^2 * D + 2L * D^2$ =$2L * D * (2L + D)$ ≈ $2L * D * D$, about twice of baseline system: $L * D^2$.\n\n②Emperically, we compare the training time of the baseline model and our model in the same environment. On 4 NVIDIA 2080Ti GPU, train both models on the WMT16 En-Ro dataset, baseline system updates 3.15 steps/s on average, and our model updates 1.58 steps/s on average.\n\n③We further test the decoding speed of our models, by inferencing the WMT14 En-De testset 5 times and calculating the average speed. This experiment is conducted on 1 Tesla T4 GPU. Our full model generates about 3377 tokens/s, and the baseline system generates about 3390 tokens/s. We conjecture this is because at the inference stage, the Encoder is forwarded only once, but the Decoder will be forwarded many times, mitigating the inference time gap between our model and the baseline system. (We only modified the Encoder)\n\nAlthough our model is computationally complex, it significantly improved translation quality by facilitating phrasal semantic information. Also, our \"semantic only\" setting model is less complex than the baseline system but performs better (in Table 4). Moreover, we will try to diminish the gap between the \"semantic only\" model and the full model in future work, in which case our model will be less computationally complex.\n\n2. **Evaluate the approach with other input languages**: \n\nWe reverse the translation direction of the datasets mentioned in our paper, and the results are below. The improvements over the baseline system still hold measured by statistical and model-based metrics.\n| **dataset** | **method** | **model scale** | **multibleu** | **sacrebleu** | **chrf** | **comet** | **bleurt** |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| De-En | Transformer | base | 32.60 | 31.65 | 57.96 | 82.94 | 71.04  |\n|  | Ours| base | **33.07** | **32.02** | **58.31** | **83.21** | **71.37**  |\n| Ro-En| Transformer | base | 33.83 | 33.67 | 59.90 | 79.45 | 67.09  |\n|  | Ours| base | **34.44** | **34.21** | **60.28** | **80.00** | **67.66**  |\n| Zh-En | Transformer | base | 24.40 | 23.86 | 52.97 | 81.04 | 66.88  |\n|  | Ours | base | **24.97** | **24.41** | **53.4** | **81.22** | **67.03** |\n\n3. **Explanation for Figure 5**: \n\nSince UMST is not our approach, we will try our best to answer it according to our understanding. UMST proposed a multi-scale structure, modeling the relationships not only between subword and word but also between words that are syntactically related. A syntactic tree can be drawn from almost any sentence, while a few sentences do not contain any subwords and phrases (the 0 spans situation). Therefore, in 0 spans situation, UMST can still use extra information to improve translation quality, but our ASF model gets no information gains. \n\nWhy is sentence BLEU of UMST much higher than others? We conjecture this is because sentences with no spans are usually short in length. Also, BLEU is more sensitive to shorter sentences than longer ones. Therefore, UMST only needs to translate one or two more words correctly and can improve sentence BLEU significantly.\n\n4. **Why not PMI/PPMI**: \n\nWe have tried PMI as the score function for WPE, but we found that PMI-based WPE tends to extract phrases composed of low-frequency words, mostly uncommon names. We demonstrate why PMI is less suitable in our scenario by comparing it with equation(3) as follows:\n\nProof:\n\nlet N represents the number of unigram word in the corpus, $\\hat{N}$ represents the number of bigram word pairs in the corpus:\n$$\nN=\\sum_{w\\in corpus}{freq(w)}\n$$\n$$\n\\hat{N}=\\sum_{(w_1,w_2)\\in corpus}{freq(w_1,w_2)}\n$$\nHere, N is slightly larger than $\\hat{N}$. So we can rewrite PMI with \"count()\" function:\n\n$$\nPMI=\\log{\\dfrac{p(w_1,w_2)}{p(w_1)p(w_2)}}=\\log{\\dfrac{\\dfrac{count(w_1,w_2)}{\\hat{N}}}{\\dfrac{count(w_1)\\times count(w_2)}{N^2}}} \\approx\\log{\\dfrac{N\\times count(w_1,w_2)}{count(w_1)\\times count(w_2)}}\n$$\n\nSince PMI is followed by an argmax function, the *log* can be removed to get PMI':\n$$\nPMI'=\\dfrac{N\\times count(w_1,w_2)}{count(w_1)\\times count(w_2)}, argmax(PMI)=argmax(PMI')\n$$\nThe score function in our paper is similar to PMI':\n$$\nscore=\\dfrac{count(w_1,w_2)}{\\sqrt{count(w_1)\\times count(w_2)}}\n$$\nNow, let's compare PMI' and our proposed score function:\n$$\n\\dfrac{PMI'}{score}=\\dfrac{N}{\\sqrt{count(w_1)\\times count(w_2)}}\n$$\nwhen $count(w_1)\\times count(w_2)$ gets smaller, the ratio of *PMI'* and *score* becomes larger, indicating that **although both methods tend to find phrases with higher co occurrence of words, PMI prefers words with smaller single word frequencies, and our criteria finds out more frequent words**.\nRemember our motivation is to find more frequently used word pairs. Therefore, our proposed score function outweighs PMI in this scenario.\n\nWe further show some examples from PMI-based WPE and frequency-based WPE:\n* PMI-based: ①Pridnestrovskaia Moldavskaia, ②FLIK FLAK, ③Latterie Friulane, ④Raccordo Anulare, ⑤DOMINA SILVIA ...\n* frequency-based: ①Hong Kong, ②contextual listings, ③Buenos Aires, ④Luggage Storage, ⑤Deposit Box ...\n\n5. **Missing References**: \n\nGreat thanks for providing related works and we will include it in the future version.\n\n6. **Presentation Improvements**: \n\nThanks for your patience and precious advice, we will fix the typos soon and humbly consider your suggestions for better organizing this paper. These problems will be settled in the next version.\n\nThanks again for your hard work and kindness in reviewing, as well as your valuable suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is simple and appears to be reasonably effective.\n\nThe analysis of the approach is satisfactory. The authors evaluate different combinations of the token and phrase representations, run a hyper-parameter search for word pair encoding (although the improvement over random phrase extraction is limited), and evaluate performance based on the number of extracted phrases.", "reasons_to_reject": "The encoder input length can double in the worst case. As self-attention has quadratic complexity, this can increase the computational costs significantly, but this is not discussed or evaluated.\n\n[Less critical] It would be preferable to also evaluate the approach with input languages other than English.", "questions_to_authors": "A. In figure 5, why is UMST much better with 0 spans?\n\nB. Why equation (3) exactly instead of e.g. PMI/PPMI?", "paper_topic_and_main_contributions": "This paper proposes an approach to use phrase-level information for neural machine translation. Phrases are extracted with word pair encoding (WPE), a variant of BPE. Phrase representations are then extracted by constructing 3 query vectors with min/max/avg pooling and applying attention over the phrase tokens (Attentive Semantic Fusion (ASF)). The token and phrase representations are then concatenated to form the transformer encoder input.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Wang et al. Translating Phrases in Neural Machine Translation. EMNLP 2017. https://aclanthology.org/D17-1149/"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The description of the approach is possibly too verbose (e.g. \"concatenated vertically, yielding a vector with triple-sized embed_dim but a length of 1\". Information such as \"length of the output vector matches the length of the query vector\" (or its reformulation \"size of attention output is determined by its query vector\") is arguably trivial. Some information could likely be moved to the appendix. \n\nI would present WPE before ASF because it is applied first.\n\nThere is a space missing between Fig./Table and some of the following numbers (e.g. L172, 440, 482, 569).\n\nL490: \"Besides,\" without follow-up\n\nFigure 2: Attntive -> Attentive"}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to use semantics phrase-level information for Neural Machine Translation. The phrases are extracted with Word Pair Encoding (similarly to Byte Pair Encoding) and are integrated using an Attentive Semantic Fusion (ASF) by extending the network input.
The paper is clear and well written. The experiments are sound and the results show the beneficial impact of the method (although sometimes, rather low quantitative improvements are obtained).
In the rebuttal, the authors address most of the reviewers' comments. Their answers are convincing and, if all modifications are made, should lead to an improved version of the paper for the camera ready. This is much appreaciated and strengthen the submission.
Decision: Accept-Findings