Incorporating Noisy Length Constraints into Transformer
with Length-aware Positional Encodings

Yui Oka, Katsuki Chousa, Katsuhito Sudoh and Satoshi Nakamura (at Nara Institute of Science and Technology, NAIST)
Quick Summary

Ø NMT model sometimes made too short sentences, which called under-translation.
Ø We control the output length using LDPE/LRPE in NMT.
Ø We inject noise to LDPE/LRPE in training, for the various output lengths.
Ø We use BERT as prediction model of the output length.
Ø In short sentences, our approach improved translation accuracy.

Length-aware PE [Takase et al. (2019)]
PE : The sinusoidal positional encoding in Transformer [Vaswani et al. (2017)]

234
234
$%('(),+,) = sin()
$%
=cos()
('(),+,9:)
10000+,/810000+,/8LRPE
LDPE
Length-Ratio Positional Encoding
Length-Difference Positional
con-siders the remaining distance
Encoding considers the ratio ofto the terminal position
the remaining length to the final
position
?@A − 234
234
=>$%('(),+,) = sin()
=C$%('(),+,) = sin( +,/8 )
+,/810000
?@A234
?@A − 234
=C$%('(),+,9:) = cos( +,/8 )
=>$%('(),+,9:) = cos()
?@A10000+,/8ResultBLEU
4039383736353433323130
29

ü Random noise injection to LRPE/LDPE length constraints, as follows;

LRPE(our)

?@A + A3E4@ − 234
=>$%('(),+,) = sin()
+,/8
10000
?@A + A3E4@ − 234
=>$%('(),+,9:) = cos()
+,/8
10000

LDPE(our)

234
=C$%('(),+,) = sin()
+,/8(?@A + A3E4@)
234
=C$%('(),+,9:) = cos()
+,/8(?@A + A3E4@)

We randomly chose an integer from two pattern, [-2,-1,0,1,2] and [-4, -3, -2, -1, 0, 1, 2, 3, 4].
ü BERT-based output length prediction(BERT_pred)
In inference, we need a reasonable length estimate. We used the [CLS] vector in the last layer of
the BERT encoder to predict the output length through an output layer as a regression problem.

Setup

• Our task is English-to-Japanese translation, we used ASPEC.
• Hyperparameter setting is from OPENNMT-py FAQ.
• Evaluation: BLEU, LR(Length-ratio), VAR(variance between reference and output)
• Comparison model: BERT_pred(our approach), src_len(using the source sentence
length in inference, [Lakew et al. (2017)]), ref_len(using the reference length, this isgold score)

Model
Transformer

Input length in inference
29.51(0.916)
BERT_pred
src_len
ref_len
LDPE(no noise) 19.91(0.938) 71.83(1.226)
0(1.000)
LRPE(no noise) 21.71(0.929) 59.65(1.198) 1.497(0.987)
LDPE[-2,2]
20.11(0.927) 55.85(1.162) 2.411(0.972)
LDPE[-4,4]
21.15(0.921) 48.25(1.091) 6.429(0.951)
LRPE[-2,2]23.87(0.911) 35.18(1.075) 7.502(0.943)
LRPE[-4,4]
25.12(0.911) 28.06(1.015) 12.506(0.931)
LDPE

LRPE

BERT_pred

LDPE[-2,2] LDPE[-4,4] LRPE[-2,2] LRPE[-4,4]

src_len

ref_len

Transformer

Analysis
Accuracy of the length prediction using BERT

51494745
43
4139373533312927

ASPEC dataset analysis
AVG Error VAR
Train 7.52
99.43
Dev
7.06
97.00
Test
6.55
72.45

AVG Error VAR
Corrpredicted-ref
3.00
19.92
0.93TestØ !"# gets the target length of train set in training, and !"# gets the fixed length ininference. LRPE and LDPE apply to only the decoder.

Approach

VAR(LR)

BLEU in length datasets(BERT_pred)

1-10 tokens

Transformer

11-20 tokens

LDPE(no noise)

LRPE(no noise)

21-40 tokens

LDPE[-2,2]

LDPE[-4,4]

Corr
0.86
0.84
0.90

41-80 tokens

LRPE[-2,2]

LRPE[-4,4]

Output example
Source

▁The ▁image ▁( I ) ▁was ▁formed ▁in ▁laser ▁pulse ▁irradiation , ▁and ▁showed ▁the▁stability ▁over ▁one ▁year ▁at ▁room ▁temperature .

Reference▁   

Baseline

▁ILDPE [-2, 2]

▁   

$ ( I ) " , 

  

#"  , 

1 &

$ ( I ) " ,  &

%!  %

 &

%

