Semantic Features Based on Word Alignments for Estimating Quality of Text Simpliﬁcation

Tomoyuki Kajiwara

Atsushi Fujita

Tokyo Metropolitan University

National Institute of Information and Communications Technologykajiwara-tomoyuki@ed.tmu.ac.jp

atsushi.fujita@nict.go.jp

Motivation

Quality Estimation for Text Simpliﬁcation
Data
Training: 505 sentence pairs
Test: 126 sentence pairs
Four diﬀerent evaluation criteria
Grammatically
Meaning preservation
Simplicity
Overall quality
3-class judgments for each criterion{good, ok, bad}
Evaluation metrics
A: Accuracy
E: Mean Absolute Error
F: Weighted F-score
Best systems in QATS workshop
SimpleNets: neural networks
SMH: MT metrics
http://qats2016.github.io/

Neural networks are rather unstable because of the diﬃculty oftraining on a limited amount of data.
MT metrics are incapable of properly capturing deletions andparaphrases that are prevalent in text simpliﬁcation.
→ In order to properly account for the surface-level inequivalency
occurring in text simpliﬁcation, we examine semantic similarity
features based on word embeddings and paraphrase lexicons.

Semantic Features Based on Word Alignments
1. Additive Embeddings Similarity

5. Word Moverʼs Distance

2. Average Alignment Similarity

6. Diﬀerence of Word Embeddings

3. Maximum Alignment Similarity

7. Paraphrase Alignment Similarity

4. Hungarian Alignment Similarity

Evaluation using QATS dataset
Classiﬁers based on our features greatly outperformed the state-of-the-art methods
in terms of Simplicity (Random Forest Classiﬁer) and Overall quality (SVM Classiﬁer).
MT-baseline features do not help ours further.
→ Word embeddings are superior to surface-level processing in ﬁnding corresponding words.
GrammaticalityMeaning
Simplicity
OverallSystemA↑
E↓
F↑
A↑
E↓
F↑
A↑
E↓
F↑
A↑
E↓
F↑
Majority-class76.2 18.3 65.9
57.9 29.0 42.5 55.6 29.4 39.7 43.7 28.2 26.5
Best score on QATS-2016 (Štajner+ 2016) 76.2 17.1 71.8 69.1 20.2 68.1 57.1 25.0 56.4 52.4 25.8 48.6
SVM Classiﬁers
MT-baseline: BLEU, METEOR, TER, WER
MT-baseline
76.2 18.3 65.9 66.7 20.2 62.7 50.8 26.2 48.3 38.1 41.7 37.5
Our SVM
76.2 18.3 65.9
65.1 22.2 58.3 57.1 27.8 43.9 57.9 23.4 57.7
Our SVM w/ MT-baseline
76.2 18.3 65.9 66.7 21.0 63.7 57.1 27.0 46.9 47.6 29.0 46.8
Neural Network Classiﬁers
SimpleNets-MLP: multi-layer perceptron based on language model features
SimpleNets-MLP (Paetzold and Specia, 2016) 74.6 17.1 68.8 65.9 21.0 63.5 53.2 27.0 49.8 38.1 32.5 33.7
Our MLP
68.3 24.6 66.9
59.5 25.4 56.4 59.5 23.4 58.2 52.4 25.8 51.9
Our MLP w/ MT-baseline
63.5 26.6 63.8
64.3 21.4 62.7 52.4 26.2 53.2 46.0 31.8 45.5
Random Forest Classiﬁers
SMH: based on automatic evaluation metrics and QE features for MT
SMH-RandForest (Štajner+ 2016)75.4 17.5 71.8 65.9 20.6 64.4 52.4 27.8 53.0 44.4 31.8 44.5
Our RandForest
76.2 18.3 65.9 66.7 23.0 63.2 63.5 21.8 59.8 51.6 26.6 48.3
Our RandForest w/ MT-baseline
76.2 18.3 65.9
61.9 24.6 57.6 62.7 22.6 56.1 46.0 29.0 43.6
Ablation on Accuracy
ALL-AES-MAS (Orig, Simp)-MAS (Simp, Orig)
-PAS
-DWE
-WMD
-AAS
-HASG
76.2
76.2
76.2
76.2
76.2
76.276.2
76.2
76.2

M
65.1
65.1
57.9
64.3
57.9
57.9
57.9
57.9
57.9

S
57.1
57.1
55.6
57.1
55.6
55.6
55.6
55.6
55.6

O
57.9
57.1
56.4
54.8
53.2
51.6
46.8
45.2
35.7

Correlation
BLEU
METEOR
WMD
AAS
HAS

length
-0.765-0.6170.788-0.335
0.061

label
0.245
0.257
-0.2150.318-0.050

HAS was not biased by the length diﬀerence
almost at all, and AAS and highly correlatedwith the manually-labeled quality.

Example: A sentence pair judged “good” in terms of overall quality. HAS reaches 0.85, while BLEU is 0.54.
Original:
Simple:While historians concur that the result itself was not manipulated, the voting process was neither free nor secret.
Most historians agree that the result was not ﬁxed, but the voting process was neither free nor secret.

Hungarian Alignment