Semantic Features Based on Word Alignments
for Estimating Quality of Text Simplification
Tomoyuki Kajiwara† and Atsushi Fujita‡†

Tokyo Metropolitan University, Tokyo, Japan
National Institute of Information and Communications Technology, Kyoto, Japan
kajiwara-tomoyuki@ed.tmu.ac.jp, atsushi.fujita@nict.go.jp‡

MetricsBLEU
METEOR
TER
WER

Abstract
This paper examines the usefulness of semantic features based on word alignments
for estimating the quality of text simplification. Specifically, we introduce seventypes of alignment-based features computed on the basis of word embeddings
and paraphrase lexicons. Through an empirical experiment using the QATS dataset(Štajner et al., 2016b), we confirm that
we can achieve the state-of-the-art performance only with these features.

1

Introduction

Text simplification is the task of rewriting complex text into a simpler form while preserving its
meaning. Systems that automatically pursue thistask can potentially be used for assisting reading
comprehension of less language-competent people, such as learners (Petersen and Ostendorf,2007) and children (Belder and Moens, 2010).
Such systems would also improve the performanceof other natural language processing tasks, such as
information extraction (Evans, 2011) and machinetranslation (MT) (Štajner and Popović, 2016).
Similarly to other text-to-text generation tasks,
such as MT and summarization, the outputs of text
simplification systems have been evaluated subjectively by humans (Wubben et al., 2012; Štajneret al., 2014) or automatically by comparing withhandcrafted reference texts (Specia, 2010; Coster
and Kauchak, 2011; Xu et al., 2016). However, theformer is costly and not replicable, and the latterhas achieved only a low correlation with humanevaluation.
On the basis of this backdrop, Quality Estimation (QE) (Specia et al., 2010), i.e., automatic evaluation without reference, has been drawing muchattention in the research community. In the sharedrlength
-0.765-0.6170.741
0.757

rlabel0.245
0.257-0.233-0.230

Table 1: The QATS training data shows that typical MT metrics are strongly biased by the length
difference between original and simple sentences(rlength ), while they are less correlated with themanually-labeled quality (rlabel ).
task on quality assessment for text simplification(QATS),1 two tasks have been addressed (Štajneret al., 2016b). One is to estimate a real-value quality score for given sentence pair, while the otheris to classify given sentence pair into one of thethree classes (good, ok, and bad). In the classification task of the QATS workshop, systems basedon deep neural networks (Paetzold and Specia,
2016a) and MT metrics (Štajner et al., 2016a) haveachieved the best performance. However, deepneural networks are rather unstable because of the
difficulty of training on a limited amount of data;for instance, the QATS dataset offers only 505 sentence pairs for training. MT metrics are incapableof properly capturing deletions that are prevalent
in text simplification (Coster and Kauchak, 2011),
as they are originally designed to gauge semantic equivalence. In fact, as shown in Table 1,
well-known MT metrics are strongly biased by thelength difference between original and simple sentences, even though it is rather unrelated with thequality of text simplification assessed by humans.
In order to properly account for the surfacelevel inequivalency occurring in text simplification, we examine semantic similarity features
based on word embeddings and paraphrase lexicons. Unlike end-to-end training with deep neuralnetworks, we quantify word-level semantic corre1

http://qats2016.github.io/shared.html

spondences using two pre-compiled external resources: (a) word embeddings learned from largescale monolingual data and (b) a large-scale paraphrase lexicon. Using the QATS dataset, we empirically demonstrate that a supervised classifier
trained upon such features achieves good performance in the classification task.

2

Semantic Features Based on Word
Alignments

We bring a total of seven types of featuresthat are proven useful for the similar task, i.e.,
finding corresponding sentence pairs within English Wikipedia and Simple English Wikipedia(Hwang et al., 2015; Kajiwara and Komachi,2016). Specifically, we assume that some of thesefeatures are useful to capture inequivalency between original sentence and its simplified version
introduced during simplification, such as lexical
paraphrases and deletion of words and phrases.
Throughout this section, original sentence andits simplified version are referred to as x and y,
respectively.
2.1 AES: Additive Embeddings Similarity
Given two sentences, x and y, AES between themis computed as follows.

AES(x, y) = cos 

|x|

∑

|y|

⃗xi ,

i=1

∑

⃗yj 

(1)

j=1

AAS (Song and Roth, 2015) averages the cosine similarities between all pairs of words within
given two sentences, x and y, calculated over theirembeddings.

As MAS is asymmetric, we calculate it for eachdirection, i.e., MAS(x, y) and MAS(y, x), unlike
Kajiwara and Komachi (2016) who has averagedthese two values.
2.4 HAS: Hungarian Alignment Similarity
AAS and MAS deal with many-to-many and oneto-many word alignments, respectively. On theother hand, HAS (Song and Roth, 2015) is basedon one-to-one word alignments.
The task of identifying the best one-to-one wordalignments H is regarded as a problem of bipartite
graph matching, where the two sets of vertices respectively comprise words within each sentence x
and y, and the weight of a edge between xi and yj
is given by the cosine similarity calculated overtheir word embeddings. Given H identified using the Hungarian algorithm (Kuhn, 1955), HAS
is computed by averaging the similarities between
embeddings of the aligned pairs of words.
HAS(x, y) =1 ∑cos(⃗xi , ⃗yj )
|H|

(4)

where |H| = min(|x|, |y|), as H contains only
one-to-one word alignments.
2.5 WMD: Word Mover’s Distance
WMD (Kusner et al., 2015) is a special case of
the Earth Mover’s Distance (Rubner et al., 1998),
which solves the transportation problem of words
between two sentences represented by a bipartite
graph.2 Let n be the vocabulary size of the language, WMD is computed as follows.
WMD(x, y) = min

n ∑
n
∑

Auv eud(⃗xu , ⃗yv )u=1 v=1|y|

1 ∑∑cos(⃗xi , ⃗yj )|x||y|(3)i=1(i,j)∈H

2.2 AAS: Average Alignment Similarity

|x|

|x|

1 ∑
MAS(x, y) =max cos(⃗xi , ⃗yj )
j
|x|



where each sentence is vectorized with the sumof the word embeddings of its component words,⃗xi and ⃗yj , assuming the additive compositionality(Mikolov et al., 2013).

AAS(x, y) =noise by considering only the best word alignment
for each word in one sentence as follows.

(2)
subject to :i=1 j=1

n
∑

(5)
Auv =

1
freq(xu , x)
|x|

Auv =

1
freq(yv , y)
|y|

v=1
n
∑

2.3 MAS: Maximum Alignment Similarity
AAS inevitably involves noise, as many wordpairs are semantically irrelevant to each other.
MAS (Song and Roth, 2015) reduces this kind of

u=1
2

Note that the vertices in the graph represent the word
types, unlike the token-based graph for HAS.

where Auv is a nonnegative weight matrix representing the flow from a word xu in x to a word yv
in y, eud(·, ·) the Euclidean distance between twoword embeddings, and freq(·, ·) the frequency of aword in a sentence.
2.6 DWE: Difference of Word Embeddings
We also introduce the difference between sentence
embeddings so as to gauge their differences interms of meaning and simplicity. As the representation of a sentence, we used the averaged wordembeddings (Adi et al., 2017).
|x|

|y|

i=1

j=1

1 ∑
1 ∑
⃗xi −
⃗yj
DWE(x, y) =
|x|
|y|

(6)

2.7 PAS: Paraphrase Alignment Similarity
PAS (Sultan et al., 2014, 2015) is computed basedon lexical paraphrases. This feature has been
proven useful in the semantic textual similarity
task of SemEval-2015 (Agirre et al., 2015).
PA(x, y) + PA(y, x)(7)|x| + |y|{|x|∑
1 ∃j : xi ⇔ yj ∈ yPA(x, y) =0 otherwise
i=1

PAS(x, y) =where xi ⇔ yj holds if and only if the word pair
(xi , yj ) is included in a given paraphrase lexicon.

3

Experiment

The usefulness of the above features was evaluatedthrough an empirical experiment using the QATS
dataset (Štajner et al., 2016b).
3.1 Data
The QATS dataset consists of 505 and 126 sentence pairs for training and test, respectively,
where each pair is evaluated from four different
aspects: Grammaticality, Meaning preservation,
Simplicity, and Overall quality. Evaluations are
given by one of the three classes: good, ok, andbad.
We used two pre-compiled external resources tocompute our features. One is the pre-trained 300dimensional CBOW model3 to compute the features based on word embeddings, while the otheris PPDB 2.0 (Pavlick et al., 2015)4 for PAS.
3

https://code.google.com/archive/p/
word2vec/4
http://paraphrase.org/

3.2 Evaluation Metrics
Each system is evaluated by the three metrics asin the QATS classification task (Štajner et al.,2016b): Accuracy (A), Mean Absolute Error (E)and Weighted F-score (F). To compute Mean Absolute Error, class labels were converted into threeequally distant numeric scores retaining their relation, i.e., good = 1, ok = 0.5, and bad = 0.
3.3 Baseline Systems
As the baseline, we employed four types of systems from the QATS workshop (Štajner et al.,2016b): two typical baselines and two top-ranked
systems. “Majority-class” labels all the sentencepairs with the most frequent class in the training
data. “MT-baseline” combines BLEU (Papineni
et al., 2002), METEOR (Lavie and Denkowski,2009), TER (Snover et al., 2006), and WER (Levenshtein, 1966), using a support vector machine(SVM) classifier.
SimpleNets (Paetzold and Specia, 2016a) has
two different forms of deep neural network architectures: multi-layer perceptron (SimpleNetsMLP) and recurrent neural network (SimpleNetsRNN). SimpleNets-MLP uses seven features ofeach sentence: the number of characters, tokens,and word types, 5-gram language model probabilities estimated on the basis of either SUBTLEX(Brysbaert and New, 2009), SubIMDB (Paetzold and Specia, 2016b), Wikipedia, and Simple
Wikipedia (Kauchak, 2013). SimpleNets-RNN,
which does not require such feature engineering,uses embeddings of word N -grams.
SMH (Štajner et al., 2016a) has two types ofclassifiers: logistic classifier (SMH-IBk/Logistic)
and random forest classifier (SMH-RandForest,
SMH-RandForest-b). Both are trained relying on
the automatic evaluation metrics for MT, such as
BLEU, METEOR, and TER, in combination with
the QE features for MT (Specia et al., 2013).
Instead of reimplementing the above baseline
systems, we excerpted their performance scoresfrom (Štajner et al., 2016b).
3.4 Systems with Proposed Features
We evaluated our proposed features in the supervised classification fashion as previous work.
Specifically, we compared three types of supervised classifiers that had been also used in theabove baseline systems: SVM, MLP, and RandForest. Hyper-parameters of each system were de-System
Majority-classMT-baseline
SimpleNets-MLP
SimpleNets-RNN (N = 2)
SimpleNets-RNN (N = 3)
SMH-IBk/Logistic
SMH-RandForest
SMH-RandForest-b
Best score among the above
Our SVM
Our MLP
Our RandForest
Our SVM w/ MT-baseline
Our MLP w/ MT-baseline
Our RandForest w/ MT-baseline

Grammaticality
A↑E↓
F↑
76.2 18.3 65.9
76.2 18.3 65.9
74.6 17.1 68.8
75.4 18.7 65.5
74.6 19.1 65.1
70.6 19.4 71.6
75.4 17.5 71.8
75.4 18.3 70.0
76.2 17.1 71.8
76.2 18.3 65.9
68.3 24.6 66.9
76.2 18.3 65.9
76.2 18.3 65.9
63.5 26.6 63.8
76.2 18.3 65.9

A↑
57.9
66.7
65.9
57.9
51.6
69.1
65.9
61.9
69.1
65.1
59.5
66.7
66.7
64.3
61.9

MeaningE↓F↑
29.0 42.5
20.2 62.7
21.0 63.5
27.4 51.3
28.2 46.6
20.2 68.1
20.6 64.4
23.8 59.7
20.2 68.1
22.2 58.3
25.4 56.4
23.0 63.2
21.0 63.7
21.4 62.7
24.6 57.6

A↑
55.6
50.8
53.2
50.0
52.4
50.0
52.4
57.1
57.1
57.1
59.5
63.5
57.1
52.4
62.7

Simplicity
E↓F↑
29.4 39.7
26.2 48.3
27.0 49.8
27.0 47.5
25.0 50.0
28.2 51.1
27.8 53.0
25.4 56.4
25.0 56.4
27.8 43.9
23.4 58.2
21.8 59.8
27.0 46.9
26.2 53.2
22.6 56.1

A↑
43.7
38.1
38.1
52.4
47.6
47.6
44.4
48.4
52.4
57.9
52.4
51.6
47.6
46.0
46.0

Overall
E↓
28.2
41.7
32.5
25.8
27.8
28.2
31.8
29.0
25.8
23.4
25.8
26.6
29.0
31.8
29.0

F↑
26.5
37.5
33.7
46.1
40.8
47.5
44.5
48.6
48.6
57.7
51.9
48.3
46.8
45.5
43.6

Table 2: Results on QATS classification task. The best scores of each metric are highlighted in bold.
Scores other than ours are excerpted from Štajner et al. (2016b).
Feature set
ALL-AES-MAS(original, simple)-MAS(simple, original)
-PAS
-DWE
-WMD
-AAS
-HAS

C
1.0
1.0
0.1
1.0
0.1
0.01
0.01
0.1
0.01

γ
0.1
0.1
0.1
0.1
0.1
1.0
0.1
0.1
0.01

Grammaticality
76.2
76.2
76.2
76.2
76.2
76.2
76.2
76.2
76.2

Meaning
65.1
65.1
57.9
64.3
57.9
57.9
57.9
57.9
57.9

Simplicity
57.1
57.1
55.6
57.1
55.6
55.6
55.6
55.6
55.6

Overall
57.9
57.1
56.4
54.8
53.2
51.6
46.8
45.2
35.7

Table 3: Ablation analysis on accuracy. Features are in descending order of overall accuracy.
termined through 5-fold cross validation using thetraining data, regarding accuracy in terms of overall quality as the objective.
For the SVM classifier, we used the RBF kernel.
The trinary classification was realized by meansof the one-versus-the-rest strategy. For a given setof features, we examined all the combinations ofhyper-parameters among C ∈ {0.01, 0.1, 1.0} andγ ∈ {0.01, 0.1, 1.0}; for the full set of features,
C = 1.0 and γ = 0.1 were chosen.
As for the MLP classifier, among 1 to 3 layers with all the combinations of dimensionality
among {100, 200, 300, 400, 500} and “ReLu” forthe activation function among {Logistic, tanh,
ReLu}, the 2-layer one with 200 × 200 dimensionality was optimal. We used Adam (Kingma
and Ba, 2015) as the optimizer.
For the RandForest classifier, we examined all
the combinations of the following three hyperparameters: {10, 50, 100, 500, 1000} for number of trees, {5, 10, 15, 20, ∞} for the maximum
depth of each tree, and {1, 5, 10, 15, 20} for theminimum number of samples at leaves. The optimal combination for the full set of features was(500, 15, 1).

3.5 Results
Experimental results are shown in Table 2. The
SVM classifier based on our features greatly outperformed the state-of-the-art methods in terms ofoverall quality. The RandForest classifier somehow achieved the best simplicity scores ever, eventhough we had optimized the system with respect
to the accuracy of overall quality. As we expected, MLP did not beat the other two classifiers, presumably due to the scarcity of the training data. The bottom three rows reveal that theperformance in terms of overall quality was deteriorated when MT-baseline features were incorporated on top of our feature set. This suggeststhat word embeddings are superior to surface-levelprocessing in finding corresponding words within
sentence pairs.
Focusing on the overall quality, we conducted
an ablation analysis of the SVM classifier. The
analysis revealed, as shown in Table 3, that HAS,
AAS, and WMD were the most important features. This can be explained by the role of wordalignments during the computation. Since MT
metrics, such as BLEU, rely only on surfacelevel matches, they are insensitive to meaning-Original
SimpleHungarian
Alignment

While historians concur that the result itself was not manipulated, the voting process was neither free nor secret.
Most historians agree that the result was
not fixed, but the voting process was neither free nor secret.
(while, but), (concur, agree),(itself, most), (manipulated, fixed),
and identical word pairs.

Table 4: An example of word alignment. Differences between the original and simplified versions
are presented in bold. This is a sentence pair fromgood class on overall quality. HAS using wordlevel similarity reaches 0.85, while BLEU is 0.54.
FeatureAES
AAS
MAS(original, simple)
MAS(simple, original)
HAS
WMD
PAS

rlength
-0.661-0.335-0.817
0.092
0.061
0.788
-0.120rlabel
0.185
0.318
0.226
-0.090-0.050-0.215-0.039

Table 5: Correlation between each feature and thedifference of sentence length and the manuallylabeled quality. Note that DWE cannot be included, as it is not a scalar value but the differential
vector between original and simplified sentences.
preserving rewritings from original sentence tosimple one. On the other hand, as exemplified
in Table 4, HAS and some other features can detect the linkages between complex words and theirsimpler counterparts. As a result of properly capturing the alignments between such lexical paraphrases, our system successfully classified thissentence into good in terms of overall quality.
We expected that AAS could yield noise, as it
involves irrelevant pairs of words, but in fact, it
contributed to the QATS task. We speculate that ithelps to evaluate the appropriateness of substituting a word to other one considering the semantic
matching with the given context, as in lexical simplification (Biran et al., 2011) and lexical substitution (Melamud et al., 2015; Roller and Erk, 2016;
Apidianaki, 2016).
The contribution of WMD was expected as it
was proven effective in the sentence alignment
task of English Wikipedia and Simple English
Wikipedia (Kajiwara and Komachi, 2016).
Table 5 shows that some of our semantic similarity features are also strongly biased by thelength difference between original and simple sentences, as MT metrics (cf. Table 1). Nonetheless,

HAS was not biased by the length difference almost at all, and AAS and WMD highly correlatedwith the manually-labeled quality.

4 Conclusions
We presented seven types of semantic similarity
features based on word alignments for quality estimation of text simplification. Unlike existing MT
metrics, our features can flexibly deal with wordalignments, taking deletions and paraphrases into
account. Our SVM classifier based on these features achieved the best performance on the QATS
dataset.

Acknowledgments
This work was carried out when the first author
was taking up an internship at NICT, Japan. We
are deeply grateful to the anonymous reviewers fortheir insightful comments and suggestions. This
work was conducted under the program “Promotion of Global Communications Plan: Research,
Development, and Social Demonstration of Multilingual Speech Translation Technology” of the
Ministry of Internal Affairs and Communications(MIC), Japan.

References
Yosshi Adi, Einat Kermany, Yonatan Belinkov, Ofer
Lavi, and Yoav Goldberg. 2017. Fine-grained
Analysis of Sentence Embeddings Using Auxiliary
Prediction Tasks. In International Conference on
Learning Representations.
Eneko Agirre, Carmen Banea, Claire Cardie, DanielCer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei
Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada
Mihalcea, German Rigau, Larraitz Uria, and Janyce
Wiebe. 2015. SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability. In Proceedings of the 9th International
Workshop on Semantic Evaluation. pages 252–263.
Marianna Apidianaki. 2016. Vector-space models for
PPDB paraphrase ranking in context. In Proceedings of the 2016 Conference on Empirical Methods
in Natural Language Processing. pages 2028–2034.
Jan De Belder and Marie-Francine Moens. 2010. Text
Simplification for Children. In Proceedings of
the SIGIR Workshop on Accessible Search Systems.
pages 19–26.
Or Biran, Samuel Brody, and Noemie Elhadad. 2011.
Putting it Simply: a Context-Aware Approach to
Lexical Simplification. In Proceedings of the 49th

Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.
pages 496–501.
Marc Brysbaert and Boris New. 2009. Moving beyond
Kučera and Francis: A Critical Evaluation of Current Word Frequency Norms and the Introductionof a New and Improved Word Frequency Measure
for American English. Behavior Research Methods
41(4):977–990.
William Coster and David Kauchak. 2011. Simple English Wikipedia: A New Text Simplification Task.
In Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics: Human
Language Technologies. pages 665–669.
Richard J. Evans. 2011. Comparing methods forthe syntactic simplification of sentences in information extraction. Literary and Linguistic Computing
26(4):371–388.
William Hwang, Hannaneh Hajishirzi, Mari Ostendorf,
and Wei Wu. 2015. Aligning Sentences from Standard Wikipedia to Simple Wikipedia. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies. pages
211–217.
Tomoyuki Kajiwara and Mamoru Komachi. 2016.
Building a Monolingual Parallel Corpus for Text
Simplification Using Sentence Similarity Based on
Alignment between Word Embeddings. In Proceedings of the 26th International Conference on Computational Linguistics. pages 1147–1158.
David Kauchak. 2013. Improving Text Simplification Language Modeling Using Unsimplified Text
Data. In Proceedings of the 51st Annual Meetingof the Association for Computational Linguistics.
pages 1537–1546.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Harold W. Kuhn. 1955. The Hungarian Method for
the Assignment Problem. Naval Research Logistics
Quarterly 2:83–97.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian
Weinberger. 2015. From Word Embeddings To Document Distances. In Proceedings of The 32nd International Conference on Machine Learning. pages
957–966.
Alon Lavie and Michael Denkowski. 2009. The METEOR Metric for Automatic Evaluation of MachineTranslation. Machine Translation 23(2-3):105–115.
Vladimir Iosifovich Levenshtein. 1966. Binary Codes
Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady 10(8):707–710.

Oren Melamud, Omer Levy, and Ido Dagan. 2015. A
Simple Word Embedding Model for Lexical Substitution. In Proceedings of the 1st Workshop on Vector
Space Modeling for Natural Language Processing.
pages 1–7.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Advances in Neural Information Processing
Systems. pages 3111–3119.
Gustavo H. Paetzold and Lucia Specia. 2016a. SimpleNets: Evaluating Simplifiers with ResourceLight Neural Networks. In LREC 2016 Workshop& Shared Task on Quality Assessment for Text Simplification. pages 42–46.
Gustavo H. Paetzold and Lucia Specia. 2016b. Unsupervised Lexical Simplification for Non-Native
Speakers. In Proceedings of the Thirtieth AAAI
Conference on Artificial Intelligence. pages 3761–
3767.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: a Method for Automatic
Evaluation of Machine Translation. In Proceedingsof 40th Annual Meeting of the Association for Computational Linguistics. pages 311–318.
Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch,
Benjamin Van Durme, and Chris Callison-Burch.
2015. PPDB 2.0: Better paraphrase ranking, finegrained entailment relations, word embeddings, andstyle classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing. pages 425–
430.
Sarah E. Petersen and Mari Ostendorf. 2007. Text Simplification for Language Learners: A Corpus Analysis. In Proceedings of the Speech and Language
Technology in Education Workshop. pages 69–72.
Stephen Roller and Katrin Erk. 2016. PIC a Different
Word: A Simple Model for Lexical Substitution inContext. In Proceedings of the 2016 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies. pages 1121–1126.
Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas.
1998. A Metric for Distributions with Applications
to Image Databases. In Proceedings of the Sixth International Conference on Computer Vision. pages
59–66.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study ofTranslation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the
Association for Machine Translation in the Americas. pages 1–9.

Yangqiu Song and Dan Roth. 2015. Unsupervised
Sparse Vector Densification for Short Text Similarity. In Proceedings of the 2015 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies. pages 1275–1280.
Lucia Specia. 2010. Translating from Complex to Simplified Sentences. In Proceedings of the 9th International Conference on Computational Processing of
the Portuguese Language. pages 30–39.
Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010.
Machine Translation Evaluation versus Quality Estimation. Machine Translation 24(1):39–50.
Lucia Specia, Kashif Shah, Jose G.C. de Souza, andTrevor Cohn. 2013. QuEst - A translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational
Linguistics. pages 79–84.
Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to Basics for Monolingual Alignment: Exploiting Word Similarity and Contextual
Evidence. Transactions of the Association for Computational Linguistics 2:219–230.
Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2015. DLS@CU: Sentence Similarity fromWord Alignment and Semantic Vector Composition.
In Proceedings of the 9th International Workshop on
Semantic Evaluation. pages 148–153.
Sanja Štajner, Ruslan Mitkov, and Horacio Saggion.
2014. One Step Closer to Automatic Evaluationof Text Simplification Systems. In Proceedings ofthe 3rd Workshop on Predicting and Improving Text
Readability for Target Reader Populations. pages 1–
10.
Sanja Štajner and Maja Popović. 2016. Can Text Simplification Help Machine Translation? Baltic Journal of Modern Computing 4(2):230–242.
Sanja Štajner, Maja Popović, and Hanna Béchara.
2016a. Quality Estimation for Text Simplification.
In LREC 2016 Workshop & Shared Task on Quality
Assessment for Text Simplification. pages 15–21.
Sanja Štajner, Maja Popović, Horacio Saggion, Lucia Specia, and Mark Fishel. 2016b. Shared Task
on Quality Assessment for Text Simplification. In
LREC 2016 Workshop & Shared Task on Quality Assessment for Text Simplification. pages 22–31.
Sander Wubben, Antal van den Bosch, and Emiel
Krahmer. 2012. Sentence Simplification by Monolingual Machine Translation. In Proceedings of the50th Annual Meeting of the Association for Computational Linguistics. pages 1015–1024.
Wei Xu, Courtney Napoles, Ellie Pavlick, QuanzeChen, and Chris Callison-Burch. 2016. Optimizing
Statistical Machine Translation for Text Simplification. Transactions of the Association for Computational Linguistics 4:401–415.