Modality-Preserving Phrase-Based Statistical
Machine Translation
Masamichi Ideue, Kazuhide Yamamoto

Masao Utiyama, Eiichiro Sumita

Department of Electrical Engineering
Nagaoka University of Technology
Nagaoka, Japan{ideue,yamamoto}@jnlp.org

Universal Communication Research Institute
NICT
Soraku, Japan{mutiyama,eiichiro.sumita}@nict.go.jp

Abstract—In machine translation (MT), modality errors areoften critical. We propose a phrase-based statistical MT methodthat preserves the modality of input sentences. The method
introduces a feature function that counts the number of phrases
in a sentence that are characteristic words for modalities. This
simple method increases the number of translations that havethe same modality as the input sentences.
Keywords—statistical machine translation; modality;

I. I NTRODUCTION
Outputs of statistical machine translation (SMT) usuallycontain word selection and word order errors. These errors
are often evident to users because the outputs are awkward.
However, there are instances in which users are likely tooverlook modality errors: for example, “I do not like apples”
may be translated as “I like apples.” In this case, the users
would not be able to detect an error. Thus, preserving themodality of input sentences is essential.
We propose a method for preserving the modality of input
sentences in phrase-based SMT (PBSMT). We introduce a
special feature function in the PBSMT model that counts thenumber of characteristic words for modalities. Although weapply our method to Japanese-English PBSMT, it is applicable
to any language pair. Our method increases the number oftranslations that have the same modality as the input sentences.
II. R ELATED S TUDIESPrevious studies in this regard have aimed to preserve
sentence modality in SMT.
Finch et al. [1] divided their training data into question
sentences and other sentences. They trained two models foreach of the two types of the sentences. They also trained athird model with the entire training data. A class-dependent
mixture was utilized to use these three models.
Goh et al. [2] proposed the use of various global features
for discriminative reranking in an SMT framework. Theirframework employs an online large-margin-based training algorithm for structural output support vector machines based on
Margin Infused Relaxed Algorithm. The framework uses theprobabilities of sentence types, such as affirmations, negations,questions, and predication.
Both of the above-mentioned studies improved the translation accuracy; however, neither of the studies discusses

what expressions influence modalities. Our study focuses on
characteristic modality words in negations, affirmations, andquestions.
III. T RANSLATION M ODEL U SING C HARACTERISTICM ODALITY W ORDSFirst, we manually extracted characteristic words for each
modality, which are shown in Table I . We preserved the casedistinctions in these words because capitalized words usuallyoccur at the beginning of question sentences. We used theMoses toolkit [3] to tokenize English sentences.
TABLE IM ANUALLY EXTRACTED CHARACTERISTIC MODALITY WORDS FORE NGLISH .
negationquestion

not?
Could
Can
When
Did

No
WhyIs
Do
Where
WascannotWill
How
Are
Have
May’t
What
Does
Which
Does
Shall

A. Translation Model
The output ê of a translation is determined by the following
log-linear equation in a well-known phrase-based translation
model.
ê = arg maxe∑

λi hi (e, f , c)

(1)

i=0

where f and e are Japanese input and translation hypotheses
respectively, c is the alignment between the phrases in thetwo languages; hi (e, f , c) is a feature function such as thelanguage model and the phrase translation probability; and λis the weight assigned to each feature function.
B. Characteristic Words in the Two Languages
The feature function defined in equation (2) only considers
the output phrases. However, to preserve the modality of theinput sentence, we need to consider both input and outputphrases. For this purpose, we use another feature function
h(e, f ):h(e, f ) =∑fc (e¯i , f¯i )i{1if|C∩C|&|C¯ēEf ∩ CF | ≥ 1fc (ē, f¯) =0 otherwise

(2)
(3)

where f¯i is a phrase that occurs in f . Setting CF as theset of characteristic Japanese modality words and CE as theset of characteristic English modality words, fc (ē, f¯) is 1 if
both ē and f¯ contain characteristic modality words for theirrespective languages.
The modality characteristic words in Japanese were extracted manually as shown in Table II . Japanese sentences
are divided into morphemes using the morphological analyzer
ChaSen1 .
TABLE II
M ANUALLY EXTRACTED CHARACTERISTIC MODALITY WORDS FORJAPANESE .
negationquestionnai?

mase nka .

LLR = sign(ad − bc)LLR0{+1 if z ≥ 0sign(z) =−1 otherwise

(4)
(5)

P r(D|Hdep )(6)
P r(D|Hindep )anbn= a log+ b log+(a + b)(a + c)
(a + b)(b + d)
dncn+ d logc log(a + b)(a + c)
(c + d)(b + d)

LLR0 =

where D is the state of the parameters in Table III . In
equation (6), P r(D|Hindep ) is the probability of observing
the contingency table under the null hypothesis that theoccurrences of the word w in the negative and affirmative
sentences are independent of one another, while Hdep is thecase in which the occurrences are dependent. The words areranked in the decreasing order of their LLR scores.
We extracted the top N words as the characteristic words
for the negation modality. The characteristic words for thequestion were extracted by the same method.
IV. E XPERIMENTS

C. Automatic Extraction of Characteristic Words using theLog-Likelihood Ratio
The characteristic modality words shown in Tables I and II
were manually extracted.
We also used a statistical measure to automatically extract
characteristic words for the modalities. We used the loglikelihood ratio (LLR) [4] because the parallel corpus we chosefor our experiments consisted of sentences from the travel
domain; the LLR has been found to be effective for extracting
characteristic words in this domain [5].
To calculate the LLR scores, we constructed a contingency
table, as shown in Table III
TABLE IIIC ONTINGENCY TABLE USED IN THE EXTRACTION OF CHARACTERISTIC
WORDS FOR THE NEGATION MODALITY.
w=1
w=0negationacaffirmation
bd

In Table III , w denotes the word for which an LLR scorewill be calculated. The value w = 1 signifies that w occurredin the phrase. a is the number of occurrences of w in negative
sentences. while b is the number of occurrences of w inaffirmative sentences. On the other hand, c is the differenceof the total number of negative sentences and a, while d isthe difference of the total number of affirmative sentences andb. Setting, n = a + b + c + d, the LLR score is defined inequation (4).
1 http://chasen-legacy.sourceforge.jp/

A. Identification of Sentence Modality
Three modalities of parallel sentences in the corpus, negation, question, and affirmation were identified through themodalities of the English sentences. The identification rules
were as follows:• Negation: The sentence includes negation words shownin Table I• Question: The sentence ends with a question mark.
• Affirmation: All other cases.
In general, the identification of the modalities of Japanese
sentences is more difficult than that of English sentences.
Hence, we did not use the Japanese sentences to identify themodalities of the parallel sentences.
B. Characteristic Words Extracted by LLR
Using LLR, 30 characteristic modality words were automatically extracted (LLR30). We also experimented with 20,50, and 100 (referred to as LLR20, LLR50, and LLR100,
respectively) characteristic modality words and found thatthe Bilingual Evaluation Understudy (BLEU) scores evaluatedusing these values were similar to those evaluated using 30words. The BLEU scores are shown in Table IV , and thecharacteristic words extracted by LLR30 are shown in TableV.
TABLE IVP RELIMINARY EXPERIMENTAL EVALUATION TO DECIDE THE THRESHOLDRANK FOR LLR.

BLEU

LLR20
32.58

LLR3032.75

LLR50
32.57

LLR10032.61

TABLE V
C HARACTERISTIC WORDS EXTRACTED USING LLR.

Negation
EnglishJapanese’t
mase
notnaidonnDon
hadidnnaka
canamaridoesn
madaisnariwondekihavenjayetiieanysonnanibutwakaracouldn
sonna
know
taku
worry
koto
No
rare
wasn
naku
cannotdesi
Isinpaihasn
wakarashouldn
wakariaren
gawouldnmoanything
yoku
itdesosiri
afraidwake
understandnawhatzannen

Question
English Japanese?
ka
you
doko
What
nani
How?
Dodou
Is
ikura
Can
haWheredono
Coulditadake
May
nanzi
doari
Would
morae’t
desyo
Are
ikaga
does
masu
Will
donna
any
kurai
Which
dotiraWhyyorosii
there
desu
have
dore
thisu
Who
itu
When
osie
Does
mase
Didii
donkakarilongkono
Shalloit
dousiteC. SMT Experiments
We used the Moses toolkit [3] for the SMT experiments.
The weights of the feature functions were determined throughthe minimum error rate training [6].
We used the Basic Travel Expression Corpus (BTEC)
English-Japanese corpus[7]. The training set consisted of
70,000 sentence pairs, and the test set included 500 sentencepairs for each modality. The development set contained 500
sentence pairs for each modality.
We compared three settings. The baseline setting used thedefault setting of the Moses toolkit. The MAN E settingused the manually extracted characteristic English words usedto evaluate equation (2), while MAN EJ used the manually
extracted characteristic English and Japanese words used toevaluate equation (4). LLR30 used the automatically extracted
characteristic English and Japanese words to evaluate equation(4). We did not use automatically extracted words to evaluate
equation (2) because the translation accuracy of MAN EJ was
better than that of MAN E.
D. Manual evaluation
We randomly extracted 90 sentence pairs to test the methods: none of these occurred in the evaluation set for the manualevaluation. The result of the evaluation is shown in Table

TABLE VI
N UMBERS OF MANUALLY EVALUATED SENTENCES WITH EACH SCORE .
Baseline
MAN E
MAN EJ
LLR30

S6066
5560

A57405456

B34384438

C26292928

D93
978888TABLE VIIT RANSLATION ACCURACY FOR EACH MODALITY (“A FF ”, “N EG ” AND“Q UE ” ARE AFFIRMATION , NEGATION AND QUESTION , RESPECTIVELY ).
Baseline
MAN E
MAN EJ
LLR30Aff (135)
86.67
71.11
87.41
87.41

Neg (51)
39.22
80.39
64.71
62.75

Que (84)
90.48
95.24
90.48
95.24

VI and Table VII . The column labels columns “S”, “A”,“B”, “C” and “D” in Table Table VI indicate the grades forthe translation accuracy. The respective grades are defined as
below:• S: Completely perfect and sounds like it was spoken bya native speaker.
• A: While completely grammatically correct, it does notsound like it was spoken by a native speaker.
• B: There are grammatical mistakes, but it is easy tounderstand and contains all of the information from theoriginal sentence.
• C: Contains many grammatical mistakes and is very
difficult to understapppnd.
• D: Either it is completely mistranslated or it is completely
incomprehensible.
Table VII shows the percentage of the outputs that preserved the modality of the input; the numbers in parentheses
column headings indicate the number of sentence-pairs foreach modality.
As seen in Table VI , all the methods have the sametranslation accuracy if we assume that the output grades “S”,“A” and “B” indicate good translations.
E. Accuracy of each modality
As indicated by Table VII , the accuracy of the negationmodality for the baseline is 39.22%, while those of the otherthree methods indicate a marked improvement. An exampleof the improved output achieved by our proposed method isshown below.
• Input: sa-kasu to doubutu en, dotti ni iko u ka . (whichmeans “The circus or the zoo, which shall we go to?”)• Baseline: Let’s go to the circus and, the zoo? (×)• MAN EJ: Which one shall we go to the circus and Zoo?
In this example, the input has the question modality, but
the output produced by the baseline is not a question. On
the other hand, the output produced by MAN EJ has thecharacteristic question word “Which”. In other words, it has
the question modality. As our proposed method of considering
the characteristic words for each modality tends to selectphrases that include characteristic words, the output tend
to include the characteristic words, and thus preserves themodality of the input.
In this case of affirmative sentences, the translation accuracyof MAN E was lower than that of the others. We only added
the features φneg and φque to equation (2), with no suchconsideration for affirmative sentences; hence, our proposed
method tends to select phrases that include characteristic
words for negations or questions if the weight of φneg or
φcharacteristic is non-negative. MAN E, which considers only
characteristic words of English, incorrectly selects phrase pairs
in which the English phrase contains the characteristic words.
On the other hand, MAN EJ is constrained by the requirementthat both the Japanese and English phrases in the phrase
pair include the characteristic word, which prevents incorrect
selection.
The accuracy of LLR30 was better than the baseline accuracy of all modalities. Even when the characteristic words areautomatically extracted, our proposed method can improve theaccuracy of preserving sentence modality in a translation.
F. Examples of Incorrect Translations
In this section, we discuss examples of incorrect translations produced by our proposed method. The following is an
example in which the output of MAN E is a question because
the tag question “isn’t it?” has been added.
• Input: yasasiku utte kudasai ne . (Please go easy.)• MAN E: Please go easy, isn’t it? (×)• MAN EJ: Please go easy.
The expression “isn’t it?” includes a characteristic question
word “?” and a characteristic negation word “’t”, and therefore,a tag question tends to be in the translation. MAN EJ which
both the Japanese and English words in a phrase pair, produceda correct translation.
The following is an example in which the translation
produced by MAN EJ has the negation modality despite thequestion modality of the input:• Input: kyanseru si te mo kamai mase n ka . (May I cancelit?)• Baseline: May I cancel?
• MAN EJ: I don’t mind if you cancel it? (×)
The reason for this is that the input includes the characteristic negative modality words “mase n” and the question word
“ka .”, “mase n” was manually extracted as a phrase with astrong negative characteristic. However, as this example show,
the phrase does not always express this modality.
V. C ONCLUSION
In this paper, we proposed a method that adds a featurefunction to the phrase-based statistical translation model that
identifies characteristic words for modalities, and we compared
our method with a baseline using both manually and automatically extracted words.
Our experimental results demonstrate that our method produces more translations that retain the modality of the input

sentence than the baseline does. In particular, our method improved the translation of sentences with the negation modality.
We also compared the results obtained by the two methods
for extracting characteristic words: manual extraction andautomatic extraction using LLR. We confirmed that automatic
extraction performed the same as or better than manual extraction even though the automatically extracted characteristicwords included noise.
R EFERENCES[1] A. Finch, E. Sumita, and S. Nakamura, “Class-Dependent Modeling forDialog Translation,” IEICE TRANSACTION on Information and Systems,
vol. E92-D, no. 12, pp. 2469-2477, 2009.
[2] C. Goh, T. Watanabe, A. Finch, and E. Sumita, “Discriminative Reranking
for SMT using Various Global Features,” In Proceedings of 4th International Universal Communication Symposium (IUCS 2010), 2010, pp.8–
14.
[3] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico,N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar,A. Constantin, and E. Herbst, “Moses: Open Source Toolkit for Statistical
Machine Translation,” In Proceedings of the 45th Annual Meeting of theAssociation for Computational Linguistics(ACL), 2007, pp. 177–180.
[4] D. Ted, “Accurate Methods for the Statistics of Surprise and Coincidence,” Computational Linguistics, vol. 19, no. 1, pp. 61–74.
[5] K. Chujo, M. Utiyama, and K. Oghigian, “Selecting Level-Specific Kyoto
Tourism Vocabulary Using Statistical Measures,” New Aspects of English
Language Teaching and Learning, pp. 126–138.
[6] F. J. Och, “Minimum Error Rate Training in Statistical Machine Translation,” In Proceedings of the 41st Annual Meeting of the Association for
Computational Linguistics(ACL), 2003, pp. 160–167.
[7] T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto, and S. Yamamoto,“Toward a Broad-Coverage Bilingual Corpus for Speech Translation ofTravel Conversations in the Real World.” In Proceedings of International
Conference on Language Resources and Evaluation, 2002, pp. 147–152.