Japanese to English Machine Translation using Preordering andCompositional Distributed Semantics
Sho Hoshino

Hubert Soyer Yusuke Miyao Akiko Aizawa
National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japan{hoshino,soyer,yusuke,aizawa}@nii.ac.jp

Abstract

inject additional information into the pipeline ofour baseline system.

The pipeline of modern statistical machinetranslation (SMT) systems consists of several stages, presenting interesting opportunities to tune it towards improved performance on distant language pairs like
Japanese and English. We explore modifications to several parts of this pipeline. We
include a preordering method in the preprocessing stage, a neural network basedmodel in the tuning stage and a recurrent
neural network language model in the postprocessing stage.
To our knowledge this is the first work
tightly integrating a neural network basedmodel into the tuning stage of a SMT
system for the Japanese-English language
pair. As a first step in this direction we provide several insights into how this integration should be approached and give rise to
future work in this area.

1• We apply preordering to the input text asa way of compensating for syntactic differences between English and Japanese.
• We insert scores into the translation model ofour baseline system that are computed fromsemantically meaningful distributed vector
representations.
• As postprocessing, we utilize a recurrent neural network language model to re-score the
100 best translation candidates for each output sentence of our system. Being able to handle variable length context, it complements
the n-gram based language model used within
the pipeline.

2 System
We built our baseline system with Moses (Koehn
et al., 2007) as a phrase-based machine translation
system loosely following the setup described by
the WAT 2014 organizers (Nakazawa et al., 2014),
with some modifications.
We will quickly go through every step of ourtraining.

Introduction

Modern models for statistical machine translation
constitute a pipeline of different components. This
pipeline usually involves a preprocessing part, alanguage model, a translation model and a postprocessing part. While this or a similar structure isbasis of most systems and generally agreed upon,a lot of research has been focusing on modifying
and extending the individual components.
Many parts rely on probabilities acquired
through word frequencies in fixed contexts, discarding additional syntactical and semantical information. Problems arising from these strong
assumptions become particularly apparent when
dealing with distant language pairs like Japaneseand English.
We focus exclusively on translating Japanese
sentences to English and take several measures to

1. Tokenization (section 3)
2. Training of compositional distributed vector
representations (section 2.1.2)
3. Preordering (section 2.1.1)
4. Generation of 6-gram language model with
SRILM
5. Training of translation model with MGIZA++
6. Tuning with compositional distributed semantics features
7. Translation of devtest and test set using
Moses decoder
8. Training of RNN LM and reranking of 100
best translation candidates for every sentence
55

Proceedings of the 1st Workshop on Asian Translation (WAT2014), pages 55‒63,
Tokyo, Japan, 4th October 2014.

9. Evaluation of final output through BLEU and
RIBES scores and submission to WAT 2014
human evaluation
2.1

apply the following three rules: Rule 1 transforms
each chunked sentence into a form that is moresuitable for the preordering rules that follow thisstep. Rule 2 reorders entire chunks while Rule 3
reorders the words inside of every chunk resulting
in the final reordered version of the sentence.

Extensions

In the following, we will describe the modifications to our baseline system. They are grouped into
three parts, one part describing the preordering of
the Japanese input text, the second part explaining the integration of a neural network based distributed compositional semantics model and lastly,
the reranking of the translation candidates for eachsentence using a recurrent neural network language model.

Rule 1 (chunking) Given an input sentence withl chunks (input = c1 c2 ... cl ) we
merge all coordinated chunks into one chunk(c1 c2 ... cl → c1 c2 ... cm where m ≤ l). After that, we split punctuations2 from chunks
applying the following rules:
Rule 1-A A chunk cx (1 < x ≤ m) will not
be split when a predecessor chunk cx−1 is anoun.

2.1.1 Preordering Japanese Text
We employ a preordering method for Japanese-toEnglish translation. We preorder the input text inthe preprocessing stage to reduce the difference between the word order of the Japanese input sentence and the word order of the English target sentence.
Parts of a Japanese sentence can be scrambled/shuffled without changing the meaning of
the sentence or making it grammatically incorrect. Therefore, one English sentence can potentially correspond to several shuffled variations of
the same Japanese sentence.
Yoshida et al. (2014) show that normalizing theword order of Japanese sentences can benefit readability.
Taking scrambling into account not only increases readability, it also plays an importantrole in machine translation (Isozaki et al., 2014).
Hayashi et al. (2013) were able to improve theresults of statistical machine translation systems
by generating English determiners in the Japanese
input text and reordering its words. Kudo et al.
(2014) applied preordering and generated zerosubjects to achieve state-of-the-art results on a
web-text corpus.
We employ the preordering rules introduced in(Hoshino et al., 2014) to reduce order ambiguity in
the Japanese input text. This method has achieved
state-of-the-art results on the NTCIR patent corpus.
Following this method, we parse input sentenceswith the Japanese dependency parser KNP to obtain chunked1 dependency and coordination labels
corresponding to the dependency. After that, weRule 1-B Even if Rule 1-A is applied to a chunkcx , the chunk will be split into three new
chunks (words,particle,punctuation) if thischunk ends with a particle3 .
Rule 1-C A chunk cx will always be split intowords and punctuation when it ends with apunctuation.
Rule 2 (inter-chunk reordering) After Rule 1
produced a new chunked sentence, all of then chunks between punctuations (c1 c2 ... cn
where cx (1 ≤ x ≤ n) is not a punctuation) are reordered (c1 c2 ... cn →w1 ... wq−1 ci−1 ... c1 wq cn ... ci+1 ), where a
chunk ci (1 ≤ i ≤ n) contains q words, ending with a particle word (ci = w1 , ..., wq−1 wq
where wq is a particle).
Rule 3 (intra-chunk reordering) Given a chunkcx which has p content words and q function
words (cx = w1 ... wp wp+1 ... wp+q ),we swap the content words and the function words, then reverse the function words(cx=
w1 ... wp wp+1 ... wp+q →wp+q ... wp+1 w1 ... wp ).
Figure 1 illustrates how the introduced preordering rules are applied to an example sentence.
The sentence comprises 6 chunks two of which arelabeled as coordinated by the KNP parser (表1とand 図2は).
Step 1 displays the basic, unaltered source sentence. In the second step the two coordinated2
We only regard Japanese comma “、” or period “。” aspunctuation.
3
We only consider the Japanese topic case marker particles“ga” and “ha”.

1This chunk actually is a bunsetsu, a linguistic unit inJapanese, as used in (Yoshida et al., 2014)56

Figure 1: Illustration of the pre-ordering rules described in 2.1.1 to an example.
chunks are merged into one following Rule 1. Rule
1, Rule 1 A, B and C result in several splits (topic
marker,“.”, “,”) in step 3.
In step 4, Rule 2 causes “represent” and “following” to be switched. Finally, Rule 3 is applied instep 5 and affects only the Japanese text. いる andを, both labeled as function words by the parser.
2.1.2

Compositional Distributed Semantics

Many machine learning algorithms require fixed
length vectors as input. Of the various different ways to map words to vectors, neural network
based models have proven very effective recently.
Vector representations (embeddings) created bythese models have been utilized as features toachieve state-of-the-art results in several different Natural Language Processing tasks (Collobert
et al., 2011; Mikolov et al., 2013; Baroni et al.,2014), giving rise to many new variants. Methods have been developed that are capable of embedding words from different languages into thesame vector space (Zou et al., 2013), there aremodels that can induce representations of wholephrases or sentences instead of only single tokens(Socher et al., 2010; Le and Mikolov, 2014; Blacoe and Lapata, 2012) and there are models implementing a mixture of these two ideas, embedding
phrases from different languages into the same
vector space (Hermann and Blunsom, 2014; Choet al., 2014).
Work has been published about integrating embeddings and neural network models into thestatistical machine translation pipeline for various language combinations, however, we are notaware of any previous work attempting this for theJapanese/English language pair.
As a first step in this direction, we integrate themodel introduced by Hermann et al. (Hermann andBlunsom, 2014) into our baseline system, specifically, we use a slightly modified version of the“BI” model described in the paper.

Figure 2: Illustration of model described in (Hermann and Blunsom, 2014), slightly modified touse mean instead of addition
From this point on we will refer to the model
as embedding model or neural network model.
Learned in an unsupervised way, we hope to indirectly inject context knowledge through the neuralnetwork model into our baseline machine translation system.
Figure 2 illustrates how the model is trained.
The training is centered around a model M thatlooks up a word representation wi for each word
in a given sentence and combines them by wrapping a hyperbolic tangent function (tanh) aroundthe sum of the vectors of each bi-gram and computing the mean of these intermediate results. The
original version described in (Hermann and Blunsom, 2014) applies addition instead of the mean.
However, in preliminary experiments on an information retrieval task we obtained more promising57

Feature Function We created a new feature function that is utilized in the tuning phase of
the log-linear translation model of our baseline. Moses allows us to access the textof each sub-phrase candidate pair evaluatedduring decoding. For each sub-phrase wecompute its corresponding Japanese and English vector representation using our embedding model. We proceed by calculating asimilarity score between the English and theJapanese sub-phrase vector. We chose the angular similarity, a normalized version ([0, 1])of the cosine similarity, as it was superior to
the Euclidean similarity in our previous experiments on an information retrieval task.
Depending on the outcome we create one of
3 Sparse Features. One feature for highly
similar phrases (similarity > 0.5), one forless similar phrases (similarity ≤ 0.5) and
one for out-of-vocabulary cases where noneof the tuples in one or both phrases could
be found. This feature split is necessary tocounteract problems with out-of-vocabularycases. We encountered these cases specifically for phrases containing mostly named entities. Named entities are often not part ofthe pre-trained word representations. Even
though moses would translate these tokens
correctly (often just by transliteration) thescore produced by the neural model would be
not defined due to out-of-vocabulary errors.
Not using this feature split lead to very poor
performance.

results using the mean and therefore decided to exploit this insight.
One training step requires this model to be applied to three sentences: One English sentence, theJapanese sentence aligned to this English sentence
and a Japanese sentence chosen at random from thecorpus. There are two sets of word representations,a Japanese set and an English set, that are initially
initialized at random by sampling values from a
Gaussian centered at 0 with a standard deviation of
0.1. When looking up the vector representations tocompose in the model, we choose the set of representations corresponding to the language that thecurrently handled sentence is in.
During training we then modify the word vectors to minimize the Euclidean distance between
the vectors composed for the sentences of thevalid, sentence aligned Japanese-English sentencepair and maximize the distance between the vectors of the English sentence and the randomly sampled Japanese sentence. By repeating this procedure for many sentence pairs the model learns to
distinguish between valid translation pairs and invalid, randomly sampled pairs. At application time
we exploit this property and compute a normalized
distance between two given phrases or sub-phrases
to judge their similarity.
Because this model only weakly incorporates
order information on the bi-gram level it is applicable to very short phrases and even single tokens
and can potentially even deal with preordered text.
Due to this flexibility we can insert it into almost
any part of the machine translation pipeline.
We trained the model on the 1 million parallel
sentences in ASPEC that feature the highest alignment scores. We omitted the remaining 2 million
sentence pairs completely to avoid introducing errors caused by incorrect alignment. Replacing tokens that occur less than 6 times in the training set
resulted in more stable learning and better performance on a simple information retrieval task. After hyperparameter optimization the vectors in ourembedding model were chosen to be 128 dimensional, the margin for the noise-contrastive objective was set to 1.0 and we regularized the word
representations with a l2-regularization of 10−5 .
For faster training we utilized RMSProp (Tielemanand Hinton, 2012) with a decay of 0.99 and a learning rate of 0.001.
We explored two ways to include the method
into our baseline system.

Phrase Table We computed a score for eachphrase tuple in the phrase table of our system, following (Cho et al., 2014). We iterated
through all phrase combinations present in thephrase table and added an additional similarity score computed by the neural model inthe same way as described above for the feature function case. This does not require extending our baseline system, the system only
needs to incorporate one more feature whenreading the phrase table.
2.1.3 Reranking with the Recurrent Neural
Network Language Model
Neural Networks have been applied successfullyin language modeling tasks over the past years(Mikolov et al., 2011a) and are increasingly often
found in combination with or as an extension tostatistical machine translation systems (Devlin et58

al., 2014; Cho et al., 2014; Zhang et al., 2014; Liuet al., 2014).
We employ the Recurrent Neural Network Language Model Toolkit (RNNLM) (Mikolov et al.,2011b) to rerank the 100 best translation candidates for each sentence and use the best candidates
for our submission.

3

Data

Embeddings

BLEU

RIBES

No
No
No

None (Baseline)
Feature Function
Phrase Table (full)

19.16
18.95
18.82

63.41
63.48
63.23

Yes
Yes
Yes

Feature Function
None (Baseline)
Phrase Table (full)

18.92
18.55
-

61.76
61.44
-

No

Phrase Table (1 col)

14.53

60.59

Table 1: Scores for different ways to include theCompositional Distributed Semantics model, withand without preordering. In the Feature Function
setting, the embedding model was integrated asa feature function into Moses; in the Phrase Table setting we employed the model to produce ascore for each entry in the phrase table of the baseline translation model. (full) means that we usedthe default moses scores plus the newly computed
score, for (1 col) we used only the score without
the Moses default scores.

We use only the provided ASPEC corpus and do
not rely on any external resources. We train our
language model with all 3 million sentences of the
training set. For the translation model we only utilize the top 1 million sentences (regarding alignment scores). For hyperparameter tuning we use
the development split of the corpus. Our preliminary evaluation is conducted on the devtest split,
only the final results specified in the paper were
calculated on the test split.
The default moses tokenization script does not
perform Unicode NFKC normalization. Converting Japanese full width roman characters to half
width characters is crucial to tackle transliteration
cases from Japanese to English. Therefore we apply the following preprocessing steps before executing the moses tokenization script.

matic evaluation website differ due to tokenization
mismatches.
4.1 General Experiments
As listed in Table 1 our baseline achieved a scoreof 19.16 BLEU and 63.41 RIBES. Both of thesescores can be considered low compared to results
acquired e.g. on the Chinese to Japanese task of
WAT 2014 where the phrase-based baseline model
provided by the organizers achieved a BLEU scoreof 27.96. The same baseline system only achieved
18.45 on the Japanese to English task which confirms that the Japanese to English translation task
is highly difficult.
Settings with preordering consistently perform
worse than their counterparts without preordering,even though the employed preordering method has
proven very successful (Hoshino et al., 2014) on
the NTCIR patent corpus before. We suspect themain cause to be the different domain of the text.
The NTCIR workshop revolved around text fromthe patent and legal domain while WAT 2014 isbased on the ASPEC corpus comprising abstracts
from scientific papers. Sentences in patent andlegal text must be phrased in a very concise andunambiguous way and therefore feature a lot ofrecurring phrases. Writers of scientific text havemore freedom to express content in English andcan draw from a larger set of ways to phrase theirideas. This looser structure leads to mismatches
when applying the heuristic preordering rules.
Both ways to integrate the scores computed with

1. Conversion of XML entity names to Unicodecharacters
2. Character normalization by Unicode NFKC
and deletion of double spaces
3. First letter lowercasing in all sentences without an all-uppercase first word
4. Conversion of double quotations to “ and ”
5. Commas, periods, and round brackets wereregarded as tokens
In the final submissions, we applied the Moses detokenization script and converted the first letter of
each sentence to upper case in order to avoid possible biases in the human evaluation.

4

Preordering

Evaluation

We evaluated our results on the test split provided with the ASPEC corpus. To avoid tokenization mismatches we applied the same tokenization method to the test set that we had previously used on the training set and computed
BLEU and RIBES scores with the Travatar4 scoring script. The scores published on the WAT auto4
http://www.phontron.com/travatar/evaluation.html

59the neural network model into the Moses pipeline
perform roughly equally well, falling just a little
short of the baseline but failing to beat it. (seePhrase Table (full) and Feature Function)
Examining phrase pairs and their scores revealed that the scores make sense in general, even
identifying transliterations correctly with a highconfidence. However, the model fails to capture an
important property that prevents it from contributing information to Moses that is not already covered by the default features. It almost entirely neglects syntactic structure. Because of its objective
to only distinguish valid translations from invalid
ones, the model learns to ignore special characters, determiners and other words that do not possess a high discriminative value. The translation
hypotheses of our baseline system, however, include many cases where the recognition of syntax
and special characters is crucial to assign sensible
scores. The values of word vectors of many stop
words, determiners and special characters revealthat most of them are in close proximity to the zero
vector. This vector consisting only of zeros constitutes the neutral element of the mean function as
well as the hyperbolic tangent. Therefore, having
or not having such a token in a sentence will only
marginally change the composed representation.
This issue becomes particularly problematic forpairs of the form
Japanese

English

トークン
トークン
トークン

token
token !
the token the the

Systemaveraged Kendall’s tau

Baseline
Baseline + Preordering

0.29900.3712

Table 2: Preordering Evaluation with Kendall’stau.
the neural model score can substitute the default
Moses scores.
4.2 Preordering
To separate the evaluation of our preordering
method from the machine translation evaluation,
we calculated an intrinsic quality measure specificto preordering.
We apply the procedure previously introducedby (Isozaki et al., 2010b; Hoshino et al., 2014).
In our baseline method we rely on MGiza++ toalign Japanese and English sub-phrases. Without
preordering MGiza++ will perform a lot of nonmonotonic alignments. The goal of preordering
is to reduce the number of these non-monotonicalignments, in the best case leading to exclusively
monotonic alignments. Utilizing Kendall’s tau we
can compare the alignments resulting from input
with and without preordering. The closer this coefficient is to 1.0, the more monotonic are the alignments and the higher is the intrinsic benefit of thepreordering. Table 2 lists the averaged coefficientscomputed with the baseline and the preordered input. It shows that our preordering performs better than the non-preordered system on this intrinsic
measure.
Observations at the previously held NTCIR
workshops indicated that better values in this measure correspond to better machine translation quality in terms of automatic as well as human evaluation. Our results in section 4.1 show that this intuition does not hold for the ASPEC corpus.

All three of these examples will have an almost
identical score since the vectors of “!” and “the”
were learned to be very close to the zero vector.
In an information retrieval context where processed sentences are sensible and sane, this property does not pose a huge problem. However, inthe case of a phrase table or hypotheses in the tuning stage of an SMT system, cases like the one illustrated above occur regularly and should exhibit
not the same but very different scores.
The Phrase Table (1 col) setting achieves 14.53
BLEU without utilizing the default Moses scoresin the phrase table. We were surprised that having
only one score value in the phrase table the system could reach a BLEU value that is not on an
entirely different level than the baseline. We leaveit to future research to investigate to what extend

4.3 RNNLM Reranking
After training our baseline system including theneural network score feature function and obtaining 100 translation candidates for each input sentence of the test set, we apply the Recurrent Neural Network Language Model toolkit describedin (Mikolov et al., 2011b) to re-rank these 100best candidates according to their language modelscore.
Mikolov’s implementation offers a variety ofhyperparameters to tune the neural network training for optimal performance. Specifically, we
60

RIBES

BLEU

hiddendirectPreordering

63.90
63.74
63.72
63.66
63.41
62.06
62.03
61.99
61.98
61.44

18.37
18.40
18.61
18.30
19.16
18.36
18.29
18.34
18.48
18.55

300
300
200
200
200
200
300
300
-

150
100
100
150
100
150
150
100
-

No
No
No
No
No
Yes
Yes
Yes
Yes
Yes

4.4 WAT 2014 Evaluation Board
The BLEU and RIBES scores on the official evaluation board as reported in Table 4 differ largely
from the results presented in Table 1. This is due
to tokenization mismatches.
Since our baseline system differs slightly fromthe system provided by the WAT 2014 organizers
we submitted our baseline as a point of reference
for our own experiments to the human evaluation.
As our second submission to the human evaluation we chose our baseline with preordering. Preordering has proven successful in previous NTCIR
workshops but has failed to improve upon the baseline in WAT 2014.
Table 4 shows that our baseline performs
slightly worse than the baseline of the organizers.
We are primarily interested in the difference between our baseline and our baseline with preordering. The human evaluation confirms that preordering has a negative effect when applied to the ASPEC corpus. According to the WAT 2014 homepage, the scores of the human evaluation are calculated by repeatedly comparing sentences from thesubmissions to their corresponding outputs of thebaseline system provided by the organizers. They
therefore indicate how a submission performs incomparison to the organizer’s baseline. The finegrained results of the human evaluation in WAT
2014 offer a good opportunity to investigate theeffects of our preordering method.
Examples illustrating the effect of the applied
preordering method are shown in Table 5. We only
picked examples featuring clear annotator agreement. -1 indicates that the organizers’ baseline wasassessed as superior to our system, 1 that it wasevaluated to be inferior.
The biggest change in both of the examples shownin Table 5 was induced by the same preorderingrule (Rule 2). Due to the lack of a topic marker inboth source sentences Rule 2 caused the main verb
to be brought to the front of the sentence.
Despite the similarity in the preprocessing step thequality of the translations differs heavily. While
Example 2 was evaluated as superior to the organizers’ baseline, Example 1 was received as justthe opposite.
In example 1, the preordered main verb (“discussed”) was omitted entirely in the output of oursystem, changing the meaning from a discussion toa statement.
On the contrary, the output of our system for ex-

Table 3: Results for training the RNN LM withdifferent hyperparameters applying it to re-rank
the 100 best translation candidates for each output sentence of our SMT model. We varied thenumber of hidden units (hidden), the number of direct connections from input to output layer (direct)
and kept the number of classes for the factored
softmax fixed at 220 (following the recommendation√ in (Mikolov et al., 2011b) with #classes =size of vocabulary) and the number of steps of
backpropagation-through-time fixed at 5. The preordered column indicates whether the preordering
described in (Hoshino et al., 2014) was applied.

Baseline
Baseline + Preordering

BLEU

RIBES

human

17.47
17.01

63.08
61.08

-5.750-14.250

Table 4: Scores from WAT 2014 evaluation board.

adapted the number of hidden units and the number of direct connections from the input to the
output layer. To choose the best settings weevaluated several configurations, the results forsome of these configurations are displayed in Table 3. Reranking the translation candidates with
the RNNLM improves the RIBES score while
BLEU decreases instead. This pattern is consistent over all settings displayed in the table. Further, all settings with preordering perform worse
than their counterparts without preordering. This
goes along with previous observations presented
in Table 1. The RIBES measure was specificallydesigned with distant language pairs like English
and Japanese in mind(Isozaki et al., 2010a). Withan increase of almost 0.5 from the non-preorderedbaseline to the best results with reranking andeven more improvement for the preordered case,we conclude that reranking with an RNNLM improved our translation results significantly.
61

Example 1: Negative Impact of Preordering
human judgment
SourcePreordered Source
Baseline + Preordering
Organizers’
Reference-1 -1 -1
プラスチック以外の非金属製の義肢装具材料について論じた。
論じた ついて に 義肢 装具 材料 の 非 金属 製 の プラスチック 以外 。
The non ‐ metallic materials of a prosthesis apparatus is made of plastics.
The non ‐ made of metal except for the plastic prosthesis apparatus material are discussed.
Artificial limb apparatus nonmental material other than plastic is discussed.

human judgment
SourcePreordered Source
Baseline + Preordering
Organizers’
Reference

111
標記の新しいインライン流量計を開発した。
開発 した を インライン 流量 計 新しい の 標記 。
We have developed a new in-line flowmeter.
The titled new in-line flowmeter was developed.
The titled new in‐line flowmeter was developed.

Example 2: Positive Impact of Preordering

Table 5: Positive and negative effects of preorderingmachine translation should take word order into
account and should be trained towards objectives
that do not neglect syntactic features. The semantic similarity that can be captured by the model described in (Hermann and Blunsom, 2014) appears
to be already sufficiently covered by the default
features of our baseline system for the case of machine translation.
Employing the recursive neural network language model introduced by Mikolov et al.
(Mikolov et al., 2011a) has proven successful forreranking translation candidates and significantly
improved the RIBES score in our experiments.
Recent work (Devlin et al., 2014; Cho et al.,2014; Zhang et al., 2014; Liu et al., 2014) proves,
there is a lot of potential in utilizing neural network
based models in the machine translation pipeline.
With the lessons learned from our work we hopefor successful applications of this combination to
the Japanese-English language pair in the future.

ample 2 shows a very natural structure, following
the word order of the preordered Japanese source
almost exactly.

5

Conclusion and Future WorkIn this work we investigated several modifications
and insertions to the pipeline of our baseline statistical machine translation system.
In the preprocessing phase we preordered theJapanese input text to compensate for order differences in the distant Japanese-English language
pair. In the tuning phase of our decoder we evaluated different ways to include phrase similarity
scores computed utilizing a neural network based
compositional distributed semantics model. In the
postprocessing phase we reranked the translation
candidates for each sentence according to scorescalculated with a recurrent neural network language model.
To our knowledge this is the first attempt to utilize neural network features directly integrated into
the tuning and decoding process of a SMT system
in a Japanese-English translation task. We therefore regard this work a first step towards improving
Japanese to English translation systems by tightly
coupling them with neural networks.
Our experiments show that preordering fails toyield the same improvements in the scientific paper domain that it has previously achieved on text
from the legal/patent domain. We attribute this tothe difference in the rigidness of possible formulations in these domains. The heuristic rules ourpreordering method is based on can better capture
the more rigid sentences in the patent domain while
they are prone to mismatches on the less rigid sentences in the scientific paper domain.
We conclude that neural networks utilized in

References
Marco Baroni, Georgiana Dinu, and Germán Kruszewski.
2014. Don’t count, predict! a systematic comparison ofcontext-counting vs. context-predicting semantic vectors.
In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
William Blacoe and Mirella Lapata. 2012. A comparisonof vector-based representations for semantic composition.
In Proceedings of the 2012 Joint Conference on Empirical
Methods in Natural Language Processing and Computational Natural Language Learning.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
2014. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint
arXiv:1406.1078.
Ronan Collobert, Jason Weston, Léon Bottou, Michael
Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011.
62

Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.

Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukas
Burget, and Jan Cernocky. 2011b. RNNLM – recurrent
neural network language modeling toolkit. In Proceedingsof IEEE Automatic Speech Recognition and Understanding Workshop 2011 Demo Session.

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas
Lamar, Richard Schwartz, and John Makhoul. 2014. Fastand robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual
Meeting of the Association for Computational Linguistics.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In
Advances in Neural Information Processing Systems 26,
pages 3111–3119.

Katsuhiko Hayashi, Katsuhito Sudoh, Hajime Tsukada, Jun
Suzuki, and Masaaki Nagata. 2013. Shift-reduce word
reordering for machine translation. In Proceedings of the2013 Conference on Empirical Methods in Natural Language Processing.

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Sadao Kurohashi, and Eiichiro Sumita. 2014. Overview of the 1stworkshop on asian translation. In Proceedings of the 1st
Workshop on Asian Translation (WAT2014).

Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual
models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics.

Richard Socher, Christopher D Manning, and Andrew Y Ng.
2010. Learning continuous phrase representations andsyntactic parsing with recursive neural networks. In Proceedings of the Advances in Neural Information Processing Systems 23 Deep Learning and Unsupervised Feature
Learning Workshop, pages 1–9.

Sho Hoshino, Yusuke Miyao, Katsuhito Sudoh, and Masaaki
Nagata. 2014. A preordering method robust to parsing errors for japanese-to-english statistical machine translation.
In Proceedings of the 20th Annual Meeting of the Association for Natural Language Processing. (in Japanese).

Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5rmsprop: Divide the gradient by a running average of its
recent magnitude. COURSERA: Neural Networks for Machine Learning, 4.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010a. Automatic evaluationof translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in
Natural Language Processing.

Kazushi Yoshida, Tomohiro Ohno, Yoshihide Kato, andShigeki Matsubara. 2014. Japanese word reordering integrated with dependency parsing. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.

Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada, and Kevin
Duh. 2010b. Head finalization: A simple reorderingrule for sov languages. In Proceedings of the Joint Fifth
Workshop on Statistical Machine Translation and MetricsMATR.

Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing
Zong. 2014. Bilingually-constrained phrase embeddings
for machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.

Hideki Isozaki, Natsume Kouchi, and Tsutomu Hirao. 2014.
Dependency-based automatic enumeration of semanticallyequivalent word orders for evaluating japanese translations. In Proceedings of the Ninth Workshop on Statistical
Machine Translation.

Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D.
Manning. 2013. Bilingual word embeddings for phrasebased machine translation. In Proceedings of the 2013
Conference on Empirical Methods in Natural Language
Processing.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin,
and Evan Herbst. 2007. Moses: Open source toolkit for
statistical machine translation. In Proceedings of the 45th
Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo andPoster Sessions.
Taku Kudo, Hiroshi Ichikawa, and Hideto Kazawa. 2014. A
joint inference of deep case analysis and zero subject generation for japanese-to-english statistical machine translation. In Proceedings of the 52nd Annual Meeting of the
Association for Computational Linguistics.
Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. Journal of Machine
Learning Research, 32:1188–1196.
Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A
recursive recurrent neural network for statistical machinetranslation. In Proceedings of the 52th Annual Meetingon Association for Computational Linguistics. Association
for Computational Linguistics.
Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukàs
Burget, and Jan Cernockỳ. 2011a. Empirical evaluationand combination of advanced language modeling techniques. In Proceedings of the 12th Annual Conferenceof the International Speech Communication Association,
pages 605–608.
63