Discriminative Feature-Rich Modeling for Syntax-Based Machine Translation

State-of-the-art statistical machine translation systems are most frequently built on phrasebased (Koehn et al., 2003) or hierarchical translation models (Chiang, 2005). In addition, a wide variety of models exploiting syntactic annotation on either the source or target side (or both) have recently been developed and also give state-of-the-art performance (Galley et al., 2006; Zollmann and Venugopal, 2006; Huang et al., 2006; Shen et al., 2008; Liu et al., 2009; DeNeefe and Knight, 2009). It is unclear which of these approaches works best in general or how we can combine their virtues in a single system. Furthermore, the community’s emphasis on inventing diverse structural approaches to translation has shifted the focus away from feature-rich modeling as it has been applied to other problems in NLP. For example, adding rich non-local features that violate structural independence assumptions of traditional models has led to state-of-the-art performance in statistical parsing (Huang, 2008b; Martins et al., 2009), but long-distance features have received scant attention from machine translation researchers. In this thesis, we explore feature-rich models for machine translation that make use of nonlocal features and bring together features from multiple formalisms into a single model. To achieve this goal, we make contributions to modeling, inference, and training for statistical machine translation. We describe a novel translation modeling framework based on quasi-synchronous grammar (QG; Smith and Eisner, 2006b). QG is flexible enough to generalize any synchronous formalism, providing an ideal testbed for comparison and integration of feature sets. Since QG does not require the source and target trees to be isomorphic, it permits simulation of the majority of approaches to machine translation in the literature. The modeling innovations we propose rely on efficient algorithms for approximate inference. The models we consider—log-linear models over lattices and hypergraphs—are not natural to encode using the machinery of graphical models, so we instead propose algorithms that extend or exploit well-understood dynamic programming algorithms for these models. We describe a technique called cube summing (Gimpel and Smith, 2009a) that provides a way to extend dynamic programming algorithms to compute expectations for non-local features. To this end, we also propose importance sampling algorithms as an alternative with better-understood theoretical properties. For training translation models, we present two novel objective functions that can be optimized with gradient methods: softmax-margin and the Jensen risk bound (JRB) (Gimpel and Smith, 2010a). Unlike the widely-used technique known as minimum error rate training (MERT; Och, 2003), our objectives can handle hidden variables, support regularization, and scale well to large numbers of features. Softmax-margin and JRB training are also more efficient and easier to implement than minimizing risk (Li and Eisner, 2009). We integrate these developments in a novel machine translation system that we use to conduct controlled experiments in comparing and combining feature sets from many different translation models. Preliminary results have been obtained for a small-scale German-English translation task and show promising trends for combining features from phrase-based and dependency-based syntactic translation models using approximate inference (Gimpel and Smith, 2009b).