Improved Online Learning and Modeling for Feature-Rich Discriminative Machine Translation

Title of dissertation: IMPROVED ONLINE LEARNING AND MODELING FOR FEATURE-RICH DISCRIMINATIVE MACHINE TRANSLATION Vladimir Eidelman, Doctor of Philosophy, 2013 Dissertation directed by: Professor Philip Resnik Department of Linguistics and Institute for Advanced Computer Studies Machine translation represents one of the core tasks in natural language processing: performing an automatic analysis of input text to produce a structured output. Most modern statistical machine translation (SMT) systems learn how to translate by constructing a discriminative model based on statistics from the data. While this allows the introduction of feature functions representing important attributes of the translation process, the associated model parameters must be learned from data. A growing number of methods for discriminative training in SMT have been proposed, but most suffer from limitations that hinder their utility for training feature-rich models on large amounts of data. In this thesis, we present novel models and learning algorithms that address this issue by tackling three core problems for discriminative training: what to optimize, how to optimize, and how to represent the input. The first issue amounts to selecting an appropriate objective function, the second to properly searching for parameters that optimize that objective, and the third to extracting informative features of the data. In addressing these issues, we develop fast learning algorithms that are both suitable for large-scale SMT training and capable of generalization in high-dimensional feature spaces. The algorithms are developed in an online margin-based framework. Their online nature facilitates easily scaling to perform discriminative training on large training sets, and maximum margin optimization allows efficient feature-rich learning, permitting exploration into many new types of sparse features for translation. While online margin-based methods are firmly established in the machine learning community, their adaptation to machine translation is not straightforward. Thus, the first problem we address is what to optimize when learning for SMT, which involves loss-augmented inference over latent variables. To this end, we define a family of objective functions for large-margin learning in SMT and investigate their optimization performance in standard and high-dimensional feature spaces. We also show that this approach shows promise not just with respect to the size of the feature space, but with respect to the size of the tuning data. After establishing what to optimize, the second problem we focus on is how to improve learning in the feature-rich space. While the goal of all learning methods is to produce a model from a limited number of training instances that generalizes well to unseen data, this becomes increasingly difficult as the feature dimension grows. Following recent developments in machine learning that show that generalization ability can be improved by incorporating higher order information into the optimization, we develop an online gradient-based algorithm based on Relative Margin Machines that improves upon large-margin learning by considering and bounding the spread of the data while maximizing the margin. By utilizing the learning regimes developed thus far, we are able to focus on the third problem and introduce new features specifically targeting generalization to new domains. While domain adaptation has typically been done with manually defined domains and corpora, we employ topic models to perform unsupervised domain induction, and introduce translation model adaptation features based on probabilistic domain membership. As a final question, we look at how to take advantage of the latent derivation structure for optimization. In current models of SMT, there is an exponential number of derivations by which the same translation output can be produced. However, only the final output is observed, thereby necessitating the treatment of these paths as a latent variable. The standard practice is to sidestep this ambiguity by treating each derivation as an individual translation, and performing training and inference toward a single way of producing the output. While addressing the learning problem above, we were still firmly situated in that standard regime. In the final part of the thesis, we revisit and present a more thorough approach for what to optimize, defining a framework for latent variable models which explicitly takes advantage of all derivations in both learning and inference. We present a novel loss function for large-margin learning in that setting, and develop a suitable optimization algorithm for training an SMT system with this objective. IMPROVED ONLINE LEARNING AND MODELING FOR FEATURE-RICH DISCRIMINATIVE MACHINE TRANSLATION