Filter Corpus { syntax-st , syntax-ch , moses } Word Alignment Stanford Parser Build Syntactic Translation Model Minimum Error Rate Training Decode Sentences Build Language Model Parallel Corpus Target Language Corpus Moses Phrase Table Training syntax moses

Construction of machine translation systems has evolved into a multi-stage workflow involving many complicated dependencies. Many decoder distributions have addressed this by including monolithic training scripts – train-factored-model.pl for Moses and mr_runmer. pl for SAMT. However, such scripts can be tricky to modify for novel experiments and typically have limited support for the variety of job schedulers found on academic and commercial computer clusters. Further complicating these systems are hyperparameters, which often cannot be directly optimized by conventional methods requiring users to determine which combination of values is best via trial and error. The recently-released LoonyBin open-source workflow management tool addresses these issues by providing: 1) a visual interface for the user to create and modify workflows; 2) a well-defined logging mechanism; 3) a script generator that compiles visual workflows into shell scripts, and 4) the concept of Hyperworkflows, which intuitively and succinctly encodes small experimental variations within a larger workflow. In this paper, we describe the Machine Translation Toolpack for LoonyBin, which exposes state-of-the-art machine translation tools as drag-and-drop components within LoonyBin. 1. LoonyBin Background Empirical research in machine translation has become a complex multi-stage process with many stages being run under multiple experimental conditions (i.e. with different corpora and different sets of hyperparameters). The management of such workflows presents a real challenge in terms of keeping results organized, analyzing results at every stage, and automating the workflow. For example, in syntactic statistical machine translation, a typical experiment conc © 2009 PBML. All rights reserved. Please cite this article as: Jonathan H. Clark, Jonathan Weese, Byung Gyu Ahn, Andreas Zollmann, Qin Gao, Kenneth Heafield, Alon Lavie, The Machine Translation Toolpack for LoonyBin: Automated Management of Experimental Machine Translation HyperWorkflows: . The Prague Bulletin of Mathematical Linguistics No. ? , 2009, 1–10. PBML ? DECEMBER 2009 sists of over 20 tools with a complex network of dependencies spanning multiple machines or even clusters of machines. Parsing and phrase extraction might be run on a large cluster of hundreds of low-memory machines, preprocessing and word alignment might be run on a local server, while tuning and decoding might be done on a small cluster of large-memory machines. Further, this system might be run for two language pairs and using 10 sets of features in the translation model to verify some experimental hypothesis. With these needs in mind, LoonyBin (Clark and Lavie, 2010) accommodates workflows that: • span various machines, clusters, and schedulers • involve many separate tools, which can be invoked by arbitrary UNIX commands • have components that are run multiple times under multiple conditions • evolve quickly with tools frequently being added, removed, and swapped LoonyBin accomplishes this by providing the following advantages over current common practices: • associating sanity checks and logging directly with tools, separating these from ad hoc wrappers and automation scripts • maintaining a cleanly organized directory structure for each step and each condition under which a step is run • providing a resume-on-failure mechanism for every stage in the pipeline • making it easy for those without a detailed knowledge of each tool’s internals to run the system by providing textual descriptions of each parameter, input file, and output file in a graphical workflow designer • automatically copying required files between machines/clusters via SSH • compiling workflows into shell scripts, a medium already in widespread use by NLP researchers 1.1. Workflow Semantics We now discuss the representation of workflows in LoonyBin. In their most basic form, LoonyBin represents workflows as Directed Acyclic Graphs (DAGs). In this form, each vertex represents a tool, which produces output files given input files and parameters, and directed edges indicate relative temporal ordering of tools and information flow (files or parameters) by mapping the output of one tool to the inputs of the next. A tool descriptor defines the commands necessary to run a tool given inputs, outputs, and parameters. Custom tool descriptors can be implemented via simple user-defined Python scripts that generate shell commands. These tool descriptors contain pre-analyzers to check the sanity of the inputs and log information and post-analyzers to check the sanity of the output files, log information about the outputs, and extract log data from any third-party log file formats.