Automatic Derivation of Statistical Algorithms : The EM Family and Beyond

Machine learning has reached a point where most probabilistic methods can be understood as variations, extensions and combinations of a much smaller set of abstract themes, e.g., as different instances of the EM algorithm. This enables the systematic derivation of algorithms customized for different models. Here, we demonstrate the AUTOBAYES system which takes a high-level statistical model specification, uses powerful symbolic techniques based on schema-based program synthesis and computer algebra to derive an efficient specialized algorithm for learning that model, and generates executable code implementing that algorithm. This capability is far beyond that of code collections such as Matlab toolboxes or even tools for model-independent optimization such as BUGS for Gibbs sampling: complex new algorithms can be generated without new programming, algorithms can be highly specialized and tightly crafted for the exact structure of the model and data, and efficient and commented code can be generated for different languages or systems. We present automatically-derived algorithms ranging from closed-form solutions of Bayesian textbook problems to recently-proposed EM algorithms for clustering, regression, and a multinomial form of PCA. 1 Deriving Statistical Algorithms by Computer Overview. We describe a symbolic program synthesis method which acts as a “statistical algorithm compiler:” it can compile a statistical model specification into a custom algorithm design and from that further down into a working program implementing the algorithm design. The system implementing this method, AUTOBAYES, can be loosely thought of as “part theorem prover, part Mathematica, part statistics/learning textbook, and part Numerical Recipes. ” It provides much more than a fixed code repository such as a Matlab toolbox, and allows the creation of efficient algorithms which have never before been implemented, or even written down. The system is intended to automate the more routine application of complex methods in novel contexts. For example, recent multinomial extensions to PCA [2, 4] can be derived in this way. D R A FT The algorithm design problem. Given a dataset and a task, creating a learning method can be characterized by two main questions: 1. What is the model? 2. What algorithm will optimize the model parameters? The statistical algorithm (i.e., a parameter optimization algorithm for the statistical model) can then be implemented manually. The system in this paper answers the algorithm question given that the user has chosen a model for the data,1 and continues through to implementation. Performing this task at the state-of-the-art level requires an intertwined meld of probability theory, computational mathematics, and software engineering. However, a number of factors unite to allow us to solve the algorithm design problem computationally: 1. The existence of fundamental building blocks (e.g., standardized probability distributions, standard optimization procedures, and generic data structures). 2. The formalization of a common representation (i.e., graphical models [3, 14] and program schemas – to be described). 3. The formalization of schema guards (i.e., their known applicability constraints). 2 The challenges of algorithm design. The design problem has an inherently combinatorial nature, since subparts of a function may be optimized recursively and in different ways. It also involves the use of new data structures or approximations to gain performance. As the research statistical algorithms advances, its creative focus should move beyond the ultimately mechanical aspects and towards extending the abstract applicability of already existing schemas (algorithmic principles like EM), improving schemas in ways that generalize across anything they can be applied to, and inventing radically new schemas. 2 Combining Schema-based Synthesis and Bayesian Networks Computational logic and theorem proving. Our approach falls into a class of techniques known as computational logic. It has its basis in logic programming, as exemplified by the Prolog language (in which AUTOBAYES is implemented), a computationally efficient embodiment of the general idea of theorem proving within Horn clause logic. It begins with an initial goal and a set of initial assertions, or axioms, and adds new assertions, or theorems, by repeated application of the axioms, until the goal is proven. In our context, the goal is given by the statistical model; the derived algorithms are side effects of constructive theorems proving the existence of algorithms for the goal. Computer algebra. The first core element which makes automatic algorithm derivation feasible is the fact that we can mechanize the required symbol manipulation, using computer algebra methods. General symbolic differentiation and expression simplification are capabilities fundamental to our approach. AUTOBAYES contains a computer algebra engine using term rewrite rules which are an efficient mechanism for substitution of equal quantities or expressions and thus well-suited for this task.3 Schema-based synthesis. While attractive from a first-principles point of view, the computational cost of full-blown theorem proving grinds simple tasks to a halt while elementary and intermediate facts are reinvented from scratch. To achieve the scale of deduction required by algorithm derivation, we thus utilize a schema-based synthesis technique Strictly, there is a third question: “3. What mechanism will be used to ensure generalization? ” In AUTOBAYES generalization procedures such as cross-validation are implementable as ordinary schemas, and techniques such as likelihood penalization are implementable as additional maximization goals; here we focus on parameter optimization schemas. Learning methods outside the fixedmodel format, such as structure learning and feature selection, can be addressed in system extensions using the described capabilities in an inner loop. Guards vary widely; for example, contrast Nead-Melder simplex or simulated annealing or Gibbs sampling (which require only numeric function evaluation), conjugate gradient (both Jacobian and Hessian), EM and its variational extension [7] (a latent-variable structure). Popular symbolic packages such as Mathematica contain known errors allowing unsound derivations; they also lack the support for reasoning with vector and matrix quantities. D R A FT which breaks away from strict theorem proving. Instead, we effectively encode high-level knowledge, such as the general EM strategy, as schemas, or auxiliary lemmas with explicitly specified preconditions. The second core element which makes automatic algorithm derivation feasible is the fact that we can use Bayesian networks to efficiently encode the preconditions of complex algorithms such as EM. First-order logic representation of Bayesian networks. A theory of indexed Bayesian networks was developed in [8]; here indices are represented as Prolog variables and networks correspond to back-track free datalog programs, allowing the dependencies to be efficiently computed. We have extended these results to work with non-ground probability queries since we seek to determine probabilities over vectors and matrices. Tests for independence on these indexed Bayesian networks are easily developed in Lauritzen’s framework which uses ancestral sets and set separation [10] and is more amenable to a theorem prover than the double negatives of the more widely known d-separation criteria. Given a Bayesian network, some probabilities can easily be extracted by enumerating the component probabilities at each node: Lemma 1. Let be sets of variables over a Bayesian network with . Then descendents and parents hold in the corresponding dependency graph iff the following probability statement holds: "! $#&%' parents #(#)%+* ,. -0/ 0 21 parents 213#(#54 Symbolic probabilistic inference. How can probabilities not satisfying these conditions be converted to symbolic expressions? While many general schemes for inference on networks exist, our principal hurdle is the need to perform this over symbolic expressions incorporating real and integer variables from disparate real or infinite-discrete distributions. For instance, we might wish to compute the full maximum a posteriori probability for the mean and variance vectors of a Gaussian mixture model under a Bayesian framework. While the sum-product framework of [9] is perhaps closer to our formulation, we have out of necessity developed another scheme that lets us extract probabilities on a large class of mixed discrete and real, potentially indexed variables, where no integrals are needed and all marginalization is done by summing out discrete variables. We give the non-indexed case below; this is readily extended to indexed variables (i.e., vectors). Lemma 2. 6 descendents 7 holds and ancestors 8 is independent of given iff there exists a set of variables 9 such that Lemma 1 holds if we replace by ;:< =9 . Moreover, the unique minimal set 9 satisfying these conditions is given by ancestors ? >@ ancestors A ): 8 CB Lemma 3. Let 9 be a subset of ED descendents 7 such that ancestors F9G is independent of 7 :H 8 5D3 9I: ancestors 89G J given 9 . Then Lemma 2 holds if we replace by :K LD0 9 and by 9 . Moreover, there is a unique maximal set A9 satisfying these conditions. Lemma 2 lets us evaluate a probability by a summation: 0 M "! N#&% O ,QP2Dom R /SPGT 0 LU3%V1WU"X5 "! N# while Lemma 3 lets us evaluate a probability by a summation and a ratio: 0 M "! N#)% 0 Y8! [Z\! U "! U # ]! [Z^! U "! U # Since the lemmas also show minimality of the sets 9 and LD0 9 , they also give the minimal conditions under which a probability can be evaluated by discrete summation without integration. These inference lemmas are operationalized as network decomposition schemas. D R A FT However, we usually attempt to decompose a probability into independent components before applying this schema. 3 The AUTOBAY