MaLARea: a Metasystem for Automated Reasoning in Large Theories

MaLARea (a Machine Learner for Automated Reasoning) is a simple metasystem iteratively combining deductive Automated Reasoning tools (now the E and the SPASS ATP systems) with a machine learning component (now the SNoW system used in the naive Bayesian learning mode). Its intended use is in large theories, i.e. on a large number of problems which in a consistent fashion use many axioms, lemmas, theorems, definitions and symbols. The system works in cycles of theorem proving followed by machine learning from successful proofs, using the learned information to prune the set of available axioms for the next theorem proving cycle. Although the metasystem is quite simple (ca. 1000 lines of Perl code), its design already now poses quite interesting questions about the nature of thinking, in particular, about how (and if and when) to combine learning from previous experience to attack difficult unsolved problems. The first version of MaLARea has been tested on the more difficult (chainy) division of the MPTP Challenge solving 142 problems out of 252, in comparison to E’s 89 and SPASS’ 81 solved problems. It also outperforms the SRASS metasystem, which also uses E and SPASS as components, and solves 126 problems. 1 Motivation and Introduction In the recent years there has been a growing need for Automated Reasoning (AR) in Large Theories (ARLT). First, formal mathematical libraries created by proof assistants like Mizar, Isabelle, HOL, Coq, and others have been growing, often fed by important challenges in the field of formal mathematics (e.g. the Jordan Curve Theorem, the Kepler Conjecture, the Four Color Theorem, and the Prime Number Theorem). At least some of these proof assistants are using some automated deductive methods, and some of these libraries are (at least partially) translatable to first-order format [MP06b, Urb06] (like the TPTP language [SS98]) suitable for Automated Theorem Provers (ATPs). Also, when looking at the way how some of the hard mathematical problems like Fermat’s last theorem and Poincare’s conjecture were solved, it seems that difficult mathematical problems quite often necessitate the development of large (sometimes even seemingly unrelated) mathematical theories, which are eventually ingeniously combined to produce the required results. An attempt to create a set of notrivial large-theory mathematical problems as an initial benchmark for ARLT systems is the MPTP Challenge1 (especially its chainy division), consisting of 252 problems (some of them with quite long human-written formal proofs) in a theory with 1234 formulas and 418 symbols which represents a part of the Mizar library. http://www.cs. miami.edu/~tptp/MPTPChallenge/ Today, there also seems to be a growing interest in using Automated Reasoning methods (and translation to TPTP) for sufficiently formal large “nonmathematical” knowledge bases, like SUMO [NP01] and CyC [MJWD06]. It seems that semantic ontologies and semantic tagging and processing e.g. for publications in natural and technical sciences are currently taking off, creating more and more applications for ARLT. On the other hand, it seems to be quite a common thinking that the resolution-based ATP systems using usually some complete combination of paramodulation and resolution are easily overwhelmed when a large number of irrelevant axioms are added to the problems. Sometimes (actually surprisingly often in the AI and Formal Math communities) this is even used as an argument attempting to demonstrate the futility of using ATPs for anything “serious” in formal mathematics and large theories in general. There have been several answers to this problem (and to the general problem of the fast growing search space) from the ATP community so far. Better (i.e. less prolific while still complete) versions of the original calculi, and their combinations with even more efficient specialpurpose decision procedures, have been a constant research topic in the ATP field. More recently, a number of heuristical approaches have appeared and been implemented. These methods include strategy scheduling (competitive or even cooperative, see e.g. [Sut01]), problem classification and learning of optimal strategies for classes of problems, lemmatization (i.e. restarting the problems only with a few most important lemmas found so far, see also [Pud06]), weakening [NW04] (solving a simpler problem, and using the solution to guide the original problem), etc. The metasystem for ARLT which is described here falls into this second category of heuristical additions governing the basic ATP inference process. It is based on an assumption (often spelled-out by the critics of uniform ATP) that while working on problems in a particular domain, it is generally useful to have the knowledge of how previous problems were solved, and to be able to re-use that particular knowledge in a possibly nonuniform and even generally incomplete way. Again, this idea is not exactly new, and not only in the world of “very-AI-but-very-weak” experimental AR systems implemented in Prolog. At least the very efficient E prover [Sch02] has the optional capability to learn from previous proofs, and to re-use that knowledge for solving “similar” problems. In comparison to this advanced functionality of E, the first version of MaLARea is quite simple, and more suitable for easy experimenting with different systems. The machine learning is done by an external software, and one can imagine any reasonable learning system to take the place of the currently used SNoW [CCRR99] system. The features used for learning should be easy to extend, as well as the whole learning setting. It is quite easy to change the parameters of how and when particular subsystems are called, and to experiment with additional heuristics. In the next section we describe the structure of the metasystem in more detail, and then we show its current performance on the chainy division of the MPTP Challenge.