A Demonstration of a Robust Ocean-Based Learner

Machine learning is often modeled as the process of extrapolating samples of a function. This extrapolation requires both samples and “inductive bias. ” Bias towards low complexity, as in Occam’s Razor, is particularly important. Kolmogorov complexity was developed to formalize this process [5]. Important theoretical results have also been developed using more abstract measures of complexity [4]. Therefore, there is a strong theoretical basis for Occam-based learning. Kolmogorov complexity is a general measure, which would allow learning of many different kinds of functions (i.e. robust learning); however, it has been proven that its exact computation is not tractable. There have been some tractable measures of complexity used in actual implementations of Occam-based learning [3], such as the Abductory Inference Mechanism (AIM). However, these measures of complexity are relatively narrow, which implies non-robust learning. The challenge is to develop robust and tractable measures of complexity. One approach to this challenge is called pattern theory [6], where we think of robust complexity determination as the problem of finding a pattern. Pattern theory uses Decomposed Function Cardinality (DFC), proposed by Y. S. Abu-Mostafa as a general measure of complexity [I, p.1281. We demonstrate that this measure of complexity allows robust learning, yet is sufficiently tractable to support the learning of non-trivial functions. We develop support for the generality of the measure both theoretically and experimentally. Generality is supported theoretically by proving its relationship to the conventional measures of circuit complexity, time complexity and program length. Generality is supported experimentally by using a decomposition program (referred to as AFD) derived from the work of R. L. Ashenhurst [2] and others. The experimental work includes the measurement of the DFC of a large variety of functions, determining the correlation of DFC with more specialized measures within their domain of application, and machine learning experiments. The DFC of over 800 non-randomly generated functions was measured, including many kznds of functions (numeric, symbolic, chaotic, stringbased, graph-based, images and files). Roughly 98 percent of the non-randomly generated functions had low DFC (versus less than 1 percent for random functions). The 2 percent that did not decompose were the more complex of the non-randomly generated functions rather than some class of low complexity that AFD could not deal with. It is important to note that when AFD says the DFC is low, which it did some 800 times, it also provides an algorithm (or a description of the pattern found). AFD found the classical algorithms for a number of functions. The correlation coefficient between DFC and a ranking of the complexity of images by people was 0.8. The correlation between DFC and the compression factor of two commercial data compression programs was about 0.9. The correlation be350 tween DFC and the Lyapunov exponent for logistic functions was 0.9. These high correlations show that the single measure, DFC, reflects the essential structure in each, very different, situation. A fourth correlation experiment found no correlation between people’s ability to recognize concepts and the DFC of those concepts. This lack of correlation may be a result of the narrow range of DFC’s involved (5% of its full range). In learning experiments, AFD did as well as a back-propogation trained Neural Network (NN) on problems well-suited to NN’s. However, on other problems such as parity, AFD learned a 256 point function from 50 samples whereas the NN required all 256 points. The findings were similar for the AIM program. In both cases, the extrapolations of the NN and AIM were not robust while AFD consistently learned functions of low complexity with few samples. The experiments to date have been limited to small (less than 10 variables) binary functions. The results on these small, but non-trivial, functions have consistently pointed to a promising ability to find many different kinds of patterns. Therefore, we believe these results are the first demonstration of robust Occam-based learning and help join an important body of theoretical results with practical machine learning.