Initialising Neural Networks with Prior Knowledge

This thesis explores the relationship between two classification models: decision trees and multilayer perceptrons. Decision trees carve up databases into box-shaped regions, and make predictions based on the majority class in each box. They are quick to build and relatively easy to interpret. Multilayer perceptrons (MLPs) are often more accurate than decision trees, because they are able to use soft, curved, arbitrarily oriented decision boundaries. Unfortunately MLPs typically require a great deal of effort to determine a good number and arrangement of neural units, and then require many passes through the database to determine a good set of connection weights. The cost of creating and training an MLP is thus hundreds of times greater than the cost of creating a decision tree, for perhaps only a small gain in accuracy. The following scheme is proposed for reducing the computational cost of creating and training MLPs. First, build and prune a decision tree to generate prior knowledge of the database. Then, use that knowledge to determine the initial architecture and connection weights of an MLP. Finally, use a training algorithm to refine the knowledge now embedded in the MLP. This scheme has two potential advantages: a suitable neural network architecture is determined very quickly, and training should require far fewer passes through the data. In this thesis, new algorithms for initialising MLPs from decision trees are developed. The algorithms require just one traversal of a decision tree, and produce four-layer MLPs with the same number of hidden units as there are nodes in the tree. The resulting MLPs can be shown to reach a state more accurate than the decision trees that initialised them, in fewer training epochs than a standard MLP. Employing this approach typically results in MLPs that are just as accurate as standard MLPs, and an order of magnitude cheaper to train. ii Acknowledgements Most important of all, thanks to my wonderful wife Janet, for a truly amazing amount of encouragement and support. Many thanks to my supervisors, Associate Professor Anthony Robins and Doctor Ian McDonald. Your unwavering belief that I could do this has meant a great deal to me. Thanks also to Doctor Chris Handley, for your positive exuberance in reading and commenting on drafts. I would like to thank my colleagues Doctor Richard O’Keefe and Doctor Willem Labuschagne. Both of you have had an enormous effect on how I see the world of computing. Special mention must also be made of Professors Brian Cox and Geoff Wyvill. Without your initial interest and encouragement, I wouldn’t be doing what I’m doing. Also, special thanks to the staff and students in the Department of Computer Science at Otago. Your patience and understanding made it possible for me to finish this thesis—what a great bunch of people to work with! Especially my postgrad students: Yun Sing, Zhou, and Chris; thanks for being so patient these last few months. To Joe and Roanne at Profiler Corporation: thanks for giving me a shot. In many ways, the extra stimulation really helped me to clarify my ideas, and get this thesis finished. I would not be able to do what I do but for my parents, who worked very hard so that I could have the best education possible. Got there in the end! Thanks, Mum and Dad. And thanks Marina, Bruce, and David for always being positive and encouraging. Finally, I cannot thank enough all the friends who have provided company, food, and wine while Janet and I have worked on our doctorates. Janet’s brother David, Corrin, Andrea, Sandy, Nuran, Ben, Sana, Dave, and Miche; thank you all. I owe my thanks to so many; I hope I have not forgotten anyone. If so, please forgive my oversight, and accept my gratitude. iii