Creating artificial neural networks that generalize

-We develop a technique to test the hypothesis that multilavered../~'ed@~rward network,~ with [~'w units on the .Drst hidden layer ,~eneralize better than networks with many ttllits in the ~irst laver. Large networks are trained to per/orrn a class![)cation task and the redundant units are removed ("pruning") to produce the smallest network capable of'perf'orming the task. A teclmiqtte ,/~r inserting layers u'here /~rtttlitlg has introduced linear inseparability is also described. Two tests Of abilio' to generalize are used--the ability to classiflv training inputs corrupwd hv noise and the ability to classtlflv new patterns/)ore each class. The hypothes'is is f?~ltnd to be ,fa{s'e f~>r networks trained with noisy input.s'. Pruning to the mittitnum nt#nber c~f units in the ./irvt layer produces twtworks which correctly classify the training ,set hut j,,eneralize poor O' compared with lar~er ttetworks. Keywords--Neural Networks, Back-propagation, Pattern recognition, Generalization, t|idden units. Pruning. I N T R O D U C T I O N One of the major strengths of artificial neural networks is their ability to recognize or correctly classify patterns which have never been presented to the network before. Neural networks appear unique in their ability to extract the essential features from a training set and use them to identify new inputs. It is not known how the size or structure of a network affects this quality. This work concerns layered, feed-forward networks learning a classification task by back-propagation. Our desire was to investigate the relationship between network structure and the ability of the network to generalize from the training set. We examined the effect of noise in the training set on network structure and generalization, and we examined the effects of network size. This second could not be done simply by training networks of different sizes, since trained networks are not necessarily making effective use of all their hidden units. To address the question of the effective size of a network, we have been developing a technique of training a network which is known or suspected to be larger than required and then trimming off excess units to obtain the smallest network (Sietsma & Dow, Acknowledgement : The authors wish to thank Mr. David A. Penington for his valuable contributions, both in ideas and in excellent computer programs. Requests for reprints should be sent to J. Sietsma, USD, Material Research Laboratory, P.O. Box 50, Ascot Vale, Victoria 3032, Australia. 1988). This is slower than training the "right'" size network from the start, but it proves to have a number of advantages. When we started we had some assumptions about the relationship between size and ability to generalize, which we wished to test. If a network can be trained successfully with a small number of units on the first processing layer, these units must be extracting features of the classes which can be compactly expressed by the units and interpreted by the higher layers. This compact coding might be expected to imply good generalization performance. To have many units in a layer can allow a network to become overspecific, approximating a look-up table, particularly in the extreme where the number of units in the first processing layer is equal to the number of examplars in the training set. It has been suggested that networks with n'.,we layers, and fewer units in the early layers, may generalize better than "'shallow" networks with many units in each layer (Rumelhart, 1988). However, narrow networks with many layers are far harder to train than broad networks of one or two layers. Sometimes we found that after rigorous removal of inessential units, more layers were required to perform the task. This suggested a way of producing long, narrow networks. A broad network could be trained, trimmed to the fewest possible units, and then extra layers inserted to enable the network to relearn the solution. This avoids both the training difficulties and the problem that the smallest number of units needed for a task is not generally known. We could then test