From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: (multi-layer + shortcut connections) vs
Organization: University of Alberta, Edmonton, Canada
Date: Thu, 16 Apr 1992 18:54:18 GMT

massimo@cui.unige.ch (DEFRANCESCO Massimo) writes:

>There is an important reason why multi-layer networks are more interesting
>than general feedforward networks. I would call it `composability' or
>`modularity'.
>I'm actually working on a framework (and a simulator) for `first class
>neural networks', that is an environment in which neural networks are
>first class citizens with functional behavior:

>	1. they can be passed as parameters to functions, 
>	   assigned to variables, etc.
>	2. they can be returned by functions
>	3. They can be `composed'

>Backpropagation can be defined locally on a functional layer and
>the learning behavior of the composition is perfectly defined by the 
>composition of the learning behaviors!
>The advantage: modularity, and the possibility to define complex structures

I like your idea a lot, and agree that composability is important for
achieving complex behaviors.  In theory, the learning behavior of the
composition is a result of composition of the learning behaviors.  In
fact, I don't see the composition as, in principle, any different from
one much larger, deeper network, for the purpose of discussing
learning.

What I am in doubt about is whether the speed of BP learning will
deteriorate as the number of layers increases.  What is the current
situation with regard to large, deep BP networks or deep compositions,
do they learn well?  How about 10 or 20 layers?  I have to add that if
one specifies what some subnets are supposed to do, and trains them
separately, that makes the problem too easy.  I hope we agree on that.

For comparison, we have trained adaptive logic networks with about 17
layers, with no supervision at the intermediate levels, only at the
network output.  The reason we haven't done anything larger is just
memory.  Things start to page after that on a 4 Meg SPARC.  If we had
a composition system such as you are discussing, maybe we wouldn't be
limited in this way, so I am quite interested in your results, and
believe they might really be good for adaptive logic networks.

If deep BP networks can learn, the next question is: can they do it
efficiently?  A large BP network or composition has an evaluation time
proportional to the size of the network, at best, and a learning time
probably much worse.  An adaptive logic network, by virtue of the lazy
evaluation of boolean logic, benefits from a speedup that grows as the
depth increases.  For example, the speedup for depth 20 and a disjoint
binary tree would be by a factor of 157.  Learning speed is again
worse.  Nevertheless, this speedup that grows exponentially with depth
is the main reason I don't think BP nets will be able to compete as
problems grow larger.

Anyone should feel free to criticize these comments.  If I am deluding
myself that laziness is important for ANS, I would like to know about
it.  Thanks.

Bill

--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071