From: sgrenander@nasamail.jpl.nasa.gov (Sven Grenander)
Subject: Re: Hidden Layers, How many?
Organization: Jet Propulsion Laboratory
Date: Wed, 15 Apr 1992 21:34:48 GMT

The following message seems to have gottent truncated. As the questions are
probably of more than passing interest for "NN-engineers" I would appreciate
it if you would care to complete what was lost/cut  off.

In article <1992Apr15.145721.10666@news.media.mit.edu>, minsky@media.mit.edu (Marvin Minsky) writes:
> 
> In article <0133@calsci.UUCP> al%calsci@gvgpsa.gvg.tek.com (Al Petterson) writes:
> >
> >In article <1992Apr9.175720.7849@elroy.jpl.nasa.gov>,
> >sgrenander@nasamail.jpl.nasa.gov (Sven Grenander) writes:
> >>
> >> Using the rule of thumb supplied with BrainMaker/Mac, it ought to have
> >> around 6 hidden neurons in a single hidden layer.  I don't know how they
> >> (California Scientific Software) came up with this simple formula...
> >
> >Empirical observation.  It obviously depends on the complexity of your
> >training set.  We're engineers here, not researchers, so I know that's an
> >unsatisfying answer, but it's the best I can do.
> >
> >> To follow up on your question with a question of my own: It is my
> >> understanding that a 1-[hidden]-layer net can theoretically learn anything
> >> that an n-[hidden]-layer net can learn (please correct me if I am wrong).
> >> Having experimented with various numbers of layers I have found no advantage
> >> in increasing the number of layers.  Are there any rules of thumb governing
> >> more-than-1-layer situations ?
> >
> >The key word is "theoretically." Backpropagation isn't perfect.  In some
> >instances (such as an output neuron whose value has multiple inflection
> >points as one or several input neurons vary linearly) backprop won't train
> >with a single hidden layer.
> >
> >There's no rules of thumb, except: If one hidden layer doesn't converge, or
> >requires what seems an excessive number of hidden neurons, try two hidden
> >layers.
> >
> >I've never seen a network benefit from more than two hidden layers.
> 
> Interesting issues here.  Chapters 6 and 7 of "Perceptrons" prove
> various theorems about how many connections are required in order for
> a net with one inner layer to recognize various pattern classes.
> Generally, the NN community has not realized that this stuff is the
> main concern of the book -- and not the convergence issue.  Despite
> what losers like Hecht-Neilsson and others have published, all these
> theorems appear still to be true.  ("Flash: General Franco is still
> dead!" as they used to say on Saturday Night Live.)  I mention this
> mainly because Papert and I are merely entertained by his distortion
> of the history, but genuinely annoyed by the accompanying
> misrepresentation of the mathematical aspects.
> 
>   In any case, a little thought will show that if Pattern P has "order
> N" for a one-hidden-layer net, then it will have order of the order of
> the k-th root of N for a net with K hidden layers.
>   So, as Patterson says, we learn partly through engineering and
> partly through theory.  Has anyone tried to extend this theory to
> finding the orders -- and then, perhaps , to thus discover some bounds
> on the convergences -- of some interesting P's for K-inner-layer nets?
> And also of course, as Fahlman points out, for nets like those plus
> 
> 

Thank You.

-Sven