From: kavuri@lips.ecn.purdue.edu (Surya N Kavuri )
Subject: Re: Back-Propagation
Organization: Purdue University Engineering Computer Network
Date: Fri, 10 Apr 1992 17:16:33 GMT

In article <1992Apr10.014619.1550@Silence.pi.nctu.edu.tw>, gcp79527@csunix.csie.nctu.edu.tw writes:
>     There is a local minimum problem in MLP. A nolinear(?) net with gradient 
> search may (someone said that he often gets what he want) sufer from this.
> So, if we fix the previous learned weights, and then add extra nodes, it seems
> that it will not escape from this local minimum.  Then, does this lead to 
> a net of good generalization? If we start from one HU and add extra node(s) 
> during training, I think it is much like Principal Component Analysis, 
> doing this way may yield a better result.
> 
> Yo-Yau Lin.

 I guess that is not true.  The local minima was in the space of the weights
 *before* the addition of the next hidden node.  By adding a new node, you
  have added more weights, and the space of weights has now changed with 
  extra degrees of freedom.  Addition of a new node, therefore, need not 
  be stuck at the same local minima - in fact, that local minima may cease
  to be a local minima in the changed weight space.  

 I agree that addition of hidden nodes, one at a time, is like doing 
 PCA sequentially.  But there is a small difference.  

  To illustrate this, consider only linear units for hidden nodes.  Now
  each of these linear units will form principal components only with 
  the additional constraint that the weights coming to it from the 
  inputs are normalized (to a unit vector).  Similarly we need to 
  normalize the weights from the hidden node to output nodes.    

  When saturating functions like sigmoid are used, this alleviates the
  bad effects of not normalizing thus giving a similar effect as 
  PCA. 

 On the issue of generalization:

 1) try cross-validation 
 
 2) use some criteria like Akaike-criterion for determining when 
    the addition of more nodes is not desirable.  This sort of 
    measures the network performance while also considering the 
    model complexity.

 3) The idea of adding one node at a time while fixing previous
    nodes is "correct" only when the hidden nodes are orthogonal.
    When they are not orhogonal, they interact - meaning that 
    you should retrain the entire net.  When you avoid this 
    retraining, generalization may suffer.  
    The process of adding "hidden nodes" one at a time, has been 
    known for a long time in function approximation as an 
    expansion in orthogonal functions.  When the hidden nodes 
    are orthogonal, each of them have a definite contribution
    to the reduction of residual error.  This contribution is
    not affected by your choice to add or delete other nodes.
    This means that once and for all, you can decide whether 
    that node is good or bad.  

    It is because of this orthogonality that you are able to 
    say whether a subset of principal components (in PCA) are
    sufficient or not and evaluate their "utility" independently.

 4) Use some form of penalty terms in your objective for 
    "regularization".  This penalty should enforce smoothness.
    In fact, is not this smoothness what we mean by generalization
    most of the time ? See the paper by Poggio and Girosi in 
    IEEE proceedings, sept 1990 issue.


 Surya Kavuri
 (FIAT LUX)