From: ibm@dit.upm.es (Ignacio Bellido Montes)
Newsgroups: comp.ai.neural-nets
Subject: Re: Idea to optimize NN
Date: 16 Oct 92 09:42:09 GMT
Distribution: comp
Organization: Dept. Ingenieria de Sistemas Telematicos, UPM, Madrid, Spain.



Georg,
	First of all, many ideas are good, but sometimes somebody else has
had the same idea a couple of years ago.  In the case of yours, my belief is
that often those are therefore good ideas. As a matter of fact, your second
idea is very similar to aspects of my current research.

>If the input data to be learned are correlated ( like the fifth bit
>always equal to the sixth bit) or restricted in space 
>then this has (?!) consequences for the weights. 
>Very simple examples for this consequences or consequences
>resulting from the learning process may be that a weight is
>approximately 0 and remains there for a couple of learning steps. 

	If patterns are correlated, there is of course redundant information
which may cause long training times, but most learning methods have some
kind of filtering ability. In fact they can reduce the information just
to the amount required to learn and generalize, (due to a good internal
representation). Anyway, my impression is that your idea is to classify those
inputs and "forget" them. That's good. The main problem is to identify which
neurons are correlated to another, using only internal information, while
avoiding lots of extra computation during the learning phase.
This is not trivial.

>Why don't skip this synapse and continue learning?

	There are techniques that allow us to remove unimportant weights
during learning or afterwards. Those techniques usually increase the network
generalization capabilities because they throw out those weights that have
learned ``noisy'' information. Some of those techniques are: Weight
Elimination and Optimal Brain Damage.

>All weights of a neuron of the input layer are (quite) equal. If the
                  ^^^^^^
>input data are normalized, the output of this neuron will be the same
                ^^^^^^^^^^                ^^^^^^^^^^^
>for every data set. It can be removed without lost if the threshold of
>the following neurons are changed.

I don't really understand that. If all weights going from one input neuron
to the first hidden layer are (almost) equal, then all the hidden units are
receiving the same information from THIS input, but it will be most likely
be different for other input patterns, in which case it should't be removed.

I also don't understand the normalization of the input data you talk about.

In fact, your proposition is only valid if we are talking about one input unit
that has the same value for all the patterns; so there is no information
associated with it. It's just a bias unit and it doesn't matter whether the
weights to the next layer are equal or not.

Anyway, what you describe appears to be something related to pruning while
learning, which is an interesting topic, because pruning an excess of weights
can increase the generalization capabilities of the net.
In fact, I'm actually working on pruning weights without impairing the
learning capabilities of the net.

	Here are three interesting articles:

-	Weigend, A.S., Rumelhart, D.E., and Huberman, B.A., ``Generalization by
Weight-Elimination with Application to Forecasting'', in Advances in Neural
Information Processing Systems 3, R. Lippmann et al. Eds., Morgan Kaufmann, 1991

-	Le Cun, Y., Denker, J.S. and Solla, S.A. ``Optimal Brain Damage'', in
Advances in Neural Information Processing Systems 2, D. Touretzky Ed., Morgan
Kaufmann, 1990.

-	Bellido, I. an Fernandez, G. ``Backpropagation Growing Networks:
Towards Local Minima Elimination'', in A. Prieto Ed., Artificial Neural
Networks, Proceedings of the IWANN'91, Granada 1991, Springer Verlag 1991.


				Ignacio Bellido


--
============================================================================
Ignacio Bellido Fernandez-Montes,

Department of Telematic			IDIAP
Systems Engineering,			Institute Delle-Molle d'Intelligence
Artificial Intelligence Laboratory.	Artificielle Perceptive
Madrid University of Technology.	Martigny (Switzerland)

e-mail:	ibm@dit.upm.es		e-mail: ibm@idiap.ch
Phone:  .. 34 - 1 - 5495700 ext 440	Phone:  .. 41 - 26 - 227664
Fax:    .. 34 - 1 - 5432077		Fax:    .. 41 - 26 - 227818
============================================================================