From: whitbeck@equinox.unr.edu (Michael Whitbeck)
Newsgroups: sci.chem,comp.ai.neural-nets
Subject: Re: chemometrics: statist
Date: 10 Nov 92 18:44:32 GMT
Followup-To: sci.chem
Distribution: sci
Organization: University of Nevada, Reno

In article <1992Nov4.073306.4621@bb1t.monsanto.com> bjgaed@bb1t.monsanto.com writes:
>In article <1992Nov1.4261.6313@dosgate>, "david gebhardt" <david.gebhardt@canrem.com> writes:
>> On 10-31-92 at 5:04am, Bjgaed wrote to All:
>> 
>> BD>My question is this:  What does the neural net approach get you that
>> BD>the statistical approach does not?  Obviously you need a good deal
>> BD>more computing muscle for the neural net approach, so how big a bang
>> BD>do you get for the bucks?
>> 
>> I couldn't say for sure, but I would venture that the power of
>> prediction for the neural net would be greater than that for a
>> statistical method. Of course, that is also going to depend on the
>> neural net configuration and the amount of training that the net has
>> received.
>
>Why would you "venture" that?  This is still my question:  Is there 
>any basis for assuming better predictive power for the neural net 
>approach over the statistical approach?
>
>The software I was looking at was a pretty simple-minded neural net:  
>About 10 inputs, 1 output, and 1 hidden level.  Just about the 
>simplest neural net you can imagine, and it neede a Silicon Graphics 
>workstation to run.  As long as I have enough experiments for the 

no, the simplest is 1 input, 1 output, no hidden!
The computing power required depends on the net architecture
(number of nodes) AND the problem to be solved. A great deal of
PD nn code is written to run on plain 'ol (yuck) PC's- probably
most codes in fact!
A fully interconnected 3-layer (in-hidden-out) is said to be 
equivalent to PCA. There has been some discussion of this in
Comp.ai.neural-nets I believe. You might also look in the
neuroprose archive.

>degrees of freedom involved I could do this one on a desktop PC or Mac 
>in a walk with any number of inexpensive statistical saftware 
>packages. 

absolutely true. NN is not necessarily BETTER than standard
parametric statistical methods; it is just DIFFERENT. One of
the differences is being model-free. 
But, depending on the problem, NN can run just as well on a PC
as many 'standard' statistical methods and when properly
implemented can actually do better. But as far as I know there
is no existing analysis to aid in determining when one method
should be preferred over another.

>
>> The problem is that it is not an easy concept to explain. The neural net

matter of opinion, here is the backpropagation algorithm in c++
(just add input/output formatting lines and a matrix class
library)
 
 int layer;
 for (layer = 0; layer < layers-1; layer++)      // Feed Forward
      x[layer+1] = tanh(b[layer] + x[layer]%W[layer]);
      x[layers] = b[layers-1] + x[layers-1]%W[layers-1];

 d[layers-1] = eta*(y - x[layers]);              // Back Propagate
 for (layer = layers-1; layer > 0; layer--)
     d[layer-1] = d[layer]%W[layer].t()*(1.0 - x[layer]*x[layer]);
 
 for (layer = 0; layer < layers; layer++)        // Update
     {
     b[layer] += d[layer];
     W[layer] += d[layer]&x[layer];
     };

 
>> uses past experiences to predict future occurrences and adapts itself.
>> This should make it more accurate than a simple statistical approach.

I guess the keyword here is *simple*.
However please note that if the data is accurately modeled by a
straight line then a least-squares straight line (or at least a
nonparametric st. line -like Thiels method-) can do no
worse/better than a NN.

>
>Why?  Really, no flame intended.  As I understand it neural nets have 
>the capability of using a much wider variety of functions than linear 
>or polynomial multiple regression, but doesn't that depend on the 
>particular "teaching" algorithm the neural net uses? (The software 

No. There is a theorem that states that the 3-layer ANN can map any
function (y->f) given a weighting function g (actually the
theorem says that a function g exists such that...).
Unfortunately there is no theorem that states what g() is!
The choice in weighting or transfer functions is primarily
dictated by computational speed and whether the net is designed
to memorize or to map with 'interpolation' capability.

>developer was very reluctant to talk about the algorithm; 
>understandable, as it is probably proprietary.)  And can't you do 

there are short-cuts (e.g. quickprop) in the error correction
phase- that is most likely where the proprietary code comes into
play.

>the same in regression by transforming your input data, like using 
>-log(response) instead of the raw response when doing QSAR?
> 

Of course log(data) also transforms log(random error) ! often
overlooked!!!

The discussion so far has focussed on backpropagation of error-
This not the ONLY NN algorithm. Codes like ART[1,2,MAP] are
completely different--- 

Then there is ANN_PLS; a combination of backprop + PLS !
Gee life is complicated.

BOTTOM LINE
These new mathematical methods for data analysis are not
intrinsically (so far) superior/inferior to older methods; they
are primarily just different. In any specific application one or
the other may offer advantages/disadvantages. At the moment
neural methods are *new* (well recently revived) and are subject
to a *LOT* of abuse. The most common is *too many nodes* in the
net- leading to ridiculous run times and tremendous overfitting.
The best rule of thumb, if you plan to try any NN method, is
KISS (keep it simple stupid!). The *same* holds for
'traditional' parametric statistical methods.

Just my 2 cents....

___________________________________________________________
|Mike Whitbeck             | whitbeck@equinox.unr.edu     |
|Desert Research Inst.     | whitbeck@wrc.unr.edu         |
|POB 60220                 |                              |
|RENO, NV 89506            | 702-673-7348                 |
|__________________________|______________________________|

All academics have the potential for being insatiable...
but the chemists are the most expensive and insatiable among the
expensive and insatiable. - J. Martin in "To Rise Above Principle"