From: samodena@csemail.cropsci.ncsu.edu (S. A. Modena)
Subject: Limitations of Neural Nets with Quadrature data
Summary: A recent NN book sparks an idea about protein design
Keywords: NN neural nets Eco RI info theory Shannon hyperplane separability
Organization: Crop Science Dept., NCSU, Raleigh, NC 27695-7620
Date: Wed, 26 Aug 1992 03:22:03 GMT

"Feedforward Networks for Supervised Training:

"We simulate supervised training of neural network connection
weights and minimize specified measures of pattern-association
error.  .....we consider only feedforward connections.  We start
with the LMS algorithm for simple perceptrons....

.....

"The simple two-layer pattern associator (perceptrons) ...relates
input- and output-layer activations by

                       8
[1]     layer2[i] = SUM-OVER W[i,k]*layer1[k]  (i=1,2,...8)
                      k=1

"Our objective is to train the connection weights W[i,k] so that the
perceptron associates eight given binary (0,1) output patterns
(target patterns) with eight corresponding given binary input patterns,
say the output (0,0,0,0,0,0,0,1) with the input (1,1,0,0,0,0,0,0), etc.
This will not always work; a perceptron based on equation [1] can
distinguish input patterns only if they are "linearly separable" by a
hyperplane in pattern space. We can still obtain useful results.

......skip a lot....

"As we already noted, the input-pattern classes corresponding to
different output patterns must be linearly separable.  This means
that their pattern points must be separated by a hyperplane

        W*layer1 = CONST

in n-dimensional space.  This is, for instance, not true for the
simple two-dimensional binary input patterns (0,0), (0,1), (1,0), (1,1)
needed for the XOR (exclusive-OR) operation.  As we add extra layers,
we shall obtain more general separation hypersurfaces made up of
multiple hyperplane segments.

"Hidden Layers and Non-Linear Operations

"Augmenting the basic pattern associator... with intermediate "hidden"
layers will not help if the relations between different layer
activations are *linear* like equation [1], for such a network acts
exactly like the equivalent linear two-layer network.  Nonlinear 
operations are *absolutely* necessary.

QUOTED from:   Neural Network Experiments on Personal Computers
                      and Workstations
               Granino A. Korn
               "A Bradford Book"
               The MIT Press 1991 Cambridge
               QA76.87.K67
               ISBN 0-262-61073-6  (with computer diskette)

This reading brought to minde two things:

1:
Jack Cramer argued that not all information is in the primary DNA
sequence in the case of histone-core placement...or at least, it
is better to have supplimentary information available to "simply"
the problem of site detection.

Some argued that a neural net ought "to work" by analogy with the
perceptron "success" for promoter recognition (...feel free to
correct me).

I interprete the above to mean that perhaps a NN can do it all with
just DNA sequence, but that NONLINEAR part is the zinger.  Then, what
nonlinear partial derivitive does one use to estimate the error
terms?  That seems to be the critical point: the difference between
a theoretically possible solution and an actual solution.  Actually,
this is similar to Shannon's proofs: proving the existance that a
code exists with certain properties is not equivalent to describing the
implementation parameters of the code.

2:
The problem involved with information that is succinctly expressed
in two bit wide alphabets....the difficulty experienced by perceptrons
appears to do with the conciseness of expressing four alternative
states with a two-bit representation.  Here I feel there is a hint
(or reinforcement) of why channel codes often are implemented by
widening the number of bits for representing letters of the alphabet.
There is a need for hyperplane separability, and that is easily
achieved by increasing the character width. (as I described in
the CD ROM encoding scheme somewhat earlier).

These thoughts lead me to wonder whether proteins designed to interact
in specific ways with DNA must have a "contact layer" not unlike
Layer1 of a neural net.  And then subsequent "hidden layers" (other
parts of the folded protein structure) to perform nonlinear activation
functions in order to achieve hypersurface separability. The LayerN
neurons can be represented in a protein by the "decision action"
which in the case of Eco RI is "0" or "1": cleave or not-cleave.              
Actually in the case of Eco RI, it is 00, 01, 10, 11: no cleavage;
cleave left only; cleave right only; cleave both......four states
of decision result needed to describe wildtype Eco RI and the studied
variants.

Back to NNs:

    "As we add extra layers, we shall obtain more general separation 
hypersurfaces made up of multiple hyperplane segments."

That statement intrigues me.  It's a way to think about protein design
also.

Steve
---
+------------------------------------------------------------------+
|     In person:  Steve Modena     AB4EL                           |
|     On phone:   (919) 515-5328                                   |
|     At e-mail:  nmodena@unity.ncsu.edu                           | 
|                 samodena@csemail.cropsci.ncsu.edu                |
|                 [ either email address is read each day ]        |
|     By snail:   Crop Sci Dept, Box 7620, NCSU, Raleigh, NC 27695 |
+------------------------------------------------------------------+
         Lighten UP!  It's just a computer doing that to you.
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
         




Newsgroups: bionet.general