From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: Mapping BP neural net decision regions.
Organization: University of Alberta, Edmonton, Canada
Date: Wed, 7 Oct 1992 16:00:12 GMT

overman@ssc-vax.uucp (Tim Overman) writes:

>Is there a kind soul out there that can tell me if there is a closed form
>solution to mapping out the decision region to a three layer feedforward
>neural network. The net has 3 input, 3 hidden, and one output
>neurons and has been trained accordingly. One could obtain scatter
>plots from MontiCarlo simulations but it would be nice to be able to
>predict the decision region boundaries based on some mathmatical
>computation using the trained weights of the network.

>Thanks,

>Tim Overman
>Automation and Robotics
>Boeing Defense and Space Group
>Seattle,WA
>(206) 773-5850
>overman@ssc-vax.UUCP

Assume the inputs to the net are reals x, y, and z.
The boundary of the decision region of the trained net is given by

  a/(1 + exp -(w10 + w11 * x + w12 * y + w13 * z))
+ b/(1 + exp -(w20 + w21 * x + w22 * y + w23 * z))
+ c/(1 + exp -(w30 + w31 * x + w32 * y + w33 * z))
= d

where a, b, c are weights of the output node.  If there is a bias on the
output, it is incorporated into the choice of d.

This has the general form a * u + b * v + c * w = d, where u, v, w are
in (0,1).  In u,v,w space, think of that open cube as being cut by the plane
determined by the last equation, and choose the volume on one side of the
plane.  Its boundaries are formed by part of the plane and some of the
walls of the cube.  Take each boundary, parameterized by two of the
variables, calculate the third variable, and get a u, v, w point.  To get the
boundaries of the boundary parts on the plane, you have to solve some simple
linear equations, like when u = 0 we have b * v + c * w = d etc.  This provides
constraints on the u, v, w values, but they can be computed explicitly and
used to generate u, v, w boundary values effectively.

Now for each of the points u, v , w find the corresponding point on
the sigmoids:

u = 1 / ( 1 + exp -p )
v = 1 / ( 1 + exp -q )
w = 1 / ( 1 + exp -r )

Hence p = - ln ( (1/u) - 1) etc.

Now take p, q and r and using a Gaussian elimination routine solve

p - w10 = w11 * x + w12 * y + w13 * z
q - w20 = w21 etc.
r = w30 = w31 etc.

This gives you a decision boundary made up of parameterized
(non-linear) patches of the form x(u,v), y(u,v), z(u,v), or with u and
w or v and w if more convenient for the particular patch.

If you happen to end up with a singular matrix of weights, or an
ill-conditioned problem, then be *very* careful about using the
decision boundary you have calculated.

Of course the above assumes that you will have fairly high precision
arithmetic.  If your NNs have only few bits of precision, the decision
boundary would be a nightmare to compute and use, and you should
consider other technologies, like ALNs, for which the decision
boundary can be reliably and safely designed.

Hope this helps.

Bill 

This is a nice problem.  I think my numerical methods class, CMPUT
340, will like to play with it.  Maybe they can send you some sample
graphs if you send me some weights you are interested in.
--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071