From: al@jargon.gmd.de (Alexander Linden)
Newsgroups: comp.ai.neural-nets
Subject: Re: Network Inversion
Date: 22 Aug 92 13:52:24 GMT
Organization: GMD, Sankt Augustin, Germany

demers@cs.ucsd.edu (David DeMers) writes:

>In article <BtCqHo.AtH.1@cs.cmu.edu> tjochem+@CS.CMU.EDU (Todd Jochem) writes:
>>I'm looking for references to network inversion papers. The basic idea

The method Todd asked for is gradient descent in input
space with respect to some error measure at the output.

>There has been a fair amount of work done on this.  Or at least 
>See Jordan, Michael I. & David Rumelhart, "Forward Models: Supervised
>Learning with a Distal Teacher" (probably in Cognitive Science

The work of Jordan and others David DeMers cites here IS quite related,
 but they do not work in input space rather they utilize the gradient
in input space of an action model network (or transition model) as the
error information at the output layer of a second net, the controller.
This kind of procedure has also used by Nguyen/Widrow (1989) IJCNN, and
Werbos. An earlier publication of very related stuff is

@INPROCEEDINGS{Ersue87Y13,
AUTHOR		= {Ers\"u, E. and Tolle, H.},
TITLE		= {Hierarchical Learning Control -- An Appraoch with
			Neuron-like Associative Memories},
BOOKTITLE	= {Proceedings of IEEE Conference on Neural
		Information Processing, Denver, November 8-12},
YEAR		= 1987,
EDITOR		= {Anderson, D.},
ORGANIZATION	= {IEEE},
PUBLISHER	= {AIP},
ADDRESS		= {Denver, CO},
KEYWORDS	= {associative memory | hierarchical learning control},
REF		= {Y13}
}

>Mike Dyer and Risto Mikkulainen have also looked at the
>input deltas for some purpose which I've forgotten, also
>the cite, but should be easy to find...

They have done work in the finding of proper coding schemes. Their work
is published in

@INPROCEEDINGS{Miikkulainen88Q52,
AUTHOR	= {Miikkulainen, R. and Dyer, M.~G.},
TITLE	= {Forming Global Representations with Extended
	Backpropagation},
BOOKTITLE	= {Proceedings ICNN-88, San Diego},
YEAR	= 1988,
PAGES	= {285-292},
REF	= {Q52}
}

and also in the connectionist summer school proceedings of 1989.

>Mike Rossen in NIPS 3 has a paper on closed form inversion

I did not see that this method calculates the gradient in input space.
A direct calculation of the input seems not possible because it is 
often a many to one--mapping. Also the thresholds are not considered in
his work...

A small history of inversion of neural nets:
--------------------------------------------

The first publication I have found on this topic was Williams (1986).
He trained a backprop net with the NETTALK database and inverted the
mapping by backpropagating to the input layer.

Independently to his very early work in the ancient times of 
connectionism other detected the potentials of inversion...

Miikkulainen and Dyer (1988,1989) Forming Input Representations
Kindermann and Linden (1988, 1989) Minimal Microfeature detection,
               experiments with recognition of handwritten digits.

Thrun and Linden (1990) extended inversion to recurrent neural nets (by
unfolding)

Inversion of a forward model in neurocontrol, can be used to 
generate goal directed behaviour. It can determine the action
which produces a successor state which is as close as possible to the
systems goal (due to gradient descent the can be local minima)

Thrun, Moeller and Linden (1990) derived  lookahead planning by inversion of
				forward models and applied it to 
				target tracking and cartpole ballancing.
Kawato et al (1990) developed  this idea for trajectory formation.

Hwang et al (1990) used the idea of inversion to establish query
learning. As pointed out in Linden/Kindermann (1989, IJCNN) it is easy
possible to generate counterexamples by inversion. These
counterexamples are very near at the class boundaries, thus with human
interaction (telling the correct classification and extending the
training set) the class boundaries can be moved.

Hoskins et al (1992) applied this inversion to inverse kinematics.
Also we did some experiments with this (Linden and Weber (1992)).

Thrun and Moeller (1992) and Linden and Weber (1992) did experiments 
with exploration by utilizing gradient descent in input space, where
a minimization of a competence model is used to guide some autonomous
agent into regions which are believed unknown.

Also Ron Chrisley (1990) has thought a bit on inversion in the context
of neurocontrol. Also to mention is Steven Suddarth (1989).

Here is a list of references (UNSORTED!!!):

@INPROCEEDINGS{Williams86Z23,
AUTHOR		= {Williams, R. J.},
TITLE		= {Inverting a Connectionist Network Mapping by
Backpropagation of Error},
BOOKTITLE	= {8th Annual Conference of the Cognitive Science Society},
YEAR		= 1986,
PUBLISHER	= {Lawrence Erlbaum},
ADDRESS		= {Hillsdale, NJ},
KEYWORDS	= {backpropagation | inversion | natural language},
REF		= {Z23}
}

@ARTICLE{Hoskins92PP22,
  AUTHOR	= {Hoskins, D. A. and Hwang, J. N. and Vagners, J.},
  TITLE	= {Iterative Inversion of Neural Networks and its
                 Application to Adaptive Control},
  JOURNAL	= {IEEE Transactions on Neural Networks},
  YEAR		= {1992},
  VOLUME	= {3},
  NUMBER	= {2},
  PAGES	= {292 - 301},
  KEYWORDS	= {},
  REF		= {PP22}
}

@UNPUBLISHED{Hwang90EE22,
  AUTHOR	= {Hwang, J. N. and Choi, J. J. and Oh, S. and Marks, R. J.},
  TITLE	= {Query Learning Based on Boundary Search and Gradient
		   Computation of Trained Multilayer Perceptrons},
  NOTE		= {to appear in Proceedings of IJCNN 90, San Diego, June 17-21,
		   1990},
  YEAR		= {1990},
  KEYWORDS	= {inversion},
  REF		= {EE22}
}

@INPROCEEDINGS{Hwang90EE23,
  AUTHOR	= {Hwang, J. N. and Chan, C. H.},
  TITLE	= {Iterative Constrained Inversion of Neural Networks and its Applications},
  BOOKTITLE	= {24th Conference on Information Systems and Sciences, Priceton,
		   March 1990},
  YEAR		= {1990},
  KEYWORDS	= {inversion},
  REF		= {EE23}
}

@INPROCEEDINGS{Miikkulainen88Q52,
AUTHOR	= {Miikkulainen, R. and Dyer, M.~G.},
TITLE	= {Forming Global Representations with Extended
	Backpropagation},
BOOKTITLE	= {Proceedings ICNN-88, San Diego},
YEAR	= 1988,
PAGES	= {285-292},
REF	= {Q52}
}


@INPROCEEDINGS{Linden88Correlative,
AUTHOR		= {Linden, A. and Kindermann, J.},
TITLE		= {A Correlative View on Backpropagation},
BOOKTITLE	= {Connectionism in Perspective},
PAGES	 	= {377-384},
EDITOR		= {Pfeifer, R. and Schreter, Z. and Fogelman, F. and Steels, L.},
PUBLISHER	= {Elsevier},
ADDRESS		= {Amsterdam},
YEAR		= 1989,
NOTE		= {Proceedings of the International Conference at the University
of Z{\"u}rich, 10.-13.\ October 1988}
}


@INPROCEEDINGS{Kindermann89Microfeatures,
AUTHOR		= {Kindermann, J. and Linden, A.},
TITLE		= {Detection of Minimal Microfeatures by Internal Feedback},
EDITOR		= {Retti, J. and Leidlmair, K.},
BOOKTITLE	= {5. {\"O}sterreichische Artificial-Intelligence Tagung},
YEAR		= 1989,
PUBLISHER 	= {Springer},
ADDRESS	= {Berlin, Heidelberg}
}

@INPROCEEDINGS{Linden89Inversion,
AUTHOR		= {Linden, A. and Kindermann, J.},
TITLE		= {Inversion of Multilayer Nets},
BOOKTITLE	= {Proceedings of the First International Joint Conference on
		   Neural Networks, Washington DC},
PUBLISHER	= {IEEE},
ADDRESS		= {San Diego},
YEAR		= 1989,
REF		= {Inversion}
}


@MASTERSTHESIS{Linden90diplom,
 AUTHOR	= {Linden, A.},
  TITLE	= {{U}ntersuchung von {B}ackpropagation in konnektionistischen
		   {S}ystemen}, 
  SCHOOL	= {Universit{\"a}t Bonn, Informatik-Institutsbericht Nr. 80},
  TYPE		= {Diplomarbeit},
  YEAR		= {1990},
  REF		= {diplom}
}

@ARTICLE{Kindermann90PC,
  AUTHOR	= {Kindermann, J. and Linden, A.},
  TITLE	= {Inversion of Neural Networks by Gradient Descent},
  JOURNAL	= {Journal of Parallel Computing},
  YEAR		= {1990},
  VOLUME	= {14},
  NUMBER	= {3},
  PAGES	= {277-286},
  KEYWORDS	= {},
  REF		= {PC}
}

@INPROCEEDINGS{Thrun90Inversion,
AUTHOR		= {Thrun, S. and Linden, A.},
TITLE		= {Inversion in Time},
BOOKTITLE	= {Proceedings of the EURASIP Workshop on Neural Networks},
ADDRESS	= {Sesimbra, Portugal},
YEAR		= 1990,
REF		= {Inversion}
}

@INPROCEEDINGS{Thrun90Mental,
  AUTHOR	= {Thrun, S. and Linden, A. and M{\"o}ller, K.},
  TITLE	= {Planning with a mental modell},
  BOOKTITLE	= {Proceedings of NeuroNimes 90, France},
  YEAR		= {1990},
  REF		= {Mental}
}

@INPROCEEDINGS{Thrun91NIPS,
  AUTHOR	= {Thrun, S. and M{\"o}ller, K. and Linden, A.},
  TITLE	= {Planning with an Adaptive World Model},
  BOOKTITLE	= {Advances in Neural Information Processing Systems 3},
  YEAR		= {1991},
  EDITOR	= {Touretzky, D. S. and Lippmann, R.},
  PAGES	= {},
  ORGANIZATION	= {},
  PUBLISHER	= {Morgan Kaufmann},
  ADDRESS	= {San Mateo},
  MONTH	= {},
  KEYWORDS	= {},
  REF		= {NIPS}
}

@INCOLLECTION{Kindermann92IEEE,
  AUTHOR	= {Kindermann, J. and Linden, A.},
  TITLE	= {Inversion of Neural Networks by Gradient Descent},
  BOOKTITLE	= {Artificial Neural Networks: Concepts and Control
		   Applications},
  EDITOR	= {Vemuri, R.},
  PUBLISHER	= {IEEE Computer Society Press},
  YEAR		= {1992},
  REF		= {Kindermann92IEEE}
}


@INPROCEEDINGS{Linden92AIREADER,
  AUTHOR	= {Linden, A. and Weber, F.},
  TITLE	= {Exploring the Kinematics of a Robot Arm by Competence
                 Modelling },
  EDITOR      = {Becker, B. and Gordon, T.},
  BOOKTITLE	= {AI-Reader},
  PUBLISHER = {Erscheint als Arbeitspapier der GMD}, 
  YEAR		= {1992}
}

@INPROCEEDINGS{Thrun92OO8,
  AUTHOR	= {Thrun, S. and M\"oller, K.},
  TITLE	= {Active Exploration in Dynamic Environments},
  BOOKTITLE	= {Advances in Neural Information Processing Systems 4},
  YEAR		= {1992},
  EDITOR	= {Moody, J. E. and Hanson, S. J. and Lippmann, R. P.},
  PAGES	= {},
  ORGANIZATION	= {},
  PUBLISHER	= {Morgan Kaufmann},
  ADDRESS	= {},
  MONTH	= {},
  KEYWORDS	= {},
  REF		= {OO8}
}

@UNPUBLISHED{Suddarth89Z4,
AUTHOR		= {Suddarth, S. C. and Bourrely, J. C.},
TITLE		= {A Back-Propagation Associative Memory for Both Positive and
Negative Learning},
YEAR		= 1989,
NOTE		= {Poster Presentation at IJCNN-89},
KEYWORDS	= {backpropagation | associative memory | negative training},
REF		= {Z4}
}


@ARTICLE{Kawato90JJ16,
  AUTHOR	= {Kawato, M. and Maeda, Y. and Uno, Y. and Suzuki, R.},
  TITLE	= {Trajectory Formation of Arm Movement by Cascade
		   Neural Network Model Based on Minimum Torque-Change
		   Criterion},
  JOURNAL	= {Biological Cybernetics},
  YEAR		= {1990},
  VOLUME	= {62},
  NUMBER	= {},
  PAGES	= {275 - 288},
  KEYWORDS	= {},
  REF		= {JJ16}
}


@UNPUBLISHED{Chrisley90JJ19,
  AUTHOR	= {Chrisley, R.~L.},
  TITLE	= {Cognitive Map Construction and Use: A Parallel
		   Distributed Processing Approach},
  NOTE		= {To appear in:  Touretzky, D.~S. and Elman, J.~L.,
		   Sejnowski, T.~J. and Hinton, G.~E. (eds.) (1990)
		   Proceedings of the 1990 Connectionist Models Summer
		   School. San Mateo, CA: Morgan Kaufmann.},
  YEAR		= {1990},
  KEYWORDS	= {},
  REF		= {JJ19}
}



@INPROCEEDINGS{Rossen90NN31,
  AUTHOR	= {Rossen, M. L.},
  TITLE	= {Closed-Form Inversion of Backpropagation Networks: Theory
           and Optimiation Issues},
  BOOKTITLE	= {Conference for Neural Information Processing (NIPS)},
  YEAR		= {1990},
  EDITOR	= {},
  PAGES	= {},
  ORGANIZATION	= {},
  PUBLISHER	= {},
  ADDRESS	= {Denver},
  MONTH	= {November},
  KEYWORDS	= {},
  REF		= {NN31}
}
--
Alexander Linden             | TEL. (49 or 0) 2241/14-2435, FAX. -2618 or -2889
GMD - AI Research Division   | TELEX 889469 gmd d
P. O. BOX 1316               | email:  A.Linden@gmd.de
D-5205 Sankt Augustin 1, FRG