mtybilly/Instruct2DS huggingface dataset:
Perceiving Complex Visual Scenes: An Oscillator Neural Network Model that Integrates Selective Attention, Perceptual Organisation, and Invariant Recognition
Authors: Rainer Goebel
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Which processes underly our ability to quickly recognize familiar  objects within a complex visual input scene? In this paper an imple(cid:173) mented neural network model is described that attempts to specify  how selective visual attention, perceptual organisation, and invari(cid:173) ance transformations might work together in order to segment, select,  and recognize objects out of complex input scenes containing multi(cid:173) ple, possibly overlapping objects. Retinotopically organized feature  maps serve as input for two main processing routes:  pathway' dealing with location information and the 'what-pathway'  computing the shape and attributes of objects. A location-based at(cid:173) tention mechanism operates on an early stage of visual processing  selecting a contigous region of the visual field for preferential proces(cid:173) sing. Additionally, location-based attention plays an important role  for invariant object recognition controling appropriate normalization  processes within the what-pathway. Object recognition is supported  through the segmentation of the visual field into distinct entities. In  order to represent different segmented entities at the same time, the  model uses an oscillatory binding mechanism. Connections between  the where-pathway and the what-pathway lead to a flexible coope(cid:173) ration between different functional subsystems producing an overall  behavior which is consistent with a variety of psychophysical data. 
the 'where(cid:173)
Second order derivatives for network pruning: Optimal Brain Surgeon
Authors: David G. Stork Babak Hassibi
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We investigate the use of information from all second order derivatives of the error  function to perfonn network pruning (i.e., removing unimportant weights from a trained  network) in order to improve generalization, simplify networks, reduce hardware or  storage requirements, increase the speed of further training, and in some cases enable rule  extraction. Our method, Optimal Brain Surgeon (OBS), is Significantly better than  magnitude-based methods and Optimal Brain Damage [Le Cun, Denker and Sol1a, 1990],  which often remove the wrong weights. OBS permits the pruning of more weights than  other methods (for the same error on the training set), and thus yields better  generalization on test data. Crucial to OBS is a recursion relation for calculating the  inverse Hessian matrix H-I from training data and structural information of the net. OBS  permits a 90%, a 76%, and a 62% reduction in weights over backpropagation with weighL  decay on three benchmark MONK's problems [Thrun et aI., 1991]. Of OBS, Optimal  Brain Damage, and magnitude-based methods, only OBS deletes the correct weights from  a trained XOR network in every case. Finally, whereas Sejnowski and Rosenberg [1987J  used 18,000 weights in their NETtalk network, we used OBS to prune a network to just  1560 weights, yielding better generalization.
Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors
Authors: Patrice Y. Simard, Barak Pearlmutter Yann LeCun
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We propose a very simple, and well principled way of computing  the optimal step size in gradient descent algorithms. The on-line  version is very efficient computationally, and is applicable to large  backpropagation networks trained on large data sets. The main  ingredient is a technique for estimating the principal eigenvalue(s)  and eigenvector(s) of the objective function's second derivative ma(cid:173) trix (Hessian), which does not require to even calculate the Hes(cid:173) sian. Several other applications of this technique are proposed for  speeding up learning, or for eliminating useless parameters.
Computation of Heading Direction from Optic Flow in Visual Cortex
Authors: Josef P. Rauschecker Markus Lappe
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have designed a neural network which detects the direction of ego(cid:173) motion from optic flow in the presence of eye movements (Lappe and  Rauschecker, 1993). The performance of the network is consistent with  human psychophysical data, and its output neurons show great similarity  to "triple component" cells in area MSTd of monkey visual cortex. We  now show that by using assumptions about the kind of eye movements  that the obsenrer is likely to perform, our model can generate various  other cell types found in MSTd as well.
Statistical Modeling of Cell Assemblies Activities in Associative Cortex of Behaving Monkeys
Authors: Naftali Tishby Itay Gat
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: So far there has been no general method for relating extracellular  electrophysiological measured activity of neurons in the associative  cortex to underlying network or "cognitive" states. We propose  to model such data using a multivariate Poisson Hidden Markov  Model. We demonstrate the application of this approach for tem(cid:173) poral segmentation of the firing patterns, and for characterization  of the cortical responses to external stimuli. Using such a statisti(cid:173) cal model we can significantly discriminate two behavioral modes  of the monkey, and characterize them by the different firing pat(cid:173) terns, as well as by the level of coherency of their multi-unit firing  activity.  Our study utilized measurements carried out on behaving Rhesus  monkeys by M. Abeles, E. Vaadia, and H. Bergman, of the Hadassa  Medical School of the Hebrew University.
Harmonic Grammars for Formal Languages
Authors: Paul Smolensky
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Basic connectionist principles imply that grammars should take the  form of systems of parallel soft constraints defining an optimization  problem  the  solutions  to  which  are  the  well-formed  structures  in  the  language.  Such  Harmonic  Grammars  have  been  successfully  applied to a number of problems in the theory of natural languages.  Here  it  is  shown  that  formal  languages  too  can  be  specified  by  Harmonic  Grammars,  rather  than  by  conventional  serial  re-write  rule systems. 
1  HARMONIC  GRAMMARS 
In collaboration with Geraldine Legendre,  Yoshiro Miyata, and Alan Prince, I have  been  studying  how  symbolic computation  in  human  cognition  can  arise  naturally  as a higher-level  virtual machine realized  in appropriately designed  lower-level  con(cid:173) nectionist  networks.  The basic  computational principles of the approach are  these: 
(1) 
a.  \Vhell  analyzed  at  the  lower  level,  mental  representations  are  dis(cid:173)
tributed  patterns of connectionist  activity;  when  analyzed  at a  higher  level,  these  same representations  constitute symbolic structures.  The  particular  symbolic  structure  s  is  characterized  as  a  set  of filler/role  bindings  {f d ri},  using  a  collection  of structural  roles  {rd  each  of  which  may  be  occupied  by  a  filler  fi-a constituent  symbolic  struc-
Silicon Auditory Processors as Computer Peripherals
Authors: John Wawrzynek, M. Mahowald, Massimo Sivilotti, Dave Gillespie John Lazzaro
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Sever

Present address: f\1. Mahowald, f\1H.C ,\natol1lical Ncmophamacology  Unit, Mansfield TId, Oxfc)('d OXI :1'rl£ Ellgland . mam~vax.oxford.ac.uk  t Present address:  f\lass Siviloui. '1'(1111)(,1' H,csearrh, 180 Nort.h Vinedo  Avenue, Pasadena, CA 9Il07. mass~tanner. corn  :I: Present address: Dave Gill,>spiE', SYllapf,ics, :l()!)8 Orchard Parkway, San  Jose CA, 95131. daveg~synaptics. com
Information, Prediction, and Query by Committee
Authors: H. Sebastian Seung, Eli Shamir, Naftali Tishby Yoav Freund
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  analyze the  "query  by  committee"  algorithm, a  method for  fil(cid:173) tering  informative  queries  from  a  random  stream  of inputs.  We  show  that if the  two-member committee algorithm achieves  infor(cid:173) mation  gain  with  positive  lower  bound,  then  the  prediction  error  decreases  exponentially with the number of queries.  We show that,  in  particular,  this  exponential  decrease  holds for  query  learning of  thresholded  smooth functions.
Improving Convergence in Hierarchical Matching Networks for Object Recognition
Authors: Gene Gindi Joachim Utans
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  are  interested  in  the  use  of analog  neural  networks  for  recog(cid:173) nizing  visual  objects.  Objects  are  described  by  the  set  of parts  they  are  composed  of  and  their  structural  relationship.  Struc(cid:173) tural  models  are  stored  in  a  database  and  the  recognition  prob(cid:173) lem  reduces  to  matching  data  to  models  in  a  structurally  consis(cid:173) tent  way.  The  object  recognition  problem is  in  general  very  diffi(cid:173) cult in that it involves coupled problems of grouping, segmentation  and  matching.  We  limit the  problem here  to  the simultaneous la(cid:173) belling  of the  parts  of a  single  object  and  the  determination  of  analog  parameters.  This  coupled  problem  reduces  to  a  weighted  match  problem in  which  an  optimizing neural  network  must  min(cid:173) imize  E(M, p)  =  LO'i MO'i WO'i(p),  where  the  {MO'd  are  binary  match  variables  for  data parts  i  to  model  parts  a  and  {Wai(P)}  are weights dependent on parameters p .  In this work we  show that  by  first  solving  for  estimates p without  solving  for  M ai , we  may  obtain good initial parameter estimates that yield  better solutions  for  M  and  p. 
*Current  address: 
International  Computer  Science  Institute,  1947  Center  Street, 
Suite  600,  Berkeley,  CA 94704,  utans@icsi.berkeley.edu 
tCurrent  address:  SUNY  Stony  Brook,  Department  of  Electrical  Engineering,  Stony 
Brook,  NY  11784
A dynamical model of priming and repetition blindness
Authors: Michael I. Jordan Daphne Bavelier
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We describe a model of visual word recognition that accounts for  several aspects of the temporal processing of sequences of briefly  presented words. The model utilizes a new representation for writ(cid:173) ten words, based on dynamic time warping and multidimensional  scaling. The visual input passes through cascaded perceptual, com(cid:173) parison, and detection stages. We describe how these dynamical  processes can account for several aspects of word recognition, in(cid:173) cluding repetition priming and repetition blindness.
Unsupervised Discrimination of Clustered Data via Optimization of Binary Information Gain
Authors: Terrence J. Sejnowski Nicol N. Schraudolph
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  present the  information-theoretic derivation  of a learning algorithm  that  clusters  unlabelled  data  with  linear  discriminants.  In  contrast  to  methods  that  try  to  preserve  information  about  the  input patterns,  we  maximize  the  information  gained  from  observing  the  output  of robust  binary discriminators implemented with sigmoid nodes. We deri ve a local  weight adaptation rule via gradient ascent in this objective, demonstrate  its  dynamics on  some simple data sets,  relate our approach  to  previous  work and suggest directions in which it may be extended.
On-Line Estimation of the Optimal Value Function: HJB- Estimators
Authors: James K. Peterson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In this paper, we discuss on-line estimation strategies that model  the optimal value function of a typical optimal control problem.  We present a general strategy that uses local corridor solutions  obtained via dynamic programming to provide local optimal con(cid:173) trol sequence training data for a neural architecture model of the  optimal value function. 
ION-LINE ESTIMATORS 
In this paper, the problems of adaptive control using neural architectures are ex(cid:173) plored in the setting of general on-line estimators. 'Ve will try to pay close attention  to the underlying mathematical structure that arises in the on-line estimation pro(cid:173) cess. 
The complete effect of a control action Uk at a given time step t/.; is clouded by  the fact that the state history depends on the control actions taken after time  step tk' So the effect of a control action over all future time must be monitored.  Hence, choice of control must inevitably involve knowledge of the future history  of the state trajectory. In other words, the optimal control sequence can not be  determined until after the fact. Of course, standard optimal control theory supplies  an optimal control sequence to this problem for a variety of performance criteria.  Roughly, there are two approaches of interest: solving the two-point boundary value
Using Aperiodic Reinforcement for Directed Self-Organization During Development
Authors: P. Dayan, S.J. Nowlan, A Pouget, T.J. Sejnowski P. R. Montague
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present a local learning rule in which Hebbian learning is  conditional on an incorrect prediction of a reinforcement signal.  We propose a biological interpretation of such a framework and  display its utility through examples in which the reinforcement  signal is cast as the delivery of a neuromodulator to its target.  Three exam pIes are presented which illustrate how this framework  can be applied to the development of the oculomotor system.
A Connectionist Symbol Manipulator That Discovers the Structure of Context-Free Languages
Authors: Sreerupa Das Michael Mozer
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present a neural net architecture that can discover hierarchical and re(cid:173) cursive structure in symbol strings. To detect structure at multiple levels,  the architecture has the capability of reducing symbols substrings to single  symbols, and makes use of an external stack memory. In terms of formal  languages, the architecture can learn to parse strings in an LR(O) context(cid:173) free grammar. Given training sets of positive and negative exemplars,  the architecture has been trained to recognize many different grammars.  The architecture has only one layer of modifiable weights, allowing for a  straightforward interpretation of its behavior. 
Many cognitive domains involve complex sequences that contain hierarchical or  recursive structure, e.g., music, natural language parsing, event perception. To il(cid:173) lustrate, "the spider that ate the hairy fly" is a noun phrase containing the embed(cid:173) ded noun phrase "the hairy fly." Understanding such multilevel structures requires  forming reduced descriptions (Hinton, 1988) in which a string of symbols or states  ("the hairy fly") is reduced to a single symbolic entity (a noun phrase). We present  a neural net architecture that learns to encode the structure of symbol strings via  such red uction transformations. 
The difficult problem of extracting multilevel structure from complex, extended  sequences has been studied by Mozer (1992), Ring (1993), Rohwer (1990), and  Schmidhuber (1992), among others. While these previous efforts have made some
Analogy-- Watershed or Waterloo? Structural alignment and the development of connectionist models of analogy
Authors: Arthur B. Markman Dedre Gentner
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Neural network models have been criticized for their inability to make  use of compositional representations. In this paper, we describe a series  of psychological phenomena that demonstrate the role of structured  representations in cognition. These findings suggest that people  compare relational representations via a process of structural alignment.  This process will have to be captured by any model of cognition,  symbolic or subsymbolic.
A Recurrent Neural Network for Generation of Occular Saccades
Authors: Lina L.E. Massone
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper presents a neural network able to control saccadic  movements. The input to the network is a specification of a  stimulation site on the collicular motor map. The output is the time  course of the eye position in the orbit (horizontal and vertical angles).  The units in the network exhibit a one-to-one correspondance with  neurons in the intermediate layer of the superior colliculus (collicular  motor map), in the brainstem and with oculomotor neurons.  Simulations carried out with this network demonstrate its ability to  reproduce in a straightforward fashion many experimental observations.
Network Structuring and Training Using Rule-based Knowledge
Authors: Jürgen Hollatz, Subutai Ahmad Volker Tresp
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We demonstrate in this paper how certain forms of rule-based  knowledge can be used to prestructure a neural network of nor(cid:173) malized basis functions and give a probabilistic interpretation of  the network architecture. We describe several ways to assure that  rule-based knowledge is preserved during training and present a  method for complexity reduction that tries to minimize the num(cid:173) ber of rules and the number of conjuncts. After training the refined  rules are extracted and analyzed.
Hybrid Circuits of Interacting Computer Model and Biological Neurons
Authors: Gwendal Le Masson, Eve Marder, L. F. Abbott Sylvie Renaud-Le Masson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We demonstrate the use of a digital signal processing board to construct  hybrid networks consisting of computer model neurons connected to a  biological neural  network.  This system  operates  in  real  time.  and  the  synaptic  connections  are  realistic  effective  conductances.  Therefore.  the  synapses  made  from  the  computer  model  neuron  are  integrated  correctly  by  the postsynaptic biological neuron.  This method provides  us  with  the  ability  to  add  additional.  completely known elements  to a  biological  network  and  study  their  effect  on  network  activity.  Moreover.  by  changing  the  parameters  of  the  model  neuron.  it  is  possible to assess  the role of individual conductances in  the activity of  the  neuron.  and  in  the  network in which it participates. 
"Present  address. 
l-Enserb.  CNRSURA  846.  351  crs  de  la  Liberation.  33405  Talence  Cedex.  France. 
lXL.  Universit6  de  Bordeaux 
'Present address.  LNPC.  CNRS.  Universite de Bordeaux  1.  Place 
de Dr.  Peyneau.  33120 Arcachon.  France
A Knowledge-Based Model of Geometry Learning
Authors: Richard Lehrer Geoffrey Towell
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We propose a model of the development of geometric reasoning in children that  explicitly involves learning. The model uses a neural network that is initialized  with an understanding of geometry similar to that of second-grade children.  Through the presentation of a series of examples, the model is shown to develop  an understanding of geometry similar to that of fifth-grade children who were  trained using similar materials.
Integration of Visual and Somatosensory Information for Preshaping Hand in Grasping Movements
Authors: Naohiro Fukumura, Ryoji Suzuki, Mitsuo Kawato Yoji Uno
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The primate brain must solve two important problems in grasping move(cid:173) ments.  The  first  problem  concerns  the  recognition  of grasped objects:  specifically,  how  does  the  brain integrate  visual  and motor information  on a grasped object? The second problem concerns hand shape planning:  specifically, how does the brain design the hand configuration suited to the  shape of the  object and the manipulation task?  A neural network model  that solves these problems has been developed.  The operations of the net(cid:173) work are divided into a learning phase and an optimization phase.  In the  learning phase, internal representations, which depend on the grasped ob(cid:173) jects and the task,  are  acquired by integrating visual and somatosensory  information.  In the optimization phase, the  most suitable hand shape for  grasping an object is determined by using a relaxation computation of the  network. 

Present  Address: 

Parallel  Distributed  Processing  Research  Dept.,  Sony  Corporation, 
6-7-35 Kitashinagawa, Shinagawa-ku, Tokyo 141, Japan
Kohonen Feature Maps and Growing Cell Structures - a Performance Comparison
Authors: Bernd Fritzke
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A performance comparison of two self-organizing networks,  the Ko(cid:173) honen  Feature Map  and the recently  proposed Growing Cell Struc(cid:173) tures  is  made.  For  this  purpose  several  performance  criteria  for  self-organizing  networks  are  proposed  and  motivated.  The models  are tested with three example problems of increasing difficulty.  The  Kohonen  Feature  Map  demonstrates slightly superior  results  only  for  the simplest problem.  For the other more difficult and also more  realistic problems the Growing Cell Structures exhibit significantly  better  performance  by  every  criterion .  Additional  advantages  of  the new  model are  that  all  parameters are constant  over  time and  that size  as  well  as  structure  of the  network  are  determined  auto(cid:173) matically.
A Neural Model of Descending Gain Control in the Electrosensory System
Authors: Mark E. Nelson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In  the  electrosensory  system  of  weakly  electric  fish,  descending  pathways to a first-order sensory nucleus have been shown to influ(cid:173) ence the gain of its output neurons.  The underlying neural mecha(cid:173) nisms  that subserve this descending gain control capability  are not  yet  fully  understood.  We  suggest  that  one  possible  gain  control  mechanism could involve the regulation of total membrane conduc(cid:173) tance of the  output neurons.  In this  paper,  a  neural model  based  on this idea is  used to demonstrate how  activity levels on descend(cid:173) ing  pathways  could  control  both  the  gain  and  baseline  excitation  of a  target neuron .
Memory-Based Reinforcement Learning: Efficient Computation with Prioritized Sweeping
Authors: Christopher G. Atkeson Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present a  new algorithm, Prioritized Sweeping, for  efficient  prediction  and control of stochastic Markov systems.  Incremental learning methods  such as Temporal Differencing  and  Q-Iearning have fast  real time perfor(cid:173) mance.  Classical  methods  are  slower,  but  more  accurate,  because  they  make full  use  of the observations.  Prioritized Sweeping aims for  the  best  of both worlds.  It uses  all  previous  experiences both to  prioritize  impor(cid:173) tant dynamic  programming sweeps and to guide the exploration of state(cid:173) space.  We compare Prioritized Sweeping with other reinforcement learning  schemes for  a  number of different stochastic optimal control problems.  It  successfully  solves large state-space  real  time  problems  with  which  other  methods have difficulty. 
1  STOCHASTIC PREDICTION 
The paper introduces a memory-based technique, prioritized 6weeping, which is used  both for  stochastic  prediction and  reinforcement  learning.  A fuller  version  of this  paper is in preparation [Moore and Atkeson, 1992].  Consider the  500 state Markov  system depicted in Figure 1.  The system has sixteen absorbing states,  depicted by  white and black circles.  The prediction problem is  to estimate, for  every state, the  long-term  probability  that  it will  terminate  in  a  white,  rather  than  black,  circle.  The data available to the learner is a sequence of observed state transitions.  Let us  consider two existing methods along  with prioritized sweeping.
Learning Fuzzy Rule-Based Neural Networks for Control
Authors: Rodney M. Goodman Charles M. Higgins
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A  three-step  method for  function  approximation with  a fuzzy  sys(cid:173) tem  is  proposed.  First,  the  membership  functions  and  an  initial  rule  representation  are  learned;  second,  the  rules  are  compressed  as  much as  possible  using  information theory;  and finally,  a  com(cid:173) putational network  is  constructed  to  compute  the  function  value.  This system  is  applied  to  two control examples:  learning the truck  and trailer backer-upper control system, and learning a  cruise  con(cid:173) trol system for  a  radio-controlled model car.
Neural Network On-Line Learning Control of Spacecraft Smart Structures
Authors: Christopher Bowman
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The overall goal is to reduce spacecraft weight. volume, and cost by on(cid:173) line adaptive non-linear control of flexible structural components. The  objective of this effort is to develop an  adaptive Neural Network (NN)  controller for the Ball C-Side 1m x 3m antenna with embedded actuators  and the RAMS  sensor system.  A traditional  optimal controller for  the  major modes  is provided perturbations  by  the  NN  to compensate  for  unknown residual modes. On-line training of recurrent and feed-forward  NN  architectures  have  achieved  adaptive  vibration  control  with  unknown  modal  variations and  noisy  measurements.  On-line training  feedback to each actuator NN output is computed via Newton's method  to reduce the difference between desired and achieved antenna positions. 
1  ADAPTIVE  CONTROL  BACKGROUND  The two traditional approaches to adaptive control are 1) direct control (such as perfonned  in direct model reference adaptive controllers) and 2) indirect control (such as performed by  explicit self-tuning regulators).  Direct control techniques (e.g. model-reference adaptive  cootrul) provide good stability however are susceptible to noise. Whereas indirect control  techn;'q~es (e.g.  explicit self-tuning  regulators)  have low  noise susceptibility  and good  convergence rate.  However they require more control effort and have worse stability and  are less roblistto mismodeling. NNs synergistically augment traditional adaptive control  techniques  by  providing  improved mismodeling  robustness both adaptively on-line for  time-varying dynamics as well as in a learned control mode at a slower rate. 
The NN control approaches which correspond to direct and indirect adaptive control are  commonly known as inverse and forward modeling. respectively. More specifically, aNN  which maps the plant state and its desired perfonnance to the control command is called  an inverse model, a NN mapping both the current plant state and control to the next state  and its performance is called the forward model. 
When given a desired performwce and the current state. the inverse model generates the  control. see Figure 1. The actual perfonnance is observed and is used to  train/update the  inverse model. A significant problem occurs when the desired and achieved perfonnance  differ greatly  since the  model  near  the desired  slate  is  not changed.  This condition  is  corrected by  adding random noise to  the control outputs  so as to  extend the state  space
Hidden Markov Model Induction by Bayesian Model Merging
Authors: Stephen Omohundro Andreas Stolcke
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper describes a technique for learning both the number of states and the  topology of Hidden Markov Models from examples. The induction process starts  with the most specific model consistent with the training data and generalizes  by successively merging states. Both the choice of states to merge and the  stopping criterion are guided by the Bayesian posterior probability. We compare  our algorithm with the Baum-Welch method of estimating fixed-size models, and  find that it can induce minimal HMMs from data in cases where fixed estimation  does not converge or requires redundant parameters to converge. 
1 INTRODUCTION AND OVERVIEW 
Hidden Markov Models (HMMs) are a well-studied approach to the modelling of sequence  data. HMMs can be viewed as a stochastic generalization of finite-state automata, where  both the transitions between states and the generation of output symbols are governed by  probability distributions. HMMs have been important in speech recognition (Rabiner &  Juang, 1986), cryptography, and more recently in other areas such as protein classification  and alignment (Haussler, Krogh, Mian & SjOlander, 1992; Baldi, Chauvin, Hunkapiller &  McClure, 1993). 
Practitioners have typically chosen the HMM topology by hand, so that learning the HMM  from sample data means estimating only a fixed number of model parameters. The standard  approach is to find a maximum likelihood (ML) or maximum a posteriori probability (MAP)  estimate of the HMM parameters. The Baum-Welch algorithm uses dynamic programming
Statistical Mechanics of Learning in a Large Committee Machine
Authors: John A. Hertz Holm Schwarze
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  use  statistical  mechanics  to study  generalization in  large com(cid:173) mittee  machines.  For  an  architecture  with  nonoverlapping  recep(cid:173) tive fields a  replica calculation yields the generalization error in the  limit of a  large number of hidden units.  For continuous weights  the  generalization  error  falls  off asymptotically inversely  proportional  to  Q,  the  number  of  training  examples  per  weight.  For  binary  weights  we  find  a  discontinuous  transition  from  poor  to  perfect  generalization  followed  by  a  wide  region  of metastability.  Broken  replica symmetry is  found  within this  region  at low  temperatures.  For  a  fully  connected  architecture  the  generalization  error  is  cal(cid:173) culated  within  the  annealed  approximation.  For  both  binary  and  continuous  weights  we  find  transitions  from  a  symmetric state  to  one  with  specialized  hidden  units,  accompanied  by  discontinuous  drops  in  the generalization error.
An Information-Theoretic Approach to Deciphering the Hippocampal Code
Authors: Bruce L. McNaughton, Katalin M. Gothard William E. Skaggs
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Information  theory  is  used  to  derive  a  simple  formula  for  the  amount of information conveyed by the firing rate of a neuron about  any  experimentally measured variable or  combination of variables  (e.g.  running speed,  head  direction,  location of the  animal, etc.).  The  derivation  treats  the  cell  as  a  communication channel  whose  input is the measured variable and whose output is the cell's spike  train.  Applying the formula,  we  find  systematic differences  in  the  information  content  of hippocampal  "place  cells"  in  different  ex(cid:173) perimental conditions.
Using hippocampal 'place cells' for navigation, exploiting phase coding
Authors: John O'Keefe, Michael Recce Neil Burgess
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A model of the hippocampus as a central element in rat naviga(cid:173) tion is presented. Simulations show both the behaviour of single  cells and the resultant navigation of the rat. These are compared  with single unit recordings and behavioural data. The firing of  CAl place cells is simulated as the (artificial) rat moves in an en(cid:173) vironment. This is the input for a neuronal network whose output,  at each theta (0) cycle, is the next direction of travel for the rat.  Cells are characterised by the number of spikes fired and the time  of firing with respect to hippocampal 0 rhythm. 'Learning' occurs  in 'on-off' synapses that are switched on by simultaneous pre- and  post-synaptic activity. The simulated rat navigates successfully to  goals encountered one or more times during exploration in open  fields. One minute of random exploration of a 1m2 environment  allows navigation to a newly-presented goal from novel starting po(cid:173) sitions. A limited number of obstacles can be successfully avoided.
Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria
Authors: John E. Moody Todd K. Leen
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  ensemble  dynamics  of stochastic  learning  algorithms  can  be  studied  using  theoretical  techniques  from  statistical  physics.  We  develop  the equations  of motion for  the  weight  space  probability  densities for  stochastic learning  algorithms.  We  discuss  equilibria  in the diffusion  approximation and provide expressions for  special  cases  of the LMS  algorithm.  The equilibrium  densities  are not in  general thermal (Gibbs) distributions in the objective function be(cid:173) ing minimized, but rather depend upon an effective potential that  includes  diffusion  effects.  Finally  we  present  an  exact  analytical  expression for the time evolution of the density for a learning algo(cid:173) rithm with weight updates proportional to the sign of the gradient. 
1 
Introduction:  Theoretical Framework 
Stochastic learning algorithms involve  weight  updates of the form 
w(n+1)  =  w(n)  + /-l(n)H[w(n),x(n)] 
(1) 
where w E 7£m  is  the vector of m  weights,  /-l  is  the learning rate, H[.]  E 7£m  is  the  update function,  and  x(n)  is  the exemplar  (input or  input/target pair)  presented
Discriminability-Based Transfer between Neural Networks
Authors: L. Y. Pratt
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Previously, we have introduced the idea of neural network transfer,  where learning on a target problem is sped up by using the weights  obtained from a network trained for a related source task. Here,  we present a new algorithm. called Discriminability-Based Transfer  (DBT), which uses an information measure to estimate the utility  of hyperplanes defined by source weights in the target network,  and rescales transferred weight magnitudes accordingly. Several  experiments demonstrate that target networks initialized via DBT  learn significantly faster than networks initialized randomly.
Learning Sequential Tasks by Incrementally Adding Higher Orders
Authors: Mark Ring
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: An incremental, higher-order, non-recurrent network combines two  properties found to be useful for learning sequential tasks: higher(cid:173) order connections and incremental introduction of new units. The  network adds higher orders when needed by adding new units that  dynamically modify connection weights. Since the new units mod(cid:173) ify the weights at the next time-step with information from the  previous step, temporal tasks can be learned without the use of  feedback, thereby greatly simplifying training. Furthermore, a the(cid:173) oretically unlimited number of units can be added to reach into  the arbitrarily distant past. Experiments with the Reber gram(cid:173) mar have demonstrated speedups of two orders of magnitude over  recurrent networks.
Topography and Ocular Dominance with Positive Correlations
Authors: Geoffrey J. Goodhill
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A new computational model that addresses the formation of both topog- 
raphy and ocular dominance is presented. This is motivated by exper- 
imental evidence that these phenomena may be subserved by the same 
mechanisms. An important aspect of this model is that ocular domi- 
nance segregation can occur when input activity is both distributed, and 
positively correlated between the eyes. This allows investigation of the 
dependence of the pattern of ocular dominance stripes on the degree of 
correlation between the eyes: it is found that increasing correlation leads 
to narrower stripes. Experiments are suggested to test whether such be- 
haviour occurs in the natural system.
A Hybrid Neural Net System for State-of-the-Art Continuous Speech Recognition
Authors: Y. Zhao, R. Schwartz, J. Makhoul G. Zavaliagkos
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Untill recently, state-of-the-art, large-vocabulary, continuous speech  recognition (CSR) has employed Hidden Markov Modeling (HMM)  to model speech sounds.  In an attempt to improve over HMM we  developed a hybrid system that integrates HMM technology with neu(cid:173) ral networks. We present the concept of a "Segmental Neural Net"  (SNN) for phonetic modeling in CSR. By taking into account all the  frames of a phonetic segment simultaneously, the SNN overcomes the  well-known conditional-independence limitation of HMMs. In several  speaker-independent experiments with the DARPA Resource Manage(cid:173) ment corpus, the hybrid system showed a consistent improvement in  performance over the baseline HMM system.
A Formal Model of the Insect Olfactory Macroglomerulus: Simulations and Analytic Results
Authors: David Marsan, Claudine Masson, Michel Kerszberg, Gérard Dreyfus, Léon Personnaz Christiane Linster
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: It  is  known  from  biological  data  that  the  response  patterns  of  interneurons  in  the olfactory  macroglomerulus  (MGC) of insects are of  central importance for the coding of the olfactory signal. We propose an  analytically  tractable  model  of the  MGC  which allows us  to  relate  the  distribution of response patterns to the architecture of the network.
Using Prior Knowledge in a NNPDA to Learn Context-Free Languages
Authors: C. Lee Giles, Guo-Zheng Sun Sreerupa Das
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Although considerable interest has been shown in language inference and  automata induction using recurrent neural networks, success of these  models has mostly been limited to regular languages. We have previ(cid:173) ously demonstrated that Neural Network Pushdown Automaton (NNPDA)  model is capable of learning deterministic context-free languages (e.g.,  anbn and parenthesis languages) from examples. However, the learning  task is computationally intensive. In this paper we discus some ways in  which a priori knowledge about the task and data could be used for efficient  learning. We also observe that such knowledge is often an experimental  prerequisite for learning nontrivial languages (eg. anbncbmam ).
A Method for Learning From Hints
Authors: Yaser S. Abu-Mostafa
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  address  the  problem  of  learning  an  unknown  function  by  pu tting together several pieces of information (hints) that we  know  about the function.  We introduce a  method that generalizes learn(cid:173) ing from examples to learning from  hints.  A  canonical representa(cid:173) tion  of hints  is  defined and illustrated for  new  types of hints.  All  the hints are represented to the learning process by examples, and  examples of the function are treated on equal footing with the rest  of  the  hints.  During  learning,  examples  from  different  hints  are  selected  for  processing according to a  given  schedule.  We  present  two types of schedules; fixed schedules that specify the relative em(cid:173) phasis of each hint, and adaptive schedules that are based on how  well  each  hint  has  been  learned  so  far.  Our  learning  method  is  compatible with any descent technique that we  may choose to use.
Holographic Recurrent Networks
Authors: Tony A. Plate
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Holographic Recurrent Networks (HRNs) are recurrent networks  which incorporate associative memory techniques for storing se(cid:173) quential structure. HRNs can be easily and quickly trained using  gradient descent techniques to generate sequences of discrete out(cid:173) puts and trajectories through continuous spaee. The performance  of HRNs is found to be superior to that of ordinary recurrent net(cid:173) works on these sequence generation tasks.
Generalization Abilities of Cascade Network Architecture
Authors: H. Ritter E. Littmann
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In [5], a new incremental cascade network architecture has been  presented. This paper discusses the properties of such cascade  networks and investigates their generalization abilities under the  particular constraint of small data sets. The evaluation is done for  cascade networks consisting of local linear maps using the Mackey(cid:173) Glass time series prediction task as a benchmark. Our results in(cid:173) dicate that to bring the potential of large networks to bear on the  problem of extracting information from small data sets without run(cid:173) ning the risk of overjitting, deeply cascaded network architectures  are more favorable than shallow broad architectures that contain  the same number of nodes.
Explanation-Based Neural Network Learning for Robot Control
Authors: Sebastian B. Thrun Tom M. Mitchell
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: How can artificial neural nets generalize better from fewer examples? In order  to generalize successfully, neural network learning methods typically require  large training data sets. We introduce a neural network learning method that  generalizes rationally from many fewer data points, relying instead on prior  knowledge encoded in previously learned neural networks. For example, in robot  control learning tasks reported here, previously learned networks that model the  effects of robot actions are used to guide subsequent learning of robot control  functions. For each observed training example of the target function (e.g. the  robot control policy), the learner explains the observed example in terms of its  prior knowledge, then analyzes this explanation to infer additional information  about the shape, or slope, of the target function. This shape knowledge is used  to bias generalization when learning the target function. Results are presented  applying this approach to a simulated robot task based on reinforcement learning.
Some Solutions to the Missing Feature Problem in Vision
Authors: Volker Tresp Subutai Ahmad
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In visual processing the ability to deal with  missing and noisy informa(cid:173) tion is crucial. Occlusions and unreliable feature detectors often lead to  situations where little or no  direct information about features  is  availa(cid:173) ble.  However  the  available  information  is  usually  sufficient  to  highly  constrain  the  outputs.  We  discuss  Bayesian  techniques  for  extracting  class probabilities given partial data. The optimal solution involves inte(cid:173) grating  over the  missing dimensions  weighted  by  the  local  probability  densities.  We  show  how  to  obtain  closed-form  approximations  to  the  Bayesian  solution  using  Gaussian  basis function  networks.  The frame(cid:173) work  extends  naturally  to  the case of noisy  features.  Simulations on a  complex task (3D  hand gesture recognition)  validate the  theory.  When  both integration and weighting by input densities are used, performance  decreases gracefully with the number of missing or noisy features.  Per(cid:173) formance is substantially degraded if either step is omitted.
A Note on Learning Vector Quantization
Authors: Dana H. Ballard Virginia R. de Sa
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Vector Quantization is useful for data compression.  Competitive Learn(cid:173) ing which minimizes reconstruction error is an appropriate algorithm for  vector quantization of unlabelled data.  Vector quantization of labelled  data for classification has a different objective, to minimize the number  of misclassifications, and a different algorithm is appropriate.  We show  that a  variant of Kohonen's LVQ2.1  algorithm can  be  seen  as  a  multi(cid:173) class  extension of an  algorithm which in a  restricted  2  class  case  can  be proven to converge to the Bayes optimal classification boundary.  We  compare the performance of the LVQ2.1 algorithm to that of a modified  version having a decreasing window and normalized step size, on a ten  class vowel classification problem.
Diffusion Approximations for the Constant Learning Rate Backpropagation Algorithm and Resistence to Local Minima
Authors: William Finnoff
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In this paper we discuss the asymptotic properties of the most com(cid:173) monly used variant of the backpropagation algorithm in which net(cid:173) work weights are trained by means of a local gradient descent on ex(cid:173) amples drawn randomly from a fixed training set, and the learning  rate TJ of the gradient updates is held constant (simple backpropa(cid:173) gation). Using stochastic approximation results, we show that for  TJ ~ 0 this training process approaches a batch training and pro(cid:173) vide results on the rate of convergence. Further, we show that for  small TJ one can approximate simple back propagation by the sum  of a batch training process and a Gaussian diffusion which is the  unique solution to a linear stochastic differential equation. Using  this approximation we indicate the reasons why simple backprop(cid:173) agation is less likely to get stuck in local minima than the batch  training process and demonstrate this empirically on a number of  examples.