mtybilly/Instruct2DS huggingface dataset:
Temporal Adaptation in a Silicon Auditory Nerve
Authors: John Lazzaro
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Many  auditory  theorists  consider  the  temporal  adaptation  of the  auditory nerve a key aspect of speech coding in the auditory periph(cid:173) ery.  Experiments  with  models  of auditory  localization  and  pitch  perception  also  suggest  temporal  adaptation  is  an  important ele(cid:173) ment of practical auditory processing.  I have  designed,  fabricated,  and  successfully  tested  an  analog  integrated  circuit  that  models  many aspects of auditory nerve response,  including temporal adap(cid:173) tation.
Oscillatory Model of Short Term Memory
Authors: Marius Usher David Horn
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We investigate a model in which excitatory neurons have dynamical thresh(cid:173) olds  which  display  both  fatigue  and  potentiation.  The  fatigue  property  leads to oscillatory  behavior.  It is  responsible  for  the ability of the model  to  perform  segmentation,  i.e.,  decompose  a  mixed  input  into  staggered  oscillations  of the  activities  of the  cell-assemblies  (memories)  affected  by  it.  Potentiation  is  responsible  for  sustaining  these  staggered  oscillations  after  the  input  is  turned  off,  i.e.  the  system  serves  as  a  model for  short  term memory.  It  has  a  limited  STM  capacity, reminiscent  of the  magical  number 7 ± 2.
Estimating Average-Case Learning Curves Using Bayesian, Statistical Physics and VC Dimension Methods
Authors: Michael Kearns, Manfred Opper, Robert Schapire David Haussler
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: In this paper we investigate an average-case model of concept learning, and  give results that place the popular statistical physics and VC dimension  theories of learning curve behavior in a common framework.
Operators and curried functions: Training and analysis of simple recurrent networks
Authors: Anthony Bloesch Janet Wiles
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present a framework for programming tbe bidden unit representations of  simple recurrent networks based on the use of hint units (additional targets at  the output layer). We present two ways of analysing a network trained within  this framework: Input patterns act as operators on the information encoded by  the context units; symmetrically, patterns of activation over tbe context units  act as curried functions of the input sequences. Simulations demonstrate that a  network can learn to represent three different functions simultaneously and  canonical discriminant analysis is used to investigate bow operators and curried  functions are represented in the space of bidden unit activations.
The Efficient Learning of Multiple Task Sequences
Authors: Satinder P. Singh
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: I  present a  modular  network architecture and  a  learning algorithm  based  on incremental dynamic  programming that allows  a  single  learning agent  to  learn  to  solve  multiple  Markovian  decision  tasks  (MDTs)  with  signif(cid:173) icant  transfer  of learning  across  the  tasks.  I  consider  a  class  of  MDTs,  called  composite  tasks,  formed  by temporally  concatenating a  number  of  simpler, elemental MDTs.  The architecture is  trained on a  set of compos(cid:173) ite and  elemental MDTs.  The  temporal  structure  of a  composite  task  is  assumed  to be  unknown  and the architecture learns  to produce  a  tempo(cid:173) ral  decomposition.  It is  shown  that under certain  conditions  the  solution  of a  composite  MDT can  be  constructed  by computationally inexpensive  modifications of the solutions  of its constituent elemental MDTs.
Benchmarking Feed-Forward Neural Networks: Models and Measures
Authors: Leonard G. C. Hamey
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Existing metrics for the learning performance of feed-forward neural networks do  not provide a satisfactory basis for comparison because the choice of the training  epoch limit can determine the results of the comparison.  I propose new metrics  which  have  the  desirable property of being  independent of the  training epoch  limit.  The efficiency measures  the yield of correct networks in proportion to the  training effort expended.  The optimal epoch limit provides the greatest efficiency.  The learning performance is modelled statistically, and asymptotic performance  is estimated.  Implementation details may be found in (Harney,  1992).
CCD Neural Network Processors for Pattern Recognition
Authors: Michael L. Chuang, Jeffrey R. LaFranchise Alice M. Chiang
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A  CCD-based  processor  that we  call  the  NNC2  is  presented.  The  NNC2  implements a  fully  connected  192-input, 32-output two-layer network  and  can  be  cascaded  to  form  multilayer  networks  or  used  in  parallel  for  ad(cid:173) ditional  input or  output  nodes.  The device  computes  1.92  x  109  connec(cid:173) tions/sec when clocked at 10 MHz.  Network weights can be specified to six  bits of accuracy and are stored on-chip in  programmable digital memories.  A  neural  network  pattern  recognition  system using  NNC2  and  CCD  im(cid:173) age  feature  extractor  (IFE)  devices  is  described.  Additionally,  we  report  a  CCD  output  circuit  that  exploits  inherent  nonlinearities  in  the  charge  injection  process to realize  an  adjustable-threshold sigmoid  in  a  chip  area  of 40  x  80  J.tlU2 .
A Simple Weight Decay Can Improve Generalization
Authors: John A. Hertz Anders Krogh
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: It has  been observed  in  numerical simulations that a weight decay  can  im(cid:173) prove generalization in a feed-forward  neural network.  This paper explains  why.  It is  proven  that  a  weight  decay  has  two effects  in  a  linear  network.  First,  it  suppresses  any  irrelevant  components  of  the  weight  vector  by  choosing  the smallest  vector  that solves  the learning  problem.  Second,  if  the size  is chosen  right,  a weight  decay  can suppress some of the effects  of  static  noise  on  the  targets,  which  improves  generalization  quite  a  lot.  It  is  then  shown  how  to extend  these  results  to networks  with hidden  layers  and  non-linear  units.  Finally  the  theory  is  confirmed  by  some numerical  simulations using  the  data from  NetTalk.
Neural Network Diagnosis of Avascular Necrosis from Magnetic Resonance Images
Authors: Paul Christy, Richard Ehman Armando Manduca
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A vascular necrosis (AVN) of the femoral head is a common yet poten(cid:173) tially serious disorder which can be detected in its very early stages with  magnetic resonance imaging. We have developed multi-layer perceptron  networks, trained with conjugate gradient optimization, which diagnose  A VN from single magnetic resonance images of the femoral head with  100% accuracy on training data and 97% accuracy on test data.
Obstacle Avoidance through Reinforcement Learning
Authors: John E. W. Mayhew Tony J. Prescott
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A  method  is  described  for  generating  plan-like.  reflexive.  obstacle  avoidance behaviour in a mobile robot. The experiments reported here  use  a  simulated  vehicle  with  a  primitive  range  sensor.  Avoidance  behaviour is encoded as a set of continuous functions of the  perceptual  input space. These functions are stored  using CMACs and trained by a  variant of Barto and Sutton's adaptive critic  algorithm.  As  the  vehicle  explores  its  surroundings it adapts  its  responses  to  sensory stimuli  so  as  to  minimise  the  negative  reinforcement  arising  from  collisions.  Strategies  for  local  navigation  are  therefore  acquired  in  an  explicitly  goal-driven  fashion.  The resulting  trajectories  form  elegant collision(cid:173) free paths through the environment
The VC-Dimension versus the Statistical Capacity of Multilayer Networks
Authors: Demetri Psaltis Chuanyi Ji
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A  general  relationship  is  developed  between  the  VC-dimension  and  the  statistical lower  epsilon-capacity which shows  that the VC-dimension can  be  lower  bounded  (in  order)  by the statistical  lower epsilon-capacity of a  network  trained  with  random  samples.  This  relationship  explains  quan(cid:173) titatively  how  generalization  takes  place  after  memorization,  and  relates  the concept of generalization (consistency) with the capacity of the optimal  classifier over a class of classifiers with the same structure and the capacity  of the Bayesian classifier.  Furthermore, it provides a general methodology  to evaluate a  lower  bound for  the VC-dimension of feedforward multilayer  neural networks.  This  general  methodology  is  applied  to  two  types  of networks  which  are  important  for  hardware  implementations:  two  layer  (N  - 2L  - 1)  net(cid:173) works  with  binary  weights,  integer  thresholds  for  the  hidden  units  and  zero  threshold  for  the  output  unit,  and  a  single  neuron  ((N - 1)  net(cid:173) works)  with  binary  weigths  and  a  zero  threshold.  Specifically,  we  obtain  OC~L) ::;  d2  ::;  O(W),  and  d1  ""'  O(N).  Here  W  is  the  total  number  of weights  of  the  (N - 2L - 1)  networks.  d1  and  d2  represent  the  VC(cid:173) dimensions  for  the (N - 1)  and  (N - 2L - 1)  networks respectively.
Improving the Performance of Radial Basis Function Networks by Learning Center Locations
Authors: Thomas Dietterich Dietrich Wettschereck
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Three methods for improving the performance of (gaussian) radial basis  function (RBF) networks were tested on the NETtaik task. In RBF, a  new example is classified by computing its Euclidean distance to a set of  centers chosen by unsupervised methods. The application of supervised  learning to learn a non-Euclidean distance metric was found to reduce the  error rate of RBF networks, while supervised learning of each center's vari(cid:173) ance resulted in inferior performance. The best improvement in accuracy  was achieved by networks called generalized radial basis function (GRBF)  networks. In GRBF, the center locations are determined by supervised  learning. After training on 1000 words, RBF classifies 56.5% of letters  correct, while GRBF scores 73.4% letters correct (on a separate test set).  From these and other experiments, we conclude that supervised learning  of center locations can be very important for radial basis function learning.
Models Wanted: Must Fit Dimensions of Sleep and Dreaming
Authors: Adam N. Mamelak, Jeffrey P. Sutton J. Allan Hobson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: During waking and sleep, the brain and mind undergo a  tightly linked and  precisely  specified  set  of changes  in state.  At  the  level  of neurons,  this  process  has  been  modeled  by  variations  of Volterra-Lotka  equations  for  cyclic fluctuations of brainstem cell populations.  However, neural network  models based upon rapidly developing knowledge ofthe specific population  connectivities  and  their  differential responses  to drugs  have  not  yet  been  developed.  Furthermore, only  the  most  preliminary  attempts  have  been  made  to model across states.  Some  of our own attempts  to link rapid eye  movement (REM) sleep neurophysiology and dream cognition using neural  network approaches  are summarized in this  paper.
Merging Constrained Optimisation with Deterministic Annealing to "Solve" Combinatorially Hard Problems
Authors: Paul Stolorz
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Several parallel analogue algorithms, based upon mean field theory (MFT)  approximations to an underlying statistical mechanics formulation, and re(cid:173) quiring an externally prescribed annealing schedule, now exist for finding  approximate solutions to difficult combinatorial optimisation problems.  They have been applied to the Travelling Salesman Problem (TSP), as  well as to various issues in computational vision and cluster analysis. I  show here that any given MFT algorithm can be combined in a natural  way with notions from the areas of constrained optimisation and adaptive  simulated annealing to yield a single homogenous and efficient parallel re(cid:173) laxation technique, for which an externally prescribed annealing schedule  is no longer required. The results of numerical simulations on 50-city and  100-city TSP problems are presented, which show that the ensuing algo(cid:173) rithms are typically an order of magnitude faster than the MFT algorithms  alone, and which also show, on occasion, superior solutions as well.
Neural Control for Rolling Mills: Incorporating Domain Theories to Overcome Data Deficiency
Authors: Reimar Hofmann, Volker Tresp Martin Röscheisen
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: In  a  Bayesian  framework,  we  give  a  principled  account  of how  domain(cid:173) specific prior knowledge such  as imperfect analytic domain theories can be  optimally  incorporated  into  networks  of locally-tuned  units:  by  choosing  a  specific  architecture  and  by  applying  a  specific  training  regimen.  Our  method  proved  successful  in  overcoming  the  data  deficiency  problem  in  a  large-scale  application  to  devise  a  neural  control  for  a  hot  line  rolling  mill.  It  achieves  in  this  application  significantly  higher  accuracy  than  optimally-tuned standard  algorithms such  as  sigmoidal backpropagation,  and  outperforms the state-of-the-art solution.
Polynomial Uniform Convergence of Relative Frequencies to Probabilities
Authors: Paola Campadelli, Anna Morpurgo, Sandra Panizza Alberto Bertoni
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  define  the  concept  of  polynomial  uniform  convergence  of  relative  frequencies  to  probabilities  in  the  distribution-dependent  context.  Let  Xn  = {O, l}n, let Pn be a  probability distribution on Xn  and let Fn  C  2X ,.  be  a  family  of events.  The  family  {(Xn, Pn, Fn)}n~l  has  the  property  of polynomial uniform  convergence  if the  probability  that  the  maximum  difference  (over  Fn)  between  the  relative  frequency  and  the  probabil(cid:173) ity  of  an  event  exceed  a  given  positive  e  be  at  most  6  (0  <  6  <  1),  when  the sample on  which  the frequency  is  evaluated  has size  polynomial  in  n,l/e,l/b.  Given  at-sample  (Xl, ... ,Xt),  let  C~t)(XI, ... ,Xt)  be  the  Vapnik-Chervonenkis dimension of the family {{x}, ... ,xtl n f  I f  E  Fn}  and  M(n, t)  the  expectation  E(C~t) It).  We  show  that  {(Xn, Pn, Fn)}n~l  has  the  property of polynomial uniform convergence  iff there exists  f3  > 0  such  that  M(n, t)  = O(n/t!3).  Applications  to  distribution-dependent  PAC learning are discussed.
Forward Dynamics Modeling of Speech Motor Control Using Physiological Data
Authors: Eric Vatikiotis-Bateson, Mitsuo Kawato, Michael I. Jordan Makoto Hirayama
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  propose  a  paradigm  for  modeling  speech  production  based  on  neural  networks.  We  focus  on characteristics  of the  musculoskeletal  system.  Using  real physiological data - articulator movements and EMG from  muscle activity(cid:173) a  neural  network  learns  the  forward  dynamics  relating  motor  commands  to  muscles  and  the  ensuing  articulator  behavior.  After  learning,  simulated  perturbations, were used to asses properties of the acquired model, such as natural  frequency,  damping,  and  interarticulator  couplings.  Finally,  a  cascade neural  network is  used  to  generate  continuous  motor commands from  a  sequence  of  discrete articulatory targets.
Learning to Segment Images Using Dynamic Feature Binding
Authors: Richard S. Zemel, Marlene Behrmann Michael Mozer
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Despite  the fact  that complex visual  scenes  contain multiple,  overlapping  objects,  people  perform  object  recognition  with  ease  and accuracy.  One  operation  that facilitates  recognition  is  an early  segmentation  process  in  which  features of objects are grouped and labeled  according  to which  ob(cid:173) ject they belong.  Current computational systems  that perform this oper(cid:173) ation  are based  on predefined  grouping heuristics.  We  describe  a  system  called  MAGIC  that  learn.  how  to  group  features  based  on  a  set  of pre(cid:173) segmented  examples.  In many cases,  MAGIC  discovers  grouping heuristics  similar to those previously proposed, but it also has the capability of find(cid:173) ing  nonintuitive  structural regularities  in images.  Grouping is  performed  by  a  relaxation  network  that  aUempts  to  dynamically  bind  related  fea(cid:173) tures.  Features  transmit  a  complex-valued  signal  (amplitude  and  phase)  to one another;  binding  can thus be represented  by phase locking  related  features.  MAGIC'S  training  procedure is  a  generalization  of recurrent  back  propagation to complex-valued units. 
When a  visual image contains multiple,  overlapping objects,  recognition is  difficult  because features in the image are not grouped according to which object they belong.  Without the capability to form such groupings, it would be necessary to undergo a  massive search through all subsets of image features.  For this reason,  most machine  vision  recognition  systems include  a  component  that  performs feature  grouping or  image  .egmentation (e.g.,  Guzman,  1968; Lowe,  1985;  Marr,  1982).
Simulation of Optimal Movements Using the Minimum-Muscle-Tension-Change Model
Authors: Yoji Uno, Mitsuo Kawato, Ryoji Suzuki Menashe Dornay
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This  work  discusses  various  optimization  techniques  which  were  proposed  in  models  for  controlling  arm  movements.  In  particular,  the  minimum-muscle-tension-change  model  is  investigated.  A  dynamic  simulator of the  monkey's  arm,  including  seventeen  single  and double  joint muscles,  is  utilized  to  generate  horizontal  hand  movements.  The  hand  trajectories  produced by  this  algorithm  are discussed.
Neural Network Analysis of Event Related Potentials and Electroencephalogram Predicts Vigilance
Authors: William W. Lytton, Terrence J. Sejnowski Rita Venturini
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Automated  monitoring  of  vigilance  in  attention  intensive  tasks  such  as  air  traffic  control  or  sonar  operation  is  highly  desirable.  As  the  opera(cid:173) tor monitors the  instrument,  the  instrument  would  monitor the operator,  insuring against lapses.  We  have taken a first  step  toward this goal by  us(cid:173) ing feedforward neural networks trained with backpropagation to interpret  event  related  potentials  (ERPs)  and electroencephalogram  (EEG)  associ(cid:173) ated with periods of high and low vigilance.  The accuracy of our system on  an ERP data set  averaged  over  28  minutes was  96%,  better than the 83%  accuracy  obtained  using  linear  discriminant  analysis.  Practical  vigilance  monitoring will require prediction over shorter time periods.  We were  able  to  average  the  ERP  over  as  little  as  2  minutes  and  still  get  90%  correct  prediction of a vigilance measure.  Additionally, we  achieved similarly good  performance  using segments of EEG  power  spectrum  as  short  as  56  sec.
HARMONET: A Neural Net for Harmonizing Chorales in the Style of J. S. Bach
Authors: Johannes Feulner, Wolfram Menzel Hermann Hild
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: HARMONET, a system employing connectionist networks for music pro(cid:173) cessing, is presented. After being trained on some dozen Bach chorales  using error backpropagation, the system is capable of producing four-part  chorales in the style of J .s.Bach, given a one-part melody. Our system  solves a musical real-world problem on a performance level appropriate  for musical practice. HARMONET's power is based on (a) a new coding  scheme capturing musically relevant information and (b) the integration of  backpropagation and symbolic algorithms in a hierarchical system, com(cid:173) bining the advantages of both.
A Self-Organizing Integrated Segmentation and Recognition Neural Net
Authors: David E. Rumelhart Jim Keeler
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present a neural network algorithm that simultaneously performs seg(cid:173) mentation and recognition of input patterns that self-organizes to detect  input pattern locations and pattern boundaries. We demonstrate this neu(cid:173) ral network architecture on character recognition using the NIST database  and report on results herein. The resulting system simultaneously seg(cid:173) ments and recognizes touching or overlapping characters, broken charac(cid:173) ters, and noisy images with high accuracy.
A Comparison of Projection Pursuit and Neural Network Regression Modeling
Authors: Hang Li, Martin Maechler, R. Douglas Martin, Jim Schimert Jenq-Neng Huang
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Two  projection  based  feedforward  network  learning  methods  for  model(cid:173) free  regression  problems  are  studied  and  compared  in  this  paper:  one  is  the  popular  back-propagation learning  (BPL);  the  other  is  the  projection  pursuit learning  (PPL).  Unlike  the  totally  parametric  BPL  method,  the  PPL  non-parametrically  estimates  unknown  nonlinear  functions  sequen(cid:173) tially (neuron-by-neuron and layer-by-Iayer) at each iteration while jointly  estimating  the  interconnection  weights.  In  terms  of  learning  efficiency,  both  methods  have  comparable  training  speed  when  based  on  a  Gauss(cid:173) Newton  optimization  algorithm  while  the  PPL  is  more  parsimonious.  In  terms of learning robustness  toward noise  outliers,  the  BPL is  more sensi(cid:173) tive to  the  outliers.
Unsupervised Classifiers, Mutual Information and 'Phantom Targets
Authors: Anthony J. R. Heading, David J. C. MacKay John S. Bridle
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: David J.e. MacKay 
California Institute of Technology  139-74 
Pasadena CA 91125  U.S.A 
We derive criteria for training adaptive classifier networks to perform unsu(cid:173) pervised data analysis.  The first  criterion turns a simple Gaussian classifier  into  a  simple  Gaussian  mixture  analyser.  The  second  criterion,  which  is  much more generally applicable, is based on mutual information. It simpli(cid:173) fies  to an  intuitively  reasonable  difference  between  two entropy functions,  one  encouraging  'decisiveness,'  the  other  'fairness'  to  the  alternat.ive  in(cid:173) terpretations  of the  input.  This  'firm  but  fair'  criterion  can  be  applied  to  any  network  that  produces  probability-type  outputs,  but  it  does  not  necessarily  lead to useful  behavior. 
1  Unsupervised Classification 
One of the  main distinctions made in  discussing  neural  network  architectures,  and  pattern analysis algorithms generally, is  between  supervised and unsupervised data  analysis.  We  should  therefore  be  interested  in  any  method  of  building  bridges  between  techniques  in  these  two  categories.  For  instance,  it  is  possible  to  use  an  unsupervised system such as a  Boltzmann machine to learn  the joint distribution of  inputs and a teacher's classificat.ion labels.  The particular type of bridge we seek is a  method of taking a supervised pattern classifier and turning it into an unsupervised  data analyser.  That is,  we  are interested in  methods of "bootstrapping"  classifiers. 
Consider  a  classifier system.  Its input is  a  vector x, and the output is  a  probability  vector y(x).  (That is,  the elements ofy are positive and sum to 1.)  The elements of  y, (Yi (x), i  =  1 ... N c )  are to be taken as the probabilities that x  should be assigned  to  each  of  Nc  classes.  (Note  that  our  definition  of  classifier  does  not  include  a  decision  process.)
Fast, Robust Adaptive Control by Learning only Forward Models
Authors: Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A  large  class  of motor control  tasks  requires  that on  each cycle  the con(cid:173) troller  is  told  its  current  state  and  must  choose  an  action  to  achieve  a  specified,  state-dependent,  goal  behaviour.  This  paper  argues  that  the  optimization  of  learning  rate,  the  number  of experimental  control  deci(cid:173) sions before adequate performance is obtained, and  robustness is of prime  importance-if necessary  at  the  expense  of computation  per  control  cy(cid:173) cle  and  memory  requirement.  This  is  motivated  by  the observation  that  a  robot  which  requires  two  thousand  learning  steps  to  achieve  adequate  performance, or a  robot  which occasionally gets stuck while learning,  will  always  be  undesirable,  whereas  moderate  computational  expense  can  be  accommodated by increasingly powerful computer hardware.  It is  not un(cid:173) reasonable  to assume  the  existence  of inexpensive  100  Mflop  controllers  within  a  few  years  and  so  even  processes  with  control  cycles  in  the  low  tens of milliseconds  will  have millions of machine instructions in  which to  make their decisions.  This paper outlines a learning control scheme which  aims to make effective use of such computational power. 
1  MEMORY BASED  LEARNING  Memory-based  learning  is  an  approach applicable  to both  classification  and  func(cid:173) tion  learning  in  which  all  experiences  presented  to  the  learning  box  are  explic(cid:173) itly  remembered.  The  memory,  Mem,  is  a  set  of input-output  pairs,  Mem  =  {(Xl, YI), (X21 Y2),  ... , (Xb Yk)}.  When  a  prediction  is  required  of the  output  of a  novel input Xquery,  the memory is searched to obtain experiences with inputs close to  Xquery.  These local neighbours are used  to determine a locally consistent output for  the query.  Three memory-based techniques, Nearest Neighbour,  Kernel Regression,  and  Local Weighted Regression,  are shown in  the accompanying figure.
Induction of Finite-State Automata Using Second-Order Recurrent Networks
Authors: Gary M. Kuhn Raymond L. Watrous
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Second-order  recurrent  networks  that  recognize  simple  finite  state  lan(cid:173) guages over  {0,1}* are  induced  from  positive and negative examples.  Us(cid:173) ing the  complete gradient of the  recurrent  network  and sufficient  training  examples to  constrain  the  definition  of the  language to  be induced,  solu(cid:173) tions  are  obtained  that  correctly  recognize  strings of arbitrary  length.  A  method for  extracting  a  finite  state  automaton corresponding  to  an  opti(cid:173) mized network is demonstrated.
Network generalization for production: Learning and producing styled letterforms
Authors: David G. Stork, Ron Keesing, Steve Mims Igor Grebert
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We designed and trained a connectionist network to generate  letterfonns in a new font given just a few exemplars from  that font. During learning. our network constructed a  distributed internal representation of fonts as well as letters.  despite the fact that each training instance exemplified both a  font and a letter. It was necessary to have separate but  interconnected hidden units for " letter" and "font"  representations - several alternative architectures were not  successful.
Green's Function Method for Fast On-Line Learning Algorithm of Recurrent Neural Networks
Authors: Hsing-Hen Chen, Yee-Chun Lee Guo-Zheng Sun
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The two well known learning algorithms of recurrent neural networks are  the back-propagation (Rumelhart & el al.,  Werbos) and the  forward propa(cid:173) gation (Williams and Zipser). The main drawback of back-propagation is  its  off-line backward path in time for error cumulation. This violates the on-line  requirement in many  practical applications.  Although the  forward propaga(cid:173) tion algorithm can be used in an on-line manner, the annoying drawback is  the heavy computation load required to update the high dimensional sensitiv(cid:173) ity matrix (0( fir) operations for each time step). Therefore, to develop a fast  forward algorithm is a challenging task. In this paper w~ proposed a forward  learning algorithm which is one order faster (only 0(fV3) operations for each  time step) than the sensitivity matrix algorithm. The basic idea is that instead  of integrating  the  high  dimensional  sensitivity dynamic  equation  we  solve  forward in time for its Green's function to avoid the redundant computations,  and then update the weights whenever the error is to be corrected. 
A Numerical  example  for  classifying  state  trajectories  using  a recurrent 
network is  presented. It substantiated the faster speed of the proposed algo(cid:173) rithm than the Williams and Zipser's algorithm.
Constrained Optimization Applied to the Parameter Setting Problem for Analog Circuits
Authors: Kurt Fleischer, Lloyd Watts, Alan Barr David Kirk
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We use constrained optimization to select operating parameters for two  circuits: a simple 3-transistor square root circuit, and an analog VLSI  artificial cochlea. This automated method uses computer controlled mea(cid:173) surement and test equipment to choose chip parameters which minimize  the difference between the actual circuit's behavior and a specified goal  behavior. Choosing the proper circuit parameters is important to com(cid:173) pensate for manufacturing deviations or adjust circuit performance within  a certain range. As biologically-motivated analog VLSI circuits become  increasingly complex, implying more parameters, setting these parameters  by hand will become more cumbersome. Thus an automated parameter  setting method can be of great value [Fleischer 90]. Automated parameter  setting is an integral part of a goal-based engineering design methodology  in which circuits are constructed with parameters enabling a wide range  of behaviors, and are then "tuned" to the desired behaviors automatically.
Network Model of State-Dependent Sequencing
Authors: Adam N. Mamelak, J. Allan Hobson Jeffrey P. Sutton
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A network model with temporal sequencing and state-dependent modula(cid:173) tory features is described. The model is motivated by neurocognitive data  characterizing different states of waking and sleeping. Computer studies  demonstrate how unique states of sequencing can exist within the same  network under different aminergic and cholinergic modulatory influences.  Relationships between state-dependent modulation, memory, sequencing  and learning are discussed.
Splines, Rational Functions and Neural Networks
Authors: Peter L. Bartlett Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Connections  between  spline  approximation,  approximation with  rational  functions,  and  feedforward  neural  networks  are  studied.  The  potential  improvement in  the  degree  of approximation in  going  from  single  to two  hidden layer networks is examined.  Some results of Birman and Solomjak  regarding the degree  of approximation achievable  when  knot positions are  chosen  on the basis of the probability distribution of examples rather than  the function  values  are extended.
Learning to Make Coherent Predictions in Domains with Discontinuities
Authors: Geoffrey E. Hinton Suzanna Becker
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  have  previously  described  an  unsupervised  learning  procedure  that  discovers  spatially coherent  propertit>_<;  of the  world  by  maximizing the  in(cid:173) formation  that  parameters  extracted  from  different  parts  of  the  sensory  input convey  about some  common  underlying cause.  When given  random  dot  stereograms  of curved  surfaces,  this  procedure  learns  to  extract  sur(cid:173) face  depth  because  that  is  the  property  that  is  coherent  across  space.  It  also  learns  how  to  interpolate  the  depth  at  one  location  from  the  depths  at  nearby  locations  (Becker  and  Hint.oll.  1992).  1n  this  paper,  we  pro(cid:173) pose  two  new  models  which  handle surfaces  with discontinuities.  The first  model  attempts  to  detect  cases  of discontinuities  and  reject  them.  The  second  model  develops  a  mixture of expert  interpolators.  It  learns  to  de(cid:173) tect  the  locations of discontinuities  and  to  invoke specialized,  asymmetric  interpolators  that  do  not cross  the  discontinuities.
Bayesian Model Comparison and Backprop Nets
Authors: David J. C. MacKay
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The Bayesian model comparison framework  is  reviewed, and the Bayesian  Occam's razor is explained.  This framework can be applied to feedforward  networks,  making  possible  (1)  objective  comparisons  between  solutions  using alternative network architectures; (2)  objective choice of magnitude  and  type of weight decay terms;  (3)  quantified estimates of the error  bars  on network  parameters and on network output.  The framework  also  gen(cid:173) erates a  measure of the effective number of parameters determined by the  data.  The  relationship  of  Bayesian  model  comparison  to  recent  work  on  pre(cid:173) diction  of generalisation  ability  (Guyon  et  al.,  1992,  Moody,  1992)  is  dis(cid:173) cussed. 
1  BAYESIAN INFERENCE AND  OCCAM'S  RAZOR 
In science, a  central task is  to develop and compare models to account for  the data  that  are  gathered.  Typically,  two  levels  of inference  are  involved  in  the  task  of  data modelling.  At  the  first  level  of inference,  we  assume  that  one  of the  models  that  we  invented  is  true,  and  we  fit  that  model  to  the  data.  Typically  a  model  includes some free  parameters; fitting  the model to the data involves inferring what  values those parameters should  probably take, given the data.  This  is  repeated  for  each  model.  The second  level  of inference  is  the  task of model  comparison.  Here, 
·Current address:  Darwin  College,  Cambridge CB3  9EU,  U.K.
Networks for the Separation of Sources that are Superimposed and Delayed
Authors: Federico Faggin John C. Platt
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We have created new networks to unmix signals which have been  mixed either with time delays or via filtering. We first show that  a subset of the Herault-Jutten learning rules fulfills a principle of  minimum output power. We then apply this principle to extensions  of the Herault-Jutten network which have delays in the feedback  path. Our networks perform well on real speech and music signals  that have been mixed using time delays or filtering.
Burst Synchronization without Frequency Locking in a Completely Solvable Neural Network Model
Authors: Christof Koch Heinz Schuster
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The dynamic behavior of a network model consisting of all-to-all excitatory  coupled binary neurons with global inhibition is studied analytically and  numerically. We prove that for random input signals, the output of the  network consists of synchronized bursts with apparently random intermis(cid:173) sions of noisy activity. Our results suggest that synchronous bursts can be  generated by a simple neuronal architecture which amplifies incoming coin(cid:173) cident signals. This synchronization process is accompanied by dampened  oscillations which, by themselves, however, do not play any constructive  role in this and can therefore be considered to be an epiphenomenon.
Incrementally Learning Time-varying Half-planes
Authors: Thomas Petsche, Ronald L. Rivest Anthony Kuh
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present a distribution-free model for incremental learning when concepts vary  with time. Concepts are caused to change by an adversary while an incremental  learning algorithm attempts to track the changing concepts by minimizing the  error between the current target concept and the hypothesis. For a single half(cid:173) plane and the intersection of two half-planes, we show that the average mistake  rate depends on the maximum rate at which an adversary can modify the concept.  These theoretical predictions are verified with simulations of several learning  algorithms including back propagation.
Oscillatory Neural Fields for Globally Optimal Path Planning
Authors: Michael Lemmon
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A neural network solution is proposed for solving path planning problems  faced by mobile robots. The proposed network is a two-dimensional sheet  of neurons forming a distributed representation of the robot's workspace.  Lateral interconnections between neurons are "cooperative", so that the  network exhibits oscillatory behaviour. These oscillations are used to gen(cid:173) erate solutions of Bellman's dynamic programming equation in the context  of path planning. Simulation experiments imply that these networks locate  global optimal paths even in the presence of substantial levels of circuit  nOlse. 
1 Dynamic Programming and Path Planning 
Consider a 2-DOF robot moving about in a 2-dimensional world. A robot's location  is denoted by the real vector, p. The collection of all locations forms a set called the  workspace. An admissible point in the workspace is any location which the robot  may occupy. The set of all admissible points is called the free workspace. The  free workspace's complement represents a collection of obstacles. The robot moves  through the workspace along a path which is denoted by the parameterized curve,  p(t). An admissible path is one which lies wholly in the robot's free workspace.  Assume that there is an initial robot position, Po, and a desired final position, p J.  The robot path planning problem is to find an admissible path with Po and p J as  endpoints such that some "optimality" criterion is satisfied. 
The path planning problem may be stated more precisely from an optimal control  539
Retinogeniculate Development: The Role of Competition and Correlated Retinal Activity
Authors: David G. Stork, Carla J. Shatz Ron Keesing
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: During visual development, projections from retinal ganglion cells  (RGCs) to the lateral geniculate nucleus (LGN) in cat are refined to  produce ocular dominance layering and precise topographic mapping.  Normal development depends upon activity in RGCs, suggesting a key  role for activity-dependent synaptic plasticity. Recent experiments on  prenatal retina show that during early development, "waves" of activity  pass across RGCs (Meister, et aI., 1991). We provide the first  simulations to demonstrate that such retinal waves, in conjunction with  Hebbian synaptic competition and early arrival of contralateral axons,  can account for observed patterns of retinogeniculate projections in  normal and experimentally-treated animals.
A Neural Net Model for Adaptive Control of Saccadic Accuracy by Primate Cerebellum and Brainstem
Authors: John E. W. Mayhew, Pat Langdon Paul Dean
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Accurate saccades require interaction between brainstem circuitry and the  cerebeJJum.  A model of this interaction is described, based on Kawato's  principle  of  feedback-error-Iearning.  In  the  model  a  part  of  the  brainstem  (the superior colliculus) acts as a simple feedback controJJer  with no knowledge of initial eye position, and provides an error signal  for the cerebeJJum to correct for eye-muscle nonIinearities.  This teaches  the cerebeJJum,  modelled as a CMAC, to adjust appropriately the gain  on the brainstem burst-generator's internal feedback loop and so alter the  size of burst sent to  the  motoneurons.  With  direction-only errors  the  system rapidly learns to make accurate horizontal eye movements from  any  starting position, and  adapts realistically to  subsequent simulated  eye-muscle weakening or displacement of the saccadic target.
Segmentation Circuits Using Constrained Optimization
Authors: John G. Harris
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A novel segmentation  algorithm has  been  developed  utilizing an absolute(cid:173) value  smoothness  penalty  instead  of  the  more  common  quadratic  regu(cid:173) larizer.  This  functional  imposes  a  piece-wise  constant  constraint  on  the  segmented  data.  Since  the  minimized energy  is  guaranteed  to  be  convex,  there  are  no  problems  with  local  minima  and  no  complex  continuation  methods are  necessary  to find  the  unique  global  minimum.  By  interpret(cid:173) ing the minimized energy  as  the generalized  power  of a  nonlinear resistive  network,  a  continuous-time analog segmentation  circuit was  constructed.
Rule Induction through Integrated Symbolic and Subsymbolic Processing
Authors: Michael Mozer, Paul Smolensky Clayton McMillan
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  describe a neural network, called RufeNet, that learns explicit, sym(cid:173) bolic  condition-action  rules  in  a  formal  string  manipulation  domain.  RuleNet  discovers  functional  categories  over  elements  of the  domain,  and,  at  various  points  during  learning,  extracts  rules  that  operate  on  these  categories.  The  rules  are  then  injected  back  into  RuleNet  and  training continues,  in  a  process called iterative projection.  By  incorpo(cid:173) rating rules in this way,  RuleNet exhibits enhanced learning and gener(cid:173) alization  performance  over  alternative  neural  net  approaches.  By  integrating symbolic  rule  learning  and  subsymbolic  category  learning,  RuleNet  has capabilities  that go beyond a  purely  symbolic system.  We  show  how  this  architecture  can be  applied to  the  problem  of case-role  assignment  in  natural  language  processing,  yielding a  novel  rule-based  solution.
A comparison between a neural network model for the formation of brain maps and experimental data
Authors: K. Schulten, G. G. Blasdel K. Obermayer
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Recently, high resolution images of the simultaneous representation of  orientation preference, orientation selectivity and ocular dominance have  been obtained for large areas in monkey striate cortex by optical imaging  [1-3]. These data allow for the first time a "local" as well as "global"  description of the spatial patterns and provide strong evidence for corre(cid:173) lations between orientation selectivity and ocular dominance.  A quantitative analysis reveals that these correlations arise when a five(cid:173) dimensional feature space (two dimensions for retinotopic space, one each  for orientation preference, orientation specificity, and ocular dominance) is  mapped into the two available dimensions of cortex while locally preserving  topology. These results provide strong evidence for the concept of topology  preserving maps which have been suggested as a basic design principle of  striate cortex [4-7]. 
Monkey striate cortex contains a retinotopic map in which are embedded the highly  repetitive patterns of orientation selectivity and ocular dominance. The retinotopic  projection establishes a "global" order, while maps of variables describing other  stimulus features, in particular line orientation and ocularity, dominate cortical  organization locally. A large number of pattern models [8-12] as well as models  of development [6,7,13-21] have been proposed to describe the spatial structure of  these patterns and their development during ontogenesis. However, most models  have not been compared with experimental data in detail. There are two reasons  for this: (i) many model-studies were not elaborated enough to be experimentally  testable and (ii) a sufficient amount of experimental data obtained from large areas  of striate cortex was not available.  83 
84 
Obermayer, Schulten, and Blasdel 
Figure 1: Spatial pattern of orientation preference and ocular dominance in mon(cid:173) key striate cortex (left) compared with predictions of the SOFM-model (right). Iso(cid:173) orientation lines (gray) are drawn in intervals of 11.25° (left) and IS.00 (right), re(cid:173) spectively. Black lines indicate the borders (ws(rj = 0) of ocular dominance bands.  The areas enclosed by black rectangles mark corresponding elements of organization  in monkey striate cortex and in the simulation result (see text). Left: Data obtained  from a 3.1mm x 4.2mm patch of the striate cortex of an adult macaque (macaca  nemestrina) by optical imaging [1-3]. The region is located near the border with  area IS, close to midline. Right: Model-map generated by the SOFM-algorithm.  The figure displays a small section of a network of size N = d = 512. The param(cid:173) eters of the simulation were: € = 0.02, tTh = 5, vr,:x = 20.48, vrax = 15.36, 9 . 101  iterations, with retinotopic initial conditions and periodic boundary conditions. 
1 Orientation and ocular dominance columns in monkey