mtybilly/Instruct2DS huggingface dataset:
A Reconfigurable Analog VLSI Neural Network Chip
Authors: Srinagesh Satyanarayana, Yannis P. Tsividis, Hans Peter Graf
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: 1024 distributed-neuron synapses have been integrated in an active  area of 6.1mm  x  3.3mm using  a 0.9p.m,  double-metal,  single-poly,  n-well CMOS  technology.  The distributed-neuron synapses are ar(cid:173) ranged in blocks of 16, which we call '4  x  4 tiles'.  Switch matrices  are  interleaved between each of these  tiles  to provide programma(cid:173) bility of interconnections.  With a small area overhead (15  %),  the  1024  units  of the  network can  be rearranged  in various  configura(cid:173) tions.  Some of the possible configurations are, a  12-32-12  network,  a  16-12-12-16  network, two 12-32  networks etc.  (the numbers sep(cid:173) arated  by dashes indicate the number of units per layer, including  the  input  layer).  Weights  are  stored  in  analog  form  on  MaS  ca(cid:173) pacitors.  The synaptic weights  are  usable to a  resolution  of 1 % of  their full  scale  value.  The limitation arises  due  to charge injection  from  the access  switch and  charge leakage.  Other  parameters like  gain and shape of nonlinearity are also programmable.
Handwritten Digit Recognition with a Back-Propagation Network
Authors: Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, R. E. Howard, Wayne E. Hubbard, Lawrence D. Jackel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We present an application of back-propagation networks to hand(cid:173) written digit recognition. Minimal preprocessing of the data was  required, but architecture of the network was highly constrained  and specifically designed for the task. The input of the network  consists of normalized images of isolated digits. The method has  1 % error rate and about a 9% reject rate on zipcode digits provided  by the U.S. Postal Service.
Computational Efficiency: A Common Organizing Principle for Parallel Computer Maps and Brain Maps?
Authors: Mark E. Nelson, James M. Bower
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: It is  well-known  that  neural  responses  in  particular  brain  regions  are  spatially  organized,  but  no  general  principles  have  been  de(cid:173) veloped  that  relate  the structure of a  brain  map  to  the nature of  the associated computation.  On parallel computers, maps of a sort  quite similar to brain maps arise when a computation is distributed  across  multiple  processors.  In  this  paper  we  will  discuss  the rela(cid:173) tionship  between  maps and computations on these  computers and  suggest how similar considerations might also apply to maps in the  brain.
A Cost Function for Internal Representations
Authors: Anders Krogh, C. I. Thorbergsson, John A. Hertz
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  introduce  a  cost  function  for  learning  in  feed-forward  neural  networks  which  is  an  explicit  function  of the  internal representa(cid:173) tion  in  addition  to  the  weights.  The  learning  problem  can  then  be formulated  as two simple perceptrons and  a  search for  internal  representations.  Back-propagation  is  recovered  as  a  limit.  The  frequency  of successful  solutions  is  better for  this  algorithm than  for  back-propagation when  weights and hidden  units  are  updated  on the same timescale i.e.  once  every learning step.
HMM Speech Recognition with Neural Net Discrimination
Authors: William Y. Huang, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Two approaches were explored which integrate neural net classifiers  with  Hidden  Markov  Model  (HMM)  speech  recognizers.  Both  at(cid:173) tempt to improve speech pattern discrimination while retaining the  temporal processing advantages of HMMs.  One approach used neu(cid:173) ral nets to provide second-stage discrimination following an HMM  recognizer.  On  a  small  vocabulary  task,  Radial  Basis  Function  (RBF)  and  back-propagation  neural  nets  reduced  the  error  rate  substantially (from 7.9% to 4.2% for the RBF classifier).  In a larger  vocabulary task, neural net classifiers did not reduce the error rate.  They, however,  outperformed Gaussian, Gaussian mixture, and k(cid:173) nearest  neighbor  (KNN)  classifiers.  In  another  approach,  neural  nets  functioned  as  low-level  acoustic-phonetic  feature  extractors.  When  classifying  phonemes  based on single  10  msec.  frames,  dis(cid:173) criminant  RBF  neural  net  classifiers outperformed  Gaussian mix(cid:173) ture  classifiers.  Performance,  however,  differed  little when  classi(cid:173) fying  phones  by  accumulating scores  across  all frames  in phonetic  segments using  a single node HMM  recognizer. 
-This  work  was  sponsored  by  the  Department  of  the  Air  Force  and  the  Air  Force  Office  of 
Scientific Research. 
HMM Speech Recognition with Neural Net Discrimination
Generalization and Parameter Estimation in Feedforward Nets: Some Experiments
Authors: N. Morgan, H. Bourlard
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We have done an empirical study of the relation of the number of  parameters (weights) in a feedforward net to generalization perfor(cid:173) mance. Two experiments are reported. In one, we use simulated data  sets with well-controlled parameters, such as the signal-to-noise ratio  of continuous-valued data. In the second, we train the network on  vector-quantized mel cepstra from real speech samples. In each case,  we use back-propagation to train the feedforward net to discriminate in  a multiple class pattern classification problem. We report the results of  these studies, and show the application of cross-validation techniques  to prevent overfitting.
VLSI Implementation of a High-Capacity Neural Network Associative Memory
Authors: Tzi-Dar Chiueh, Rodney M. Goodman
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In  this  paper we  describe  the  VLSI  design  and  testing of a  high  capacity  associative  memory  which  we  call  the  exponential  cor(cid:173) relation  associative  memory  (ECAM).  The  prototype  3J.'-CMOS  programmable  chip  is  capable  of storing  32  memory  patterns  of  24 bits each.  The high capacity of the ECAM is  partly due  to the  use of special exponentiation neurons, which  are implemented via  sub-threshold MOS  transistors in this design.  The prototype chip  is  capable of performing one associative recall in  3  J.'S.
The Cascade-Correlation Learning Architecture
Authors: Scott E. Fahlman, Christian Lebiere
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Cascade-Correlation is a new architecture and supervised learning algo(cid:173) rithm for artificial neural networks.  Instead of just adjusting the weights  in a network of fixed topology. Cascade-Correlation begins with a min(cid:173) imal network,  then automatically trains  and adds new hidden  units  one  by  one,  creating a  multi-layer structure.  Once  a  new  hidden  unit  has  been added  to the network, its  input-side weights are frozen.  This  unit  then becomes a permanent feature-detector in the network, available for  producing  outputs  or for  creating other,  more complex  feature  detec(cid:173) tors.  The Cascade-Correlation architecture has  several advantages over  existing algorithms:  it  learns  very quickly,  the network . determines  its  own size and  topology, it retains  the structures  it  has  built even  if the  training set changes, and it requires no back-propagation of error signals  through  the connections of the network. 
1  DESCRIPTION OF CASCADE·CORRELATION  The  most  important  problem  preventing  the  widespread  application  of artificial  neural  networks  to real-world problems  is  the slowness  of existing learning algorithms  such  as  back-propagation  (or "backprop").  One factor contributing to  that  slowness  is  what we  call  the moving  target problem:  because all of the  weights  in  the network are changing  at once,  each  hidden  units  sees  a  constantly changing environment.  Instead  of moving  quickly to assume useful roles in the overall problem solution, the hidden units engage in  a complex dance with much  wasted motion.  The Cascade-Correlation learning algorithm  was  developed in an  attempt to solve that problem.  In  the problems  we have examined,  it learns  much faster than back-propagation and  solves  some other problems as  well. 
The Cascade-Correlation Learning Architecture 
525
Sigma-Pi Learning: On Radial Basis Functions and Cortical Associative Learning
Authors: Bartlett W. Mel, Christof Koch
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  goal  in  this  work  has  been  to  identify  the  neuronal  elements  of the cortical column that are most likely to support  the learning  of nonlinear associative  maps.  We show that  a  particular style  of  network learning algorithm based on locally-tuned  receptive fields  maps  naturally  onto cortical  hardware,  and  gives  coherence  to  a  variety of features  of cortical anatomy,  physiology,  and  biophysics  whose  relations to learning remain poorly understood.
Optimal Brain Damage
Authors: Yann LeCun, John S. Denker, Sara A. Solla
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  have used  information-theoretic ideas  to derive  a class of prac(cid:173) tical  and  nearly  optimal schemes  for  adapting the size  of a  neural  network.  By  removing  unimportant  weights  from  a  network,  sev(cid:173) eral  improvements  can  be  expected:  better  generalization,  fewer  training examples required,  and improved speed  of learning and/or  classification.  The  basic  idea  is  to  use  second-derivative  informa(cid:173) tion to make a  tradeoff between  network  complexity  and  training  set error.  Experiments confirm  the usefulness  of the methods on a  real-world  application.
Learning Aspect Graph Representations from View Sequences
Authors: Michael Seibert, Allen M. Waxman
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In our effort to develop a modular neural system for invariant learn(cid:173) ing and  recognition of 3D objects,  we  introduce here a new  module  architecture  called  an  aspect  network constructed  around  adaptive  axo-axo-dendritic synapses.  This  builds  upon  our existing system  (Seibert & Waxman, 1989) which  processes  20 shapes and classifies  t.hem  into  view  categories  (i.e.,  aspects)  invariant  to  illumination,  position,  orientat.ion,  scale,  and  projective  deformations.  From  a  sequence 'of  views,  the  aspect  network  learns  the  transitions  be(cid:173) tween  these  aspects,  crystallizing  a  graph-like  structure  from  an  initially  amorphous  network .  Object  recognition  emerges  by  ac(cid:173) cumulating evidence over  multiple views  which  activate competing  object  hypotheses.
The Effect of Catecholamines on Performance: From Unit to System Behavior
Authors: David Servan-Schreiber, Harry Printz, Jonathan D. Cohen
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: At the level of individual neurons. catecholamine release increases  the  responsivity  of cells  to  excitatory and  inhibitory  inputs.  We  present a  model  of catecholamine effects  in  a  network  of neural-like  elements.  We  argue  that  changes  in  the  responsivity  of individual  elements  do  not  affect  their  ability  to  detect  a  signal  and  ignore  noise.  However.  the same changes in cell responsivity in a network of such elements do  improve the signal detection performance of the network as a whole.  We  show how  this result can be used in a computer simulation of behavior  to  account  for  the  effect  of eNS  stimulants  on  the  signal  detection  performance of human subjects.
Synergy of Clustering Multiple Back Propagation Networks
Authors: William P. Lincoln, Josef Skrzypek
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The properties of a cluster of multiple back-propagation (BP) networks  are  examined  and  compared  to  the  performance  of a  single  BP  net(cid:173) work.  The underlying idea is that a synergistic effect within the cluster  improves the perfonnance and fault tolerance.  Five networks were ini(cid:173) tially  trained  to  perfonn  the  same  input-output  mapping.  Following  training, a cluster was created by computing an average of the outputs  generated by the individual networks.  The output of the cluster can be  used as the desired output during training by feeding it back to the indi(cid:173) vidual  networks.  In  comparison  to  a  single  BP  network,  a  cluster  of  multiple  BP's generalization  and  significant fault  tolerance.  It appear  that cluster advantage follows from  simple maxim  "you can fool  some  of the single BP's in a cluster all of the time but you cannot fool all of  them all of the time"  {Lincoln}
Incremental Parsing by Modular Recurrent Connectionist Networks
Authors: Ajay N. Jain, Alex Waibel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present a novel,  modular, recurrent connectionist network architec(cid:173) ture  which  learns  to robustly  perform  incremental  parsing of complex  sentences.  From  sequential  input,  one  word  at  a  time,  our  networks  learn  to  do  semantic  role  assignment,  noun  phrase  attachment,  and  clause structure recognition for sentences with passive constructions and  center embedded  clauses.  The  networks  make  syntactic  and  semantic  predictions  at every point in time, and previous  predictions are revised  as  expectations are affirmed or violated with the arrival of new informa(cid:173) tion.  Our networks  induce  their own "grammar rules" for dynamically  transforming  an  input  sequence of words  into a  syntactic/semantic  in(cid:173) terpretation.  These networks  generalize  and  display  tolerance  to  input  which  has  been  corrupted  in ways common in spoken  language.
Reading a Neural Code
Authors: William Bialek, Fred Rieke, Robert R. de Ruyter van Steveninck, David Warland
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Traditional methods of studying neural coding characterize  the en(cid:173) coding  of known  stimuli  in  average  neural  responses.  Organisms  face  nearly the opposite task - decoding short segments of a spike  train to extract information about an unknown, time-varying stim(cid:173) ulus.  Here  we  present strategies for  characterizing the  neural code  from  the  point of view  of the  organism, culminating in  algorithms  for  real-time  stimulus reconstruction  based  on  a  single  sample  of  the spike train.  These methods are applied to the design and anal(cid:173) ysis  of experiments on an  identified movement-sensitive neuron  in  the fly  visual system.  As far  as we  know this is  the first  instance in  which a direct  "reading"  of the neural code has been  accomplished.
A Method for the Associative Storage of Analog Vectors
Authors: Amir F. Atiya, Yaser S. Abu-Mostafa
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A method for  storing analog vectors in  Hopfield's continuous feed(cid:173) back model is proposed.  By  analog vectors we mean vectors whose  components  are  real-valued.  The  vectors  to  be  stored  are  set  as  equilibria of the network.  The network model consists of one layer  of visible  neurons  and  one  layer  of hidden  neurons.  We  propose  a  learning  algorithm,  which  results  in  adjusting  the  positions  of  the  equilibria,  as  well  as  guaranteeing  their  stability.  Simulation  results confirm the effectiveness of the method .
Designing Application-Specific Neural Networks Using the Genetic Algorithm
Authors: Steven A. Harp, Tariq Samad, Aloke Guha
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present  a  general  and  systematic  method  for  neural  network  design  based  on  the  genetic  algorithm.  The  technique  works  in  conjunction  with  network  learning  rules,  addressing  aspects  of  the  network's  gross  architecture,  connectivity,  and  learning  rule  parameters.  Networks  can  be  optimiled  for  various  application(cid:173) specific  criteria, such  as  learning speed, generalilation,  robustness  and  connectivity.  The  approach  is  model-independent.  We  describe  a  prototype  system,  NeuroGENESYS,  that employs  the  backpropagation  learning  rule.  Experiments  on  several  small  problems  have  been  conducted.  In  each  case,  NeuroGENESYS  has  produced  networks  that perform significantly  better than  the  randomly  generated  networks  of its initial population.  The  com(cid:173) putational feasibility  of our approach  is  discussed.
A Computational Basis for Phonology
Authors: David S. Touretzky, Deirdre W. Wheeler
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The phonological structure of human  languages  is  intricate,  yet highly  constrained.  Through  a  combination  of connectionist  modeling  and  linguistic  analysis,  we are attempting to develop a computational basis  for  the  nature  of phonology.  We  present  a  connectionist  architecture  that  performs  multiple  simultaneous  insertion,  deletion,  and  mutation  operations on sequences  of phonemes, and introduce a novel additional  primitive,  clustering.  Clustering  provides  an  interesting  alternative  to  both iterative and relaxation accounts of assimilation processes  such as  vowel  harmony.  Our resulting  model  is  efficient because it processes  utterances entirely in parallel  using  only  feed-forward circuitry.
Neural Network Simulation of Somatosensory Representational Plasticity
Authors: Kamil A. Grajski, Michael Merzenich
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  brain  represents  the  skin  surface  as  a  topographic  map  in  the  somatosensory  cortex.  This  map  has  been  shown  experimentally  to  be  modifiable  in  a  use-dependent  fashion  throughout  life.  We  present  a  neural  network  simulation  of  the  competitive  dynamics  underlying  this  cortical  plasticity  by  detailed  analysis  of  receptive  field  properties  of  model  neurons  during  simulations  of  skin  co(cid:173) activation, cortical  lesion,  digit amputation and nerve  section.
Generalized Hopfield Networks and Nonlinear Optimization
Authors: Gintaras V. Reklaitis, Athanasios G. Tsirukis, Manoel Fernando Tenorio
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  nonlinear  neural  framework,  called  the  Generalized  Hopfield  network,  is  proposed,  which  is  able  to  solve  in  a  parallel  distributed  manner systems  of nonlinear equations.  The  method is  applied  to  the  general  nonlinear  optimization  problem.  We  demonstrate  GHNs  implementing  the  three  most  important  optimization  algorithms,  namely the Augmented Lagrangian, Generalized Reduced Gradient and  Successive  Quadratic  Programming  methods.  The  study  results  in  a  dynamic  view of the optimization problem and offers a straightforward  model  for  the  parallelization  of  the  optimization  computations,  thus  significantly  extending  the  practical  limits  of problems  that  can  be  formulated  as  an  optimization  problem  and  which  can  gain  from  the  introduction of nonlinearities in their structure (eg. pattern recognition,  supervised learning, design of content-addressable memories). 
1 To whom correspondence should be addressed. 
356 
Reklaitis, Tsirukis and Tenorio 
1  RELATED  WORK  The  ability  of networks  of highly  interconnected  simple  nonlinear  analog  processors  (neurons)  to  solve  complicated  optimization  problems  was  demonstrated  in  a  series  of  papers by Hopfield and Tank (Hopfield,  1984), (Tank, 1986).  The  Hopfield  computational  model  is  almost  exclusively  applied  to  the  solution  of  combinatorially  complex  linear  decision  problems  (eg.  Traveling  Salesman  Problem).  Unfortunately  such problems can  not be  solved with  guaranteed  quality,  (Bruck,  1987),  getting trapped in locally optimal solutions.  Jeffrey  and  Rossner,  (Jeffrey,  1986),  extended  Hopfield's  technique  to  the  nonlinear  unconstrained  optimization  problem,  using  Cauchy  dynamics.  Kennedy  and  Chua,  (Kennedy,  1988), presented an  analog  implementation of a  network  solving a  nonlinear  optimization problem.  The underlying optimization algorithm  is a  simple transformation  method, (Reklaitis,  1983), which is known  to be relatively inefficient for large nonlinear  optimization problems. 
2  LINEAR  HOPFIELD  NETWORK  (LHN)  The computation in a Hopfield network is done by  a collection of highly interconnected  simple  neurons.  Each processing element,  i,  is characterized by  the activation  level,  Ui,  which is a function of the input received from  the external environment, Ii, and  the state  of the other neurons. The activation level of i is transmitted to  the other processors, after  passing through a filter that converts Ui to a 0-1 binary value, Vi'  The time behavior of the system is described by the following model: 
~ T·V·  - -' + I·  '  ~ 'J  J  J
Connectionist Architectures for Multi-Speaker Phoneme Recognition
Authors: John B. Hampshire II, Alex Waibel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present  a  number  of Time-Delay  Neural  Network  (TDNN)  based  architectures  for multi-speaker phoneme recognition (/b,d,g/ task).  We  use  speech  of two females  and four males to  compare the performance  of the various architectures against a baseline recognition rate of 95.9%  for a single IDNN on the six-speaker /b,d,g/ task.  This series of modu(cid:173) lar designs leads to a highly modular multi-network architecture capable  of performing the six-speaker recognition task at the speaker dependent  rate  of 98.4%.  In  addition  to  its  high  recognition  rate,  the  so-called  "Meta-Pi"  architecture  learns - without direct  supervision - ognize the speech of one particular male speaker using internal models  of other male  speakers exclusively.
Bayesian Inference of Regular Grammar and Markov Source Models
Authors: Kurt R. Smith, Michael I. Miller
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In this paper we develop a Bayes criterion which includes the Rissanen  complexity, for  inferring regular grammar models.  We develop two  methods for regular grammar Bayesian inference.  The fIrst method is  based  on  treating  the  regular  grammar as  a  I-dimensional  Markov  source, and the second is based on the combinatoric characteristics of  the regular grammar itself.  We apply the resulting Bayes criteria to a  particular example in order to show the efficiency of each method.
Discovering High Order Features with Mean Field Modules
Authors: Conrad C. Galland, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A new  form of the deterministic Boltzmann machine (DBM) learn(cid:173) ing procedure is presented  which can efficiently train network mod(cid:173) ules  to discriminate  between  input  vectors  according  to some  cri(cid:173) terion.  The new  technique directly utilizes the free  energy  of these  "mean field  modules"  to represent the probability that the criterion  is  met,  the  free  energy  being readily  manipulated by  the  learning  procedure.  Although  conventional deterministic  Boltzmann learn(cid:173) ing  fails  to  extract  the  higher  order  feature  of shift  at  a  network  bottleneck,  combining  the  new  mean  field  modules  with  the  mu(cid:173) tual information objective function  rapidly produces  modules that  perfectly extract this important higher order feature without direct  external supervision.
Sequential Decision Problems and Neural Networks
Authors: A. G. Barto, R. S. Sutton, C. J. C. H. Watkins
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Decision making tasks that involve delayed consequences are very  common yet difficult to address with supervised learning methods.  If there is an accurate model of the underlying dynamical system,  then these tasks can be formulated as sequential decision problems  and solved by Dynamic Programming. This paper discusses rein(cid:173) forcement learning in terms of the sequential decision framework  and shows how a learning algorithm similar to the one implemented  by the Adaptive Critic Element used in the pole-balancer of Barto,  Sutton, and Anderson (1983), and further developed by Sutton  (1984), fits into this framework. Adaptive neural networks can  play significant roles as modules for approximating the functions  required for solving sequential decision problems.
Asymptotic Convergence of Backpropagation: Numerical Experiments
Authors: Subutai Ahmad, Gerald Tesauro, Yu He
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Yu  He  Dept.  of Physics  Ohio  State Univ.  Columbus,  OH 43212 
We  have  calculated, both analytically and in simulations,  the rate  of convergence  at  long  times  in  the  backpropagation learning  al(cid:173) gorithm  for  networks  with  and  without  hidden  units.  Our  basic  finding for units using the standard sigmoid transfer function is  lit  convergence of the  error for  large t,  with  at most logarithmic cor(cid:173) rections  for  networks  with  hidden  units.  Other transfer functions  may lead to a  8lower polynomial rate of convergence.  Our analytic  calculations were presented in (Tesauro, He &  Ahamd, 1989).  Here  we  focus  in more detail on our empirical measurements of the con(cid:173) vergence rate in numerical simulations,  which  confirm our analytic  results.
Unsupervised Learning in Neurodynamics Using the Phase Velocity Field Approach
Authors: Michail Zak, Nikzad Benny Toomarian
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  new  concept for  unsupervised  learning based  upon  examples in(cid:173) troduced  to the neural  network  is  proposed.  Each example is  con(cid:173) sidered  as  an  interpolation  node  of the  velocity field  in  the  phase  space.  The velocities  at  these  nodes  are selected such  that all  the  streamlines converge  to an attracting set imbedded in the subspace  occupied by the cluster of examples.  The synaptic interconnections  are  found  from  learning  procedure  providing  selected  field.  The  theory  is  illustrated  by examples. 
This  paper  is  devoted  to  development  of a  new  concept  for  unsupervised  learning  based upon examples introduced to an artificial neural network.  The neural network  is  considered  as  an  adaptive  nonlinear  dissipative  dynamical  system  described  by  the following  coupled  differential equations: 
Ui + K,Ui  = L 11j g( Uj )  + Ii
A Continuous Speech Recognition System Embedding MLP into HMM
Authors: Hervé Bourlard, Nelson Morgan
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  are  developing  a  phoneme  based.  speaker-dependent  continuous  speech  recognition  system  embedding  a  Multilayer Perceptron  (MLP)  (Le .•  a  feedforward  Artificial  Neural  Network).  into  a Hidden  Markov  Model (HMM) approach.  In [Bourlard &  Wellekens]. it was  shown that  MLPs  were approximating Maximum  a Posteriori (MAP) probabilities  and  could  thus  be  embedded  as  an  emission  probability  estimator  in  HMMs.  By using contextual information from  a sliding window on the  input frames.  we  have  been  able  to  improve  frame  or phoneme  clas(cid:173) sification  performance  over the  corresponding  performance  for Simple  Maximum  Likelihood  (ML)  or even  MAP  probabilities  that  are  esti(cid:173) mated without the benefit of context.  However. recognition of words in  continuous speech was  not so simply improved by the use of an  MLP.  and  several  modifications  of the  original  scheme  were  necessary  for  getting acceptable performance.  It is  shown here that word recognition  performance for a  simple discrete density  HMM  system  appears  to  be  somewhat better when MLP methods are used to estimate the emission  probabilities.
Analysis of Linsker's Simulations of Hebbian Rules
Authors: David J. C. MacKay, Kenneth D. Miller
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Linsker has reported the development of centre---surround receptive  fields  and  oriented  receptive  fields  in  simulations  of a  Hebb-type  equation  in  a  linear  network.  The  dynamics  of the  learning  rule  are analysed in  terms of the eigenvectors of the covariance matrix  of cell activities.  Analytic and  computational results  for  Linsker's  covariance matrices,  and some general theorems,  lead  to an expla(cid:173) nation  of the  emergence  of centre---surround  and  certain  oriented  structures. 
Linsker  [Linsker,  1986,  Linsker,  1988]  has  studied  by  simulation  the  evolution  of  weight vectors under a  Hebb-type teacherless learning rule  in a  feed-forward  linear  network.  The equation for  the evolution of the weight vector w  of a  single neuron,  derived  by  ensemble  averaging  the  Hebbian  rule  over  the  statistics  of the  input  patterns,  is:! 
a at Wi = k! + L(Qij + k 2 )wj  subject to -Wmax  ~ Wi < Wmax
Analog Neural Networks of Limited Precision I: Computing with Multilinear Threshold Functions
Authors: Zoran Obradovic, Ian Parberry
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Experimental  evidence  has  shown  analog  neural  networks  to  be  ex(cid:173) ~mely fault-tolerant;  in  particular.  their  performance  does  not  ap(cid:173) pear  to  be  significantly  impaired  when  precision  is  limited.  Analog  neurons  with  limited  precision  essentially  compute  k-ary  weighted  multilinear threshold  functions.  which  divide  R"  into k  regions  with  k-l hyperplanes.  The behaviour of k-ary  neural networks  is  investi(cid:173) gated.  There  is  no  canonical  set  of  threshold  values  for  k>3.  although  they  exist  for  binary  and  ternary  neural  networks.  The  weights  can be  made  integers of only  0 «z +k ) log  (z +k » bits. where  z  is  the  number  of processors.  without  increasing  hardware  or  run(cid:173) ning  time.  The  weights  can  be  made  ±1  while  increasing  running  time  by a constant multiple and hardware by  a small polynomial  in  z  and  k.  Binary  neurons  can  be  used  if the  running  time  is allowed  to  increase  by  a larger constant  multiple  and  the  hardware  is  allowed  to  increase  by  a  slightly  larger polynomial  in  z  and k.  Any  symmetric  k-ary  function  can  be  computed  in  constant  depth  and  size  o (n k- 1/(k-2)!).  and  any  k-ary  function  can  be  computed  in  constant  depth and  size  0  (nk").  The alternating neural  networks of Olafsson  and  Abu-Mostafa.  and  the  quantized  neural  networks  of Fleisher  are  closely related  to this model. 
Analog Neural Networks of Limited Precision I 
703
Neural Implementation of Motivated Behavior: Feeding in an Artificial Insect
Authors: Randall D. Beer, Hillel J. Chiel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Most  complex  behaviors  appear  to be governed  by  internal  moti(cid:173) vational  states or  drives  that  modify  an  animal's  responses  to  its  environment.  It is  therefore of considerable  interest to understand  the  neural basis of these  motivational states.  Drawing upon work  on  the  neural  basis  of feeding  in  the  marine  mollusc  Aplysia,  we  have  developed  a  heterogeneous  artificial  neural  network  for  con(cid:173) trolling the feeding behavior of a simulated insect.  We demonstrate  that feeding in this artificial insect shares many characteristics with  the motivated behavior of natural animals.
A Neural Network to Detect Homologies in Proteins
Authors: Yoshua Bengio, Samy Bengio, Yannick Pouliot, Patrick Agin
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In order to detect the presence and location of immunoglobu(cid:173) lin (Ig) domains from amino acid sequences we built a system  based on a neural network with one hidden layer trained with  back propagation. The program was designed to efficiently  identify proteins exhibiting such domains, characterized by a  few localized conserved regions and a low overall homology.  When the National Biomedical Research Foundation (NBRF)  NEW protein sequence database was scanned to evaluate the  program's performance, we obtained very low rates of false  negatives coupled with a moderate rate of false positives.
Practical Characteristics of Neural Network and Conventional Pattern Classifiers on Artificial and Speech Problems
Authors: Yuchun Lee, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Eight  neural  net  and  conventional  pattern  classifiers  (Bayesian(cid:173) unimodal Gaussian, k-nearest neighbor, standard back-propagation,  adaptive-stepsize back-propagation, hypersphere, feature-map, learn(cid:173) ing vector  quantizer,  and  binary  decision  tree)  were  implemented  on  a  serial  computer  and  compared  using  two  speech  recognition  and two artificial tasks.  Error rates were statistically equivalent on  almost  all  tasks,  but classifiers  differed  by orders  of magnitude in  memory  requirements,  training time,  classification  time,  and ease  of adaptivity.  Nearest-neighbor  classifiers  trained  rapidly  but  re(cid:173) quired  the most memory.  Tree classifiers  provided  rapid classifica(cid:173) tion but  were  complex to adapt.  Back-propagation classifiers  typ(cid:173) ically  required  long  training  times  and  had intermediate  memory  requirements.  These results suggest that classifier selection should  often  depend  more  heavily  on  practical considerations  concerning  memory  and  computation  resources,  and  restrictions  on  training  and classification times than on error rate. 
-This  work  was  sponsored by  the  Department of the  Air  Force  and  the  Air  Force  Office  of 
Scientific Research. 
Practical Characteristics of Neural Network
Model Based Image Compression and Adaptive Data Representation by Interacting Filter Banks
Authors: Toshiaki Okamoto, Mitsuo Kawato, Toshio Inui, Sei Miyake
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: introduced. Based on 
To achieve high-rate image data compression  while maintainig a high quality reconstructed  image, a good image model and an efficient  way to represent the specific data of each  image must be  the  physiological knowledge of multi - channel  characteristics and  interactions  between them in the human visual system,  a mathematically coherent parallel architecture  for image data compression which utilizes the  Markov  Image model and  interactions between a vast number of filter  banks, is proposed.
Non-Boltzmann Dynamics in Networks of Spiking Neurons
Authors: Michael C. Crair, William Bialek
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We study networks  of spiking neurons  in which spikes  are fired  as  a  Poisson process.  The state of a  cell is  determined  by the  instan(cid:173) taneous  firing  rate,  and  in the  limit of high firing  rates our model  reduces  to  that  studied  by  Hopfield.  We  find  that  the  inclusion  of spiking  results  in several new  features,  such  as  a  noise-induced  asymmetry between "on" and "off" states of the cells and probabil(cid:173) ity currents which destroy the usual description of network dynam(cid:173) ics  in  terms  of energy  surfaces.  Taking account  of spikes  also  al(cid:173) lows us to calibrate network parameters such as  "synaptic weights"  against  experiments  on  real synapses.  Realistic  forms  of the  post  synaptic  response  alters  the  network dynamics,  which  suggests  a  novel dynamical learning mechanism.
Performance Comparisons Between Backpropagation Networks and Classification Trees on Three Real-World Applications
Authors: Les E. Atlas, Ronald A. Cole, Jerome T. Connor, Mohamed A. El-Sharkawi, Robert J. Marks II, Yeshwant K. Muthusamy, Etienne Barnard
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Multi-layer  perceptrons  and  trained  classification  trees  are  two  very  different  techniques  which  have  recently  become  popular.  Given  enough  data  and  time,  both  methods  are  capable  of performing  arbi(cid:173) trary  non-linear  classification.  We  first  consider  the  important  differences  between  multi-layer  perceptrons  and  classification  trees  and  conclude  that  there  is  not enough  theoretical  basis  for  the  clear(cid:173) cut  superiority  of one  technique  over  the  other.  For  this  reason,  we  performed  a  number  of empirical  tests  on  three  real-world  problems  in  power  system  load  forecasting,  power  system  security  prediction,  and  speaker-independent  vowel  identification.  In  all  cases,  even  for  piecewise-linear  trees,  the  multi-layer  perceptron  performed  as  well  as or better than  the trained classification  trees. 
Performance Comparisons 
623
An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2
Authors: Xiru Zhang, Michael McKenna, Jill P. Mesirov, David L. Waltz
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In this paper, we  present a novel implementation of the widely used  Back-propagation neural net learning algorithm on the Connection  Machine  CM-2  - a  general  purpose,  massively  parallel  computer  with a hypercube topology.  This implementation runs at about 180  million  interconnections per second  (IPS)  on a  64K  processor  CM- 2.  The  main  interprocessor  communication operation  used  is  2D  nearest  neighbor  communication.  The  techniques  developed  here  can  be  easily  extended  to implement  other algorithms for  layered  neural nets on  the CM-2,  or on other massively parallel computers  which have 2D or higher degree connections among their processors.
Algorithms for Better Representation and Faster Learning in Radial Basis Function Networks
Authors: Avijit Saha, James D. Keeler
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: in 
learning,  using  radial  basis  functions 
In  this  paper  we  present  upper  bounds  for  the  learning  rates  for  hybrid  models  that  employ  a  combination  of  both  self-organized  and  supervised  to  build  receptive  field  representations  the  hidden  units.  The  learning  performance  in  such  networks  with  nearest  neighbor  heuristic  can  be  improved  upon  by  multiplying  the  individual  receptive  field  widths  by  a  suitable  overlap  factor.  We  present  results  indicat!ng  optimal  values  for  such  overlap  factors.  We  also  present  a  new  algorithm  for  determining  receptive  field  centers.  This  method  negotiates  more  hidden  units  in  the  regions  of the  input  space  as  a  function  of the  output  and  is  conducive  to  better  learning  when  the  number of patterns (hidden units) is small.
The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN
Authors: John Kassebaum, Manoel Fernando Tenorio, Christoph Schaefers
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: This  work  introduces  a  new  method  called  Self  Organizing  Neural  Network  (SONN)  algorithm  and  compares  its  performance  with Back  Propagation  in  a  signal  separation  application.  The  problem  is  to  separate  two  signals;  a  modem  data signal  and  a  male  speech  signal,  added and transmitted  through  a  4 khz  channel.  The signals  are sam(cid:173) pled  at  8  khz,  and  using  supervised  learning,  an  attempt  is  made  to  reconstruct  them.  The  SONN  is  an  algorithm  that  constructs  its  own  network  topology  during  training,  which  is  shown  to  be  much  smaller  than  the  BP  network,  faster  to  trained,  and  free  from  the  trial-and(cid:173) error network design that characterize BP.
Real-Time Computer Vision and Robotics Using Analog VLSI Circuits
Authors: Christof Koch, Wyeth Bair, John G. Harris, Timothy K. Horiuchi, Andrew Hsu, Jin Luo
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The long-term goal of our laboratory is  the development of analog  resistive  network-based  VLSI implementations of early  and  inter(cid:173) mediate  vision  algorithms.  We  demonstrate  an  experimental cir(cid:173) cuit  for  smoothing  and  segmenting  noisy  and  sparse  depth  data  using  the  resistive  fuse  and  a  1-D  edge-detection  circuit  for  com(cid:173) puting zero-crossings using two  resistive grids with different space(cid:173) constants.  To  demonstrate  the  robustness  of our  algorithms  and  of the fabricated analog CMOS VLSI chips, we  are mounting these  circuits  onto  small  mobile  vehicles  operating in  a  real-time,  labo(cid:173) ratory environment.
A Large-Scale Neural Network Which Recognizes Handwritten Kanji Characters
Authors: Yoshihiro Mori, Kazuki Joe
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We propose a  new  way  to construct a  large-scale neural  network  for  3.000 handwritten  Kanji  characters  recognition.  This  neural  network  consists  of 3  parts:  a  collection  of small-scale  networks  which  are  trained individually on a small number of Kanji characters; a network  which  integrates  the  output  from  the  small-scale  networks,  and  a  process to facilitate  the integration of these neworks. The recognition  rate of the  total  system  is  comparable  with  those  of the  small-scale  networks. Our results indicate that the proposed method is effective for  constructing  a  large-scale  network  without  loss  of  recognition  performance.
TRAFFIC: Recognizing Objects Using Hierarchical Reference Frame Transformations
Authors: Richard S. Zemel, Michael Mozer, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  describe a model that can recognize  two-dimensional shapes in  an  unsegmented  image,  independent  of their orientation,  position,  and scale.  The model,  called TRAFFIC, efficiently  represents  the  structural  relation  between  an  object  and  each  of its  component  features  by  encoding  the fixed  viewpoint-invariant transformation  from the feature's reference frame to the object's in the weights of a  connectionist  network.  Using  a  hierarchy  of such  transformations,  with increasing complexity of features  at each successive  layer,  the  network  can  recognize  multiple objects  in parallel.  An implemen(cid:173) tation  of TRAFFIC  is  described,  along with  experimental  results  demonstrating  the  network's  ability to  recognize  constellations  of  stars in  a viewpoint-invariant manner.
Performance of Connectionist Learning Algorithms on 2-D SIMD Processor Arrays
Authors: Fernando J. Nuñez, José A. B. Fortes
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  mapping  of  the  back-propagation  and  mean  field  theory  learning  algorithms  onto  a  generic  2-D  SIMD  computer  is  described.  This  architecture proves to  be very  adequate for  these  applications  since  efficiencies  close  to  the  optimum  can  be  attained.  Expressions  to  find  the  learning  rates  are  given  and  then particularized to the DAP  array procesor.
Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility
Authors: Terrence J. Sejnowski, Ben P. Yuhas, Moise H. Goldstein Jr., Robert E. Jenkins
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: R.E. Jenkins  The Applied  Physics  Laboratory  The Johns Hopkins  University  Laurel,  MD  20707 
Acoustic speech recognition degrades in the presence of noise.  Com(cid:173) pensatory  information  is  available  from  the  visual  speech  signals  around  the  speaker's  mouth.  Previous  attempts  at  using  these  visual speech signals to improve automatic speech  recognition sys(cid:173) tems have combined the acoustic and visual speech information at a  symbolic level  using heuristic rules.  In this paper, we  demonstrate  an  alternative  approach  to fusing  the  visual  and  acoustic  speech  information  by  training  feedforward  neural  networks  to  map  the  visual signal onto the corresponding short-term spectral amplitude  envelope  (STSAE)  of the  acoustic  signal.  This  information  can  be  directly  combined  with  the  degraded  acoustic  STSAE.  Signif(cid:173) icant  improvements  are  demonstrated  in  vowel  recognition  from  noise-degraded  acoustic signals.  These results are compared to the  performance of humans, as well  as other pattern matching and es(cid:173) timation algorithms.
The Computation of Sound Source Elevation in the Barn Owl
Authors: Clay D. Spence, John C. Pearson
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The midbrain of the barn owl contains a map-like representation of  sound source direction which is used to precisely orient the head to(cid:173) ward targets of interest. Elevation is computed from the interaural  difference in sound level. We present models and computer simula(cid:173) tions of two stages of level difference processing which qualitatively  agree with known anatomy and physiology, and make several strik(cid:173) ing predictions.