mtybilly/Instruct2DS huggingface dataset:
The Effects of Circuit Integration on a Feature Map Vector Quantizer
Authors: Jim Mann
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The effects  of parameter  modifications  imposed  by  hardware con(cid:173) straints on  a self-organizing feature  map  algorithm were examined.  Performance was  measured  by  the  error  rate  of a  speech  recogni(cid:173) tion  system which  included  this  algorithm  as  part of the  front-end  processing.  System parameters  which  were  varied  included  weight  (connection  strength)  quantization,  adap tation  quantization,  dis(cid:173) tance  measures  and  circuit  approximations  which  include  device  characteristics  and  process  variability.  Experiments  using  the  TI  isolated word database for  16 speakers demonstrated degradation in  performance when  weight quantization fell  below 8 bits.  The com(cid:173) petitive nature  of the  algorithm  rela..xes  constraints on  uniformity  and  linearity which makes it an  excellent candidate for  a fully  ana(cid:173) log  circuit implementation.  Prototype circuits have been fabricated  and characterized following  the constraints established through the  simulation efforts.
Generalization and Parameter Estimation in Feedforward Nets: Some Experiments
Authors: H. Bourlard N. Morgan
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We have done an empirical study of the relation of the number of  parameters (weights) in a feedforward net to generalization perfor(cid:173) mance. Two experiments are reported. In one, we use simulated data  sets with well-controlled parameters, such as the signal-to-noise ratio  of continuous-valued data. In the second, we train the network on  vector-quantized mel cepstra from real speech samples. In each case,  we use back-propagation to train the feedforward net to discriminate in  a multiple class pattern classification problem. We report the results of  these studies, and show the application of cross-validation techniques  to prevent overfitting.
VLSI Implementation of a High-Capacity Neural Network Associative Memory
Authors: Rodney M. Goodman Tzi-Dar Chiueh
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In  this  paper we  describe  the  VLSI  design  and  testing of a  high  capacity  associative  memory  which  we  call  the  exponential  cor(cid:173) relation  associative  memory  (ECAM).  The  prototype  3J.'-CMOS  programmable  chip  is  capable  of storing  32  memory  patterns  of  24 bits each.  The high capacity of the ECAM is  partly due  to the  use of special exponentiation neurons, which  are implemented via  sub-threshold MOS  transistors in this design.  The prototype chip  is  capable of performing one associative recall in  3  J.'S.
The Cascade-Correlation Learning Architecture
Authors: Christian Lebiere Scott E. Fahlman
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Cascade-Correlation is a new architecture and supervised learning algo(cid:173) rithm for artificial neural networks.  Instead of just adjusting the weights  in a network of fixed topology. Cascade-Correlation begins with a min(cid:173) imal network,  then automatically trains  and adds new hidden  units  one  by  one,  creating a  multi-layer structure.  Once  a  new  hidden  unit  has  been added  to the network, its  input-side weights are frozen.  This  unit  then becomes a permanent feature-detector in the network, available for  producing  outputs  or for  creating other,  more complex  feature  detec(cid:173) tors.  The Cascade-Correlation architecture has  several advantages over  existing algorithms:  it  learns  very quickly,  the network . determines  its  own size and  topology, it retains  the structures  it  has  built even  if the  training set changes, and it requires no back-propagation of error signals  through  the connections of the network. 
1  DESCRIPTION OF CASCADE·CORRELATION  The  most  important  problem  preventing  the  widespread  application  of artificial  neural  networks  to real-world problems  is  the slowness  of existing learning algorithms  such  as  back-propagation  (or "backprop").  One factor contributing to  that  slowness  is  what we  call  the moving  target problem:  because all of the  weights  in  the network are changing  at once,  each  hidden  units  sees  a  constantly changing environment.  Instead  of moving  quickly to assume useful roles in the overall problem solution, the hidden units engage in  a complex dance with much  wasted motion.  The Cascade-Correlation learning algorithm  was  developed in an  attempt to solve that problem.  In  the problems  we have examined,  it learns  much faster than back-propagation and  solves  some other problems as  well. 
The Cascade-Correlation Learning Architecture 
525
Sigma-Pi Learning: On Radial Basis Functions and Cortical Associative Learning
Authors: Christof Koch Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  goal  in  this  work  has  been  to  identify  the  neuronal  elements  of the cortical column that are most likely to support  the learning  of nonlinear associative  maps.  We show that  a  particular style  of  network learning algorithm based on locally-tuned  receptive fields  maps  naturally  onto cortical  hardware,  and  gives  coherence  to  a  variety of features  of cortical anatomy,  physiology,  and  biophysics  whose  relations to learning remain poorly understood.
Optimal Brain Damage
Authors: John S. Denker, Sara A. Solla Yann LeCun
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  have used  information-theoretic ideas  to derive  a class of prac(cid:173) tical  and  nearly  optimal schemes  for  adapting the size  of a  neural  network.  By  removing  unimportant  weights  from  a  network,  sev(cid:173) eral  improvements  can  be  expected:  better  generalization,  fewer  training examples required,  and improved speed  of learning and/or  classification.  The  basic  idea  is  to  use  second-derivative  informa(cid:173) tion to make a  tradeoff between  network  complexity  and  training  set error.  Experiments confirm  the usefulness  of the methods on a  real-world  application.
Learning Aspect Graph Representations from View Sequences
Authors: Allen M. Waxman Michael Seibert
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In our effort to develop a modular neural system for invariant learn(cid:173) ing and  recognition of 3D objects,  we  introduce here a new  module  architecture  called  an  aspect  network constructed  around  adaptive  axo-axo-dendritic synapses.  This  builds  upon  our existing system  (Seibert & Waxman, 1989) which  processes  20 shapes and classifies  t.hem  into  view  categories  (i.e.,  aspects)  invariant  to  illumination,  position,  orientat.ion,  scale,  and  projective  deformations.  From  a  sequence 'of  views,  the  aspect  network  learns  the  transitions  be(cid:173) tween  these  aspects,  crystallizing  a  graph-like  structure  from  an  initially  amorphous  network .  Object  recognition  emerges  by  ac(cid:173) cumulating evidence over  multiple views  which  activate competing  object  hypotheses.
The Effect of Catecholamines on Performance: From Unit to System Behavior
Authors: Harry Printz, Jonathan D. Cohen David Servan-Schreiber
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: At the level of individual neurons. catecholamine release increases  the  responsivity  of cells  to  excitatory and  inhibitory  inputs.  We  present a  model  of catecholamine effects  in  a  network  of neural-like  elements.  We  argue  that  changes  in  the  responsivity  of individual  elements  do  not  affect  their  ability  to  detect  a  signal  and  ignore  noise.  However.  the same changes in cell responsivity in a network of such elements do  improve the signal detection performance of the network as a whole.  We  show how  this result can be used in a computer simulation of behavior  to  account  for  the  effect  of eNS  stimulants  on  the  signal  detection  performance of human subjects.
Meiosis Networks
Authors: Stephen Jose Hanson
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  central  problem  in  connectionist  modelling  is  the  control  of  network  and  architectural  resources  during  learning.  In  the  present  approach,  weights  reflect  a  coarse  prediction  history  as  coded  by  a  distribution  of  values  and  parameterized  in  the  mean  and  standard  deviation  of  these  weight  distributions.  Weight  updates  are  a  function  of  both  the  mean  and  standard  deviation  of  each  connection  in  the  network and vary  as  a  function  of  the  error signal  ("stochastic  delta  rule";  Hanson,  1990).  Consequently,  the  weights  their  maintain  in  "uncertainty"  establishing  a  policy  concerning  the  size  of  the  nodal  complexity  of  the  network  and  growth  of  new  nodes.  For  example,  during  problem  solving  the  present  network  can  undergo  "meiosis",  producing  two  nodes  where  there  was  one  "overtaxed"  node  as  measured  by  its  coefficient of variation.  It  is  shown  in  a  number of  benchmark  problems  that  meiosis  networks  can  find  minimal  architectures,  reduce  computational complexity,  and overall  increase  the efficiency of the representation learning interaction.
Synergy of Clustering Multiple Back Propagation Networks
Authors: Josef Skrzypek William P. Lincoln
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The properties of a cluster of multiple back-propagation (BP) networks  are  examined  and  compared  to  the  performance  of a  single  BP  net(cid:173) work.  The underlying idea is that a synergistic effect within the cluster  improves the perfonnance and fault tolerance.  Five networks were ini(cid:173) tially  trained  to  perfonn  the  same  input-output  mapping.  Following  training, a cluster was created by computing an average of the outputs  generated by the individual networks.  The output of the cluster can be  used as the desired output during training by feeding it back to the indi(cid:173) vidual  networks.  In  comparison  to  a  single  BP  network,  a  cluster  of  multiple  BP's generalization  and  significant fault  tolerance.  It appear  that cluster advantage follows from  simple maxim  "you can fool  some  of the single BP's in a cluster all of the time but you cannot fool all of  them all of the time"  {Lincoln}
Incremental Parsing by Modular Recurrent Connectionist Networks
Authors: Alex Waibel Ajay N. Jain
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present a novel,  modular, recurrent connectionist network architec(cid:173) ture  which  learns  to robustly  perform  incremental  parsing of complex  sentences.  From  sequential  input,  one  word  at  a  time,  our  networks  learn  to  do  semantic  role  assignment,  noun  phrase  attachment,  and  clause structure recognition for sentences with passive constructions and  center embedded  clauses.  The  networks  make  syntactic  and  semantic  predictions  at every point in time, and previous  predictions are revised  as  expectations are affirmed or violated with the arrival of new informa(cid:173) tion.  Our networks  induce  their own "grammar rules" for dynamically  transforming  an  input  sequence of words  into a  syntactic/semantic  in(cid:173) terpretation.  These networks  generalize  and  display  tolerance  to  input  which  has  been  corrupted  in ways common in spoken  language.
Reading a Neural Code
Authors: Fred Rieke, Robert R. de Ruyter van Steveninck, David Warland William Bialek
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Traditional methods of studying neural coding characterize  the en(cid:173) coding  of known  stimuli  in  average  neural  responses.  Organisms  face  nearly the opposite task - decoding short segments of a spike  train to extract information about an unknown, time-varying stim(cid:173) ulus.  Here  we  present strategies for  characterizing the  neural code  from  the  point of view  of the  organism, culminating in  algorithms  for  real-time  stimulus reconstruction  based  on  a  single  sample  of  the spike train.  These methods are applied to the design and anal(cid:173) ysis  of experiments on an  identified movement-sensitive neuron  in  the fly  visual system.  As far  as we  know this is  the first  instance in  which a direct  "reading"  of the neural code has been  accomplished.
A Method for the Associative Storage of Analog Vectors
Authors: Yaser S. Abu-Mostafa Amir F. Atiya
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A method for  storing analog vectors in  Hopfield's continuous feed(cid:173) back model is proposed.  By  analog vectors we mean vectors whose  components  are  real-valued.  The  vectors  to  be  stored  are  set  as  equilibria of the network.  The network model consists of one layer  of visible  neurons  and  one  layer  of hidden  neurons.  We  propose  a  learning  algorithm,  which  results  in  adjusting  the  positions  of  the  equilibria,  as  well  as  guaranteeing  their  stability.  Simulation  results confirm the effectiveness of the method .
Using Local Models to Control Movement
Authors: Christopher G. Atkeson
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: This  paper  explores  the  use  of a  model  neural  network  for  motor  learning.  Steinbuch and Taylor presented neural network designs to  do nearest  neighbor lookup in the early 1960s.  In  this  paper their  nearest neighbor network is augmented with a local model network,  which fits  a local model to a set of nearest neighbors.  The network  design  is  equivalent to local  regression.  This  network  architecture  can  represent  smooth  nonlinear functions,  yet  has  simple  training  rules  with  a  single  global  optimum.  The  network  has  been  used  for  motor  learning  of  a  simulated  arm  and  a  simulated  running  machine.
Learning in Higher-Order "Artificial Dendritic Trees
Authors: Tony Bell
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: If neurons  sum  up  their  inputs  in  a  non-linear  way,  as  some  simula(cid:173) tions  suggest,  how  is  this  distributed  fine-grained  non-linearity  ex(cid:173) ploited  during  learning?  How  are  all  the  small  sigmoids  in  synapse,  spine  and  dendritic  tree  lined  up  in  the  right areas  of their respective  input  spaces?  In  this report,  I show  how  an  abstract atemporal  highly  nested  tree  structure  with  a  quadratic  transfer  function  associated  with  each  branchpoint,  can  self organise  using  only  a  single  global  reinforcement  scalar,  to  perform  binary  classification  tasks.  The  pro(cid:173) cedure  works  well,  solving  the 6-multiplexer and  a difficult phoneme  classification  task  as  well  as  back-propagation  does,  and  faster.  Furthermore, it does not calculate an  error gradient, but uses  a  statist(cid:173) ical  scheme  to  build  moving  models of the  reinforcement signal.
Dynamic Behavior of Constained Back-Propagation Networks
Authors: Yves Chauvin
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The learning dynamics of the back-propagation algorithm are in(cid:173) vestigated when complexity constraints are added to the standard  Least Mean Square (LMS) cost function. It is shown that loss of  generalization performance due to overtraining can be avoided  when using such complexity constraints. Furthermore, "energy,"  hidden representations and weight distributions are observed and  compared during learning. An attempt is made at explaining the  results in terms of linear and non-linear effects in relation to the  gradient descent learning algorithm.
Designing Application-Specific Neural Networks Using the Genetic Algorithm
Authors: Tariq Samad, Aloke Guha Steven A. Harp
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present  a  general  and  systematic  method  for  neural  network  design  based  on  the  genetic  algorithm.  The  technique  works  in  conjunction  with  network  learning  rules,  addressing  aspects  of  the  network's  gross  architecture,  connectivity,  and  learning  rule  parameters.  Networks  can  be  optimiled  for  various  application(cid:173) specific  criteria, such  as  learning speed, generalilation,  robustness  and  connectivity.  The  approach  is  model-independent.  We  describe  a  prototype  system,  NeuroGENESYS,  that employs  the  backpropagation  learning  rule.  Experiments  on  several  small  problems  have  been  conducted.  In  each  case,  NeuroGENESYS  has  produced  networks  that perform significantly  better than  the  randomly  generated  networks  of its initial population.  The  com(cid:173) putational feasibility  of our approach  is  discussed.
A Computational Basis for Phonology
Authors: Deirdre W. Wheeler David S. Touretzky
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The phonological structure of human  languages  is  intricate,  yet highly  constrained.  Through  a  combination  of connectionist  modeling  and  linguistic  analysis,  we are attempting to develop a computational basis  for  the  nature  of phonology.  We  present  a  connectionist  architecture  that  performs  multiple  simultaneous  insertion,  deletion,  and  mutation  operations on sequences  of phonemes, and introduce a novel additional  primitive,  clustering.  Clustering  provides  an  interesting  alternative  to  both iterative and relaxation accounts of assimilation processes  such as  vowel  harmony.  Our resulting  model  is  efficient because it processes  utterances entirely in parallel  using  only  feed-forward circuitry.
Neural Network Simulation of Somatosensory Representational Plasticity
Authors: Michael Merzenich Kamil A. Grajski
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  brain  represents  the  skin  surface  as  a  topographic  map  in  the  somatosensory  cortex.  This  map  has  been  shown  experimentally  to  be  modifiable  in  a  use-dependent  fashion  throughout  life.  We  present  a  neural  network  simulation  of  the  competitive  dynamics  underlying  this  cortical  plasticity  by  detailed  analysis  of  receptive  field  properties  of  model  neurons  during  simulations  of  skin  co(cid:173) activation, cortical  lesion,  digit amputation and nerve  section.
A Neural Network for Feature Extraction
Authors: Nathan Intrator
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  paper suggests a  statistical framework for  the  parameter esti(cid:173) mation  problem associated  with  unsupervised  learning in  a  neural  network, leading to an exploratory projection pursuit network that  performs  feature  extraction, or  dimensionality  reduction.
Generalized Hopfield Networks and Nonlinear Optimization
Authors: Athanasios G. Tsirukis, Manoel Fernando Tenorio Gintaras V. Reklaitis
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  nonlinear  neural  framework,  called  the  Generalized  Hopfield  network,  is  proposed,  which  is  able  to  solve  in  a  parallel  distributed  manner systems  of nonlinear equations.  The  method is  applied  to  the  general  nonlinear  optimization  problem.  We  demonstrate  GHNs  implementing  the  three  most  important  optimization  algorithms,  namely the Augmented Lagrangian, Generalized Reduced Gradient and  Successive  Quadratic  Programming  methods.  The  study  results  in  a  dynamic  view of the optimization problem and offers a straightforward  model  for  the  parallelization  of  the  optimization  computations,  thus  significantly  extending  the  practical  limits  of problems  that  can  be  formulated  as  an  optimization  problem  and  which  can  gain  from  the  introduction of nonlinearities in their structure (eg. pattern recognition,  supervised learning, design of content-addressable memories). 
1 To whom correspondence should be addressed. 
356 
Reklaitis, Tsirukis and Tenorio 
1  RELATED  WORK  The  ability  of networks  of highly  interconnected  simple  nonlinear  analog  processors  (neurons)  to  solve  complicated  optimization  problems  was  demonstrated  in  a  series  of  papers by Hopfield and Tank (Hopfield,  1984), (Tank, 1986).  The  Hopfield  computational  model  is  almost  exclusively  applied  to  the  solution  of  combinatorially  complex  linear  decision  problems  (eg.  Traveling  Salesman  Problem).  Unfortunately  such problems can  not be  solved with  guaranteed  quality,  (Bruck,  1987),  getting trapped in locally optimal solutions.  Jeffrey  and  Rossner,  (Jeffrey,  1986),  extended  Hopfield's  technique  to  the  nonlinear  unconstrained  optimization  problem,  using  Cauchy  dynamics.  Kennedy  and  Chua,  (Kennedy,  1988), presented an  analog  implementation of a  network  solving a  nonlinear  optimization problem.  The underlying optimization algorithm  is a  simple transformation  method, (Reklaitis,  1983), which is known  to be relatively inefficient for large nonlinear  optimization problems. 
2  LINEAR  HOPFIELD  NETWORK  (LHN)  The computation in a Hopfield network is done by  a collection of highly interconnected  simple  neurons.  Each processing element,  i,  is characterized by  the activation  level,  Ui,  which is a function of the input received from  the external environment, Ii, and  the state  of the other neurons. The activation level of i is transmitted to  the other processors, after  passing through a filter that converts Ui to a 0-1 binary value, Vi'  The time behavior of the system is described by the following model: 
~ T·V·  - -' + I·  '  ~ 'J  J  J
Connectionist Architectures for Multi-Speaker Phoneme Recognition
Authors: Alex Waibel John B. Hampshire II
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present  a  number  of Time-Delay  Neural  Network  (TDNN)  based  architectures  for multi-speaker phoneme recognition (/b,d,g/ task).  We  use  speech  of two females  and four males to  compare the performance  of the various architectures against a baseline recognition rate of 95.9%  for a single IDNN on the six-speaker /b,d,g/ task.  This series of modu(cid:173) lar designs leads to a highly modular multi-network architecture capable  of performing the six-speaker recognition task at the speaker dependent  rate  of 98.4%.  In  addition  to  its  high  recognition  rate,  the  so-called  "Meta-Pi"  architecture  learns - without direct  supervision - ognize the speech of one particular male speaker using internal models  of other male  speakers exclusively.
Bayesian Inference of Regular Grammar and Markov Source Models
Authors: Michael I. Miller Kurt R. Smith
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In this paper we develop a Bayes criterion which includes the Rissanen  complexity, for  inferring regular grammar models.  We develop two  methods for regular grammar Bayesian inference.  The fIrst method is  based  on  treating  the  regular  grammar as  a  I-dimensional  Markov  source, and the second is based on the combinatoric characteristics of  the regular grammar itself.  We apply the resulting Bayes criteria to a  particular example in order to show the efficiency of each method.
Dataflow Architectures: Flexible Platforms for Neural Network Simulation
Authors: Ira Smotroff
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Dataflow architectures are general computation engines optimized for  the execution of fme-grain parallel algorithms. Neural networks can be  simulated on  these systems with  certain advantages.  In this paper, we  review  dataflow  architectures,  examine  neural  network  simulation  performance  on  a  new  generation  dataflow  machine,  compare  that  performance to other simulation alternatives, and discuss the benefits  and drawbacks of the dataflow approach. 
1  DATAFLOW ARCHITECTURES  Dataflow research has been conducted at MIT (Arvind &  Culler,  1986) and elsewhere  (Hiraki,  et.  aI.,  1987)  for  a  number  of  years.  Dataflow  architectures  are  general  computation engines that treat each instruction of a program as a separate task which is  scheduled in an asynchronous, data-driven fashion.  Dataflow programs are compiled into  graphs which explicitly describe the data dependencies of the computation. These graphs  are directly executed by the machine. Computations which are not linked by a path in the  graphs  can  be executed  in  parallel.  Each  machine  has  a  large  number  of processing  elements  with  hardware  that  is  optimized  to  reduce  task  switching  overhead  to  a  minimum.  As each  computation  executes  and  produces  a  result,  it  causes  all  of the  following computations that require the result to be scheduled. In this manner, fine grain  parallel  computation  is achieved,  with  the limit on  the  amount of possible parallelism  determined by the problem and the number of processing elements in the machine. 
Dataflow Architectures:  Flexible Platforms for Neural Network Simulation
The Perceptron Algorithm Is Fast for Non-Malicious Distributions
Authors: Eric B. Baum
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Within  the  context  of Valiant's  protocol  for  learning,  the  Perceptron 
algorithm is shown  to learn  an arbitrary half-space in time O(r;;) if D, the proba(cid:173)
bility distribution of examples,  is  taken uniform over the unit sphere sn.  Here  f  is 
the accuracy parameter.  This is surprisingly fast,  as  "standard"  approaches involve 
solution  of a  linear  programming problem involving  O( 7')  constraints in  n  dimen(cid:173)
sions.  A  modification  of Valiant's  distribution  independent  protocol  for  learning 
is  proposed  in which  the  distribution  and  the function  to be learned  may be  cho(cid:173)
sen  by adversaries,  however  these  adversaries may  not  communicate.  It is  argued 
that  this  definition  is  more  reasonable  and  applicable  to real  world  learning  than 
Valiant's.  Under  this  definition,  the  Perceptron algorithm  is  shown to be  a  distri(cid:173)
bution independent learning algorithm.  In  an  appendix we  show  that, for  uniform 
distributions,  some  classes  of infinite  V-C  dimension  including  convex  sets  and  a 
class of nested  differences of convex sets are learnable.
Discovering High Order Features with Mean Field Modules
Authors: Geoffrey E. Hinton Conrad C. Galland
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A new  form of the deterministic Boltzmann machine (DBM) learn(cid:173) ing procedure is presented  which can efficiently train network mod(cid:173) ules  to discriminate  between  input  vectors  according  to some  cri(cid:173) terion.  The new  technique directly utilizes the free  energy  of these  "mean field  modules"  to represent the probability that the criterion  is  met,  the  free  energy  being readily  manipulated by  the  learning  procedure.  Although  conventional deterministic  Boltzmann learn(cid:173) ing  fails  to  extract  the  higher  order  feature  of shift  at  a  network  bottleneck,  combining  the  new  mean  field  modules  with  the  mu(cid:173) tual information objective function  rapidly produces  modules that  perfectly extract this important higher order feature without direct  external supervision.
Sequential Decision Problems and Neural Networks
Authors: R. S. Sutton, C. J. C. H. Watkins A. G. Barto
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Decision making tasks that involve delayed consequences are very  common yet difficult to address with supervised learning methods.  If there is an accurate model of the underlying dynamical system,  then these tasks can be formulated as sequential decision problems  and solved by Dynamic Programming. This paper discusses rein(cid:173) forcement learning in terms of the sequential decision framework  and shows how a learning algorithm similar to the one implemented  by the Adaptive Critic Element used in the pole-balancer of Barto,  Sutton, and Anderson (1983), and further developed by Sutton  (1984), fits into this framework. Adaptive neural networks can  play significant roles as modules for approximating the functions  required for solving sequential decision problems.
Asymptotic Convergence of Backpropagation: Numerical Experiments
Authors: Gerald Tesauro, Yu He Subutai Ahmad
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Yu  He  Dept.  of Physics  Ohio  State Univ.  Columbus,  OH 43212 
We  have  calculated, both analytically and in simulations,  the rate  of convergence  at  long  times  in  the  backpropagation learning  al(cid:173) gorithm  for  networks  with  and  without  hidden  units.  Our  basic  finding for units using the standard sigmoid transfer function is  lit  convergence of the  error for  large t,  with  at most logarithmic cor(cid:173) rections  for  networks  with  hidden  units.  Other transfer functions  may lead to a  8lower polynomial rate of convergence.  Our analytic  calculations were presented in (Tesauro, He &  Ahamd, 1989).  Here  we  focus  in more detail on our empirical measurements of the con(cid:173) vergence rate in numerical simulations,  which  confirm our analytic  results.
Unsupervised Learning in Neurodynamics Using the Phase Velocity Field Approach
Authors: Nikzad Benny Toomarian Michail Zak
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  new  concept for  unsupervised  learning based  upon  examples in(cid:173) troduced  to the neural  network  is  proposed.  Each example is  con(cid:173) sidered  as  an  interpolation  node  of the  velocity field  in  the  phase  space.  The velocities  at  these  nodes  are selected such  that all  the  streamlines converge  to an attracting set imbedded in the subspace  occupied by the cluster of examples.  The synaptic interconnections  are  found  from  learning  procedure  providing  selected  field.  The  theory  is  illustrated  by examples. 
This  paper  is  devoted  to  development  of a  new  concept  for  unsupervised  learning  based upon examples introduced to an artificial neural network.  The neural network  is  considered  as  an  adaptive  nonlinear  dissipative  dynamical  system  described  by  the following  coupled  differential equations: 
Ui + K,Ui  = L 11j g( Uj )  + Ii
A Continuous Speech Recognition System Embedding MLP into HMM
Authors: Nelson Morgan Hervé Bourlard
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  are  developing  a  phoneme  based.  speaker-dependent  continuous  speech  recognition  system  embedding  a  Multilayer Perceptron  (MLP)  (Le .•  a  feedforward  Artificial  Neural  Network).  into  a Hidden  Markov  Model (HMM) approach.  In [Bourlard &  Wellekens]. it was  shown that  MLPs  were approximating Maximum  a Posteriori (MAP) probabilities  and  could  thus  be  embedded  as  an  emission  probability  estimator  in  HMMs.  By using contextual information from  a sliding window on the  input frames.  we  have  been  able  to  improve  frame  or phoneme  clas(cid:173) sification  performance  over the  corresponding  performance  for Simple  Maximum  Likelihood  (ML)  or even  MAP  probabilities  that  are  esti(cid:173) mated without the benefit of context.  However. recognition of words in  continuous speech was  not so simply improved by the use of an  MLP.  and  several  modifications  of the  original  scheme  were  necessary  for  getting acceptable performance.  It is  shown here that word recognition  performance for a  simple discrete density  HMM  system  appears  to  be  somewhat better when MLP methods are used to estimate the emission  probabilities.
Analysis of Linsker's Simulations of Hebbian Rules
Authors: Kenneth D. Miller David J. C. MacKay
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Linsker has reported the development of centre---surround receptive  fields  and  oriented  receptive  fields  in  simulations  of a  Hebb-type  equation  in  a  linear  network.  The  dynamics  of the  learning  rule  are analysed in  terms of the eigenvectors of the covariance matrix  of cell activities.  Analytic and  computational results  for  Linsker's  covariance matrices,  and some general theorems,  lead  to an expla(cid:173) nation  of the  emergence  of centre---surround  and  certain  oriented  structures. 
Linsker  [Linsker,  1986,  Linsker,  1988]  has  studied  by  simulation  the  evolution  of  weight vectors under a  Hebb-type teacherless learning rule  in a  feed-forward  linear  network.  The equation for  the evolution of the weight vector w  of a  single neuron,  derived  by  ensemble  averaging  the  Hebbian  rule  over  the  statistics  of the  input  patterns,  is:! 
a at Wi = k! + L(Qij + k 2 )wj  subject to -Wmax  ~ Wi < Wmax
Analog Neural Networks of Limited Precision I: Computing with Multilinear Threshold Functions
Authors: Ian Parberry Zoran Obradovic
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Experimental  evidence  has  shown  analog  neural  networks  to  be  ex(cid:173) ~mely fault-tolerant;  in  particular.  their  performance  does  not  ap(cid:173) pear  to  be  significantly  impaired  when  precision  is  limited.  Analog  neurons  with  limited  precision  essentially  compute  k-ary  weighted  multilinear threshold  functions.  which  divide  R"  into k  regions  with  k-l hyperplanes.  The behaviour of k-ary  neural networks  is  investi(cid:173) gated.  There  is  no  canonical  set  of  threshold  values  for  k>3.  although  they  exist  for  binary  and  ternary  neural  networks.  The  weights  can be  made  integers of only  0 «z +k ) log  (z +k » bits. where  z  is  the  number  of processors.  without  increasing  hardware  or  run(cid:173) ning  time.  The  weights  can  be  made  ±1  while  increasing  running  time  by a constant multiple and hardware by  a small polynomial  in  z  and  k.  Binary  neurons  can  be  used  if the  running  time  is allowed  to  increase  by  a larger constant  multiple  and  the  hardware  is  allowed  to  increase  by  a  slightly  larger polynomial  in  z  and k.  Any  symmetric  k-ary  function  can  be  computed  in  constant  depth  and  size  o (n k- 1/(k-2)!).  and  any  k-ary  function  can  be  computed  in  constant  depth and  size  0  (nk").  The alternating neural  networks of Olafsson  and  Abu-Mostafa.  and  the  quantized  neural  networks  of Fleisher  are  closely related  to this model. 
Analog Neural Networks of Limited Precision I 
703
A Systematic Study of the Input/Output Properties of a 2 Compartment Model Neuron With Active Membranes
Authors: Paul C. Rhodes
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  input/output  properties  of a  2  compartment  model  neuron  are  systematically  explored.  Taken from  the work of MacGregor (MacGregor,  1987), the model neuron  compartments contain several active conductances, including a potassium conductance in  the  dendritic  compartment driven  by  the  accumulation  of  intradendritic  calcium.  Dynamics of the conductances and potentials are governed by a set of coupled first order  differential equations which are  integrated numerically.  There are a set of 17 internal  parameters  to  this  model,  specificying  conductance rate  constants,  time  constants,  thresholds, etc. 
To study parameter sensitivity,  a set of trials were run in which the input driving the  neuron is kept fixed while each internal parameter is varied with all others left fixed. 
To study the input/output relation, the input to the dendrite (a square wave) was varied  (in frequency and magnitude) while all internal parameters of the system were left flXed,  and the resulting output firing rate and bursting rate was counted. 
The  input/output  relation of the  model  neuron  studied  turns  out  to  be  much  more  sensitive  to  modulation  of certain  dendritic  potassium  current  parameters  than  to  plasticity  of synapse  efficacy per se  (the  amount  of current  influx  due  to  synapse  activation).  This would in turn suggest, as has been recently observed experimentally,  that the potassium current may be as or more important a focus of neural plasticity than  synaptic efficacy.
Neural Implementation of Motivated Behavior: Feeding in an Artificial Insect
Authors: Hillel J. Chiel Randall D. Beer
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Most  complex  behaviors  appear  to be governed  by  internal  moti(cid:173) vational  states or  drives  that  modify  an  animal's  responses  to  its  environment.  It is  therefore of considerable  interest to understand  the  neural basis of these  motivational states.  Drawing upon work  on  the  neural  basis  of feeding  in  the  marine  mollusc  Aplysia,  we  have  developed  a  heterogeneous  artificial  neural  network  for  con(cid:173) trolling the feeding behavior of a simulated insect.  We demonstrate  that feeding in this artificial insect shares many characteristics with  the motivated behavior of natural animals.
A Neural Network to Detect Homologies in Proteins
Authors: Samy Bengio, Yannick Pouliot, Patrick Agin Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In order to detect the presence and location of immunoglobu(cid:173) lin (Ig) domains from amino acid sequences we built a system  based on a neural network with one hidden layer trained with  back propagation. The program was designed to efficiently  identify proteins exhibiting such domains, characterized by a  few localized conserved regions and a low overall homology.  When the National Biomedical Research Foundation (NBRF)  NEW protein sequence database was scanned to evaluate the  program's performance, we obtained very low rates of false  negatives coupled with a moderate rate of false positives.
Practical Characteristics of Neural Network and Conventional Pattern Classifiers on Artificial and Speech Problems
Authors: Richard P Lippmann Yuchun Lee
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Eight  neural  net  and  conventional  pattern  classifiers  (Bayesian(cid:173) unimodal Gaussian, k-nearest neighbor, standard back-propagation,  adaptive-stepsize back-propagation, hypersphere, feature-map, learn(cid:173) ing vector  quantizer,  and  binary  decision  tree)  were  implemented  on  a  serial  computer  and  compared  using  two  speech  recognition  and two artificial tasks.  Error rates were statistically equivalent on  almost  all  tasks,  but classifiers  differed  by orders  of magnitude in  memory  requirements,  training time,  classification  time,  and ease  of adaptivity.  Nearest-neighbor  classifiers  trained  rapidly  but  re(cid:173) quired  the most memory.  Tree classifiers  provided  rapid classifica(cid:173) tion but  were  complex to adapt.  Back-propagation classifiers  typ(cid:173) ically  required  long  training  times  and  had intermediate  memory  requirements.  These results suggest that classifier selection should  often  depend  more  heavily  on  practical considerations  concerning  memory  and  computation  resources,  and  restrictions  on  training  and classification times than on error rate. 
-This  work  was  sponsored by  the  Department of the  Air  Force  and  the  Air  Force  Office  of 
Scientific Research. 
Practical Characteristics of Neural Network
Contour-Map Encoding of Shape for Early Vision
Authors: Pentti Kanerva
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Contour  maps  provide  a  general  method  for  recognizing  two-dimensional  shapes.  All  but  blank  images  give  rise  to  such  maps,  and  people  are  good  at  recognizing  objects  and  shapes  from  them.  The  maps  are  encoded  easily  in  long  feature  vectors  that  are  suitable  for  recognition  by  an  associative  memory.  These  properties  of  contour  maps  suggest  a  role  for  them  in  early  visual  perception.  The  prevalence  of  direction-sensitive  neurons  in  the  visual  cortex  of  mammals  supports  this  view.
Maximum Likelihood Competitive Learning
Authors: Steven J. Nowlan
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: One popular class of unsupervised algorithms are competitive algo(cid:173) rithms.  In the traditional view of competition, only one competitor,  the  winner,  adapts for  any given  case.  I  propose  to view  compet(cid:173) itive adaptation as  attempting to fit  a  blend of simple probability  generators  (such  as  gaussians)  to a  set  of data-points.  The maxi(cid:173) mum likelihood fit  of a model of this type suggests a  "softer"  form  of competition,  in  which  all  competitors  adapt  in  proportion  to  the relative probability that the input came from each  competitor.  I  investigate one  application of the soft  competitive model,  place(cid:173) ment of radial basis function centers for function interpolation, and  show  that  the  soft  model  can  give  better  performance  with  little  additional computational cost.
Model Based Image Compression and Adaptive Data Representation by Interacting Filter Banks
Authors: Mitsuo Kawato, Toshio Inui, Sei Miyake Toshiaki Okamoto
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: introduced. Based on 
To achieve high-rate image data compression  while maintainig a high quality reconstructed  image, a good image model and an efficient  way to represent the specific data of each  image must be  the  physiological knowledge of multi - channel  characteristics and  interactions  between them in the human visual system,  a mathematically coherent parallel architecture  for image data compression which utilizes the  Markov  Image model and  interactions between a vast number of filter  banks, is proposed.
Using a Translation-Invariant Neural Network to Diagnose Heart Arrhythmia
Authors: Susan Ciarrocca Lee
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Distinctive electrocardiogram (EeG) patterns are created when the heart  is beating normally and when a dangerous arrhythmia is present. Some  devices which monitor the EeG and react to arrhythmias parameterize  the ECG signal and make a diagnosis based on the parameters. The  author discusses the use of a neural network to classify the EeG signals  directly. without parameterization. The input to such a network must  be translation-invariant. since the distinctive features of the EeG may  appear anywhere in an arbritrarily-chosen EeG segment. The input  must also be insensitive to the episode-to-episode and patient-to-patient  variability in the rhythm pattern.
Non-Boltzmann Dynamics in Networks of Spiking Neurons
Authors: William Bialek Michael C. Crair
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We study networks  of spiking neurons  in which spikes  are fired  as  a  Poisson process.  The state of a  cell is  determined  by the  instan(cid:173) taneous  firing  rate,  and  in the  limit of high firing  rates our model  reduces  to  that  studied  by  Hopfield.  We  find  that  the  inclusion  of spiking  results  in several new  features,  such  as  a  noise-induced  asymmetry between "on" and "off" states of the cells and probabil(cid:173) ity currents which destroy the usual description of network dynam(cid:173) ics  in  terms  of energy  surfaces.  Taking account  of spikes  also  al(cid:173) lows us to calibrate network parameters such as  "synaptic weights"  against  experiments  on  real synapses.  Realistic  forms  of the  post  synaptic  response  alters  the  network dynamics,  which  suggests  a  novel dynamical learning mechanism.
The "Moving Targets" Training Algorithm
Authors: Richard Rohwer
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  simple  method  for  training  the  dynamical  behavior  of  a  neu(cid:173) ral  network  is  derived.  It  is  applicable  to  any  training  problem  in  discrete-time networks with  arbitrary feedback.  The algorithm  resembles back-propagation in  that an error function  is  minimized  using a  gradient-based method,  but the optimization is carried out  in the hidden part of state space either instead of,  or in addition to  weight space.  Computational results are presented for some simple  dynamical  training  problems,  one  of which  requires response  to  a  signal  100 time steps in the past.
Performance Comparisons Between Backpropagation Networks and Classification Trees on Three Real-World Applications
Authors: Ronald A. Cole, Jerome T. Connor, Mohamed A. El-Sharkawi, Robert J. Marks II, Yeshwant K. Muthusamy, Etienne Barnard Les E. Atlas
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Multi-layer  perceptrons  and  trained  classification  trees  are  two  very  different  techniques  which  have  recently  become  popular.  Given  enough  data  and  time,  both  methods  are  capable  of performing  arbi(cid:173) trary  non-linear  classification.  We  first  consider  the  important  differences  between  multi-layer  perceptrons  and  classification  trees  and  conclude  that  there  is  not enough  theoretical  basis  for  the  clear(cid:173) cut  superiority  of one  technique  over  the  other.  For  this  reason,  we  performed  a  number  of empirical  tests  on  three  real-world  problems  in  power  system  load  forecasting,  power  system  security  prediction,  and  speaker-independent  vowel  identification.  In  all  cases,  even  for  piecewise-linear  trees,  the  multi-layer  perceptron  performed  as  well  as or better than  the trained classification  trees. 
Performance Comparisons 
623