mtybilly/Instruct2DS huggingface dataset:
Some Estimates of Necessary Number of Connections and Hidden Units for Feed-Forward Networks
Authors: Adam Kowalczyk
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The feed-forward networks with fixed hidden units (FllU-networks)  are compared against the category of remaining feed-forward net(cid:173) works with variable hidden units (VHU-networks). Two broad  classes of tasks on a finite domain X C R n are considered: ap(cid:173) proximation of every function from an open subset of functions on  X and representation of every dichotomy of X. For the first task  it is found that both network categories require the same minimal  number of synaptic weights. For the second task and X in gen(cid:173) eral position it is shown that VHU-networks with threshold logic  hidden units can have approximately lin times fewer hidden units  than any FHU-network must have.
Neural Network Model Selection Using Asymptotic Jackknife Estimator and Cross-Validation Method
Authors: Yong Liu
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Two  theorems  and  a  lemma are  presented  about  the  use  of jackknife es(cid:173) timator and  the  cross-validation  method for  model selection.  Theorem  1  gives  the asymptotic form for  the jackknife estimator.  Combined with the  model selection  criterion,  this asymptotic form  can  be  used  to obtain the  fit  of a  model.  The model selection  criterion we  used  is the negative of the  average predictive likehood, the choice of which is  based on the idea of the  cross-validation  method.  Lemma 1  provides  a  formula  for  further  explo(cid:173) ration of the asymptotics of the model selection criterion.  Theorem 2 gives  an asymptotic form of the model selection criterion for  the regression  case,  when  the  parameters optimization criterion  has a  penalty term.  Theorem  2  also  proves  the  asymptotic equivalence  of Moody's  model selection  cri(cid:173) terion  (Moody,  1992)  and  the  cross-validation method,  when  the distance  measure  between  response  y  and  regression  function  takes  the  form  of a  squared  difference.
Context-Dependent Multiple Distribution Phonetic Modeling with MLPs
Authors: Horacio Franco, Nelson Morgan, David E. Rumelhart, Victor Abrash Michael Cohen
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A  number  of  hybrid  multilayer  perceptron  (MLP)/hidden  Markov  model  (HMM:)  speech  recognition  systems  have  been  developed  in  recent  years  (Morgan  and Bourlard.  1990).  In  this  paper.  we  present  a  new  MLP  architecture  and  training  algorithm  which  allows  the  modeling  of  context-dependent  phonetic  classes  in  a  hybrid  MLP/HMM:  framework.  The  new  training  procedure  smooths  MLPs  trained  at  different  degrees  of context  dependence  in  order  to obtain  a  robust  estimate  of the  cootext-dependent  probabilities.  Tests  with  the  DARPA  Resomce  Management  database  have  shown  substantial  advantages  of  the  context-dependent  MLPs  over  earlier  cootext(cid:173) independent  MLPs.  and  have  shown  substantial  advantages  of  this  hybrid approach over a pure  HMM approach.
On the Use of Evidence in Neural Networks
Authors: David Wolpert
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The Bayesian "evidence" approximation has recently been employed to  determine the noise and weight-penalty terms used in back-propagation.  This paper shows that for neural nets it is far easier to use the exact result  than  it is  to use  the evidence approximation. Moreover,  unlike  the evi(cid:173) dence approximation, the exact result neither has to be re-calculated for  every new data set, nor requires the running of computer code (the exact  result is  closed  form).  In  addition, it turns out that the evidence proce(cid:173) dure's MAP estimate for neural nets is, in toto, approximation error. An(cid:173) other advantage of the exact analysis is that it does not lead one to incor(cid:173) rect intuition, like the claim that using evidence one can "evaluate differ(cid:173) ent  priors  in  light of the  data".  This  paper also  discusses  sufficiency  conditions for the evidence approximation to hold, why it can sometimes  give "reasonable" results, etc. 
1  THE EVIDENCE APPROXIMATION 
It has recently become popular to consider the problem of training neural nets from a Baye(cid:173) sian viewpoint (Buntine and Weigend 1991, MacKay  1992). The usual way of doing this  starts by assuming that there is some underlying target function f from Rn  to R, parameter(cid:173) ized by an N-dimensional weight vector w.  We are provided with a training set L of noise(cid:173) corrupted samples of f.  Our goal is to make a guess for w, basing that guess only on L. Now  assume we have Li.d. additive gaussian noise resulting in P(L I w, ~) oc  exp(-~ X2)), where  X2(w, L) is the usual sum-squared training set error, and ~ reflects the noise level. Assume  further that P(w I a.)  oc exp(-o.W(w)), where W(w) is the sum of the squares ofthe weights.  If the values of a. and ~ are known and fixed, to the values ~ and ~t respectively, then P(w)
Automatic Capacity Tuning of Very Large VC-Dimension Classifiers
Authors: B. Boser, V. Vapnik I. Guyon
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Large VC-dimension classifiers can learn difficult tasks, but are usually  impractical because they generalize well only if they are trained with huge  quantities of data. In this paper we show that even high-order polynomial  classifiers in high dimensional spaces can be trained with a small amount  of training data and yet generalize better than classifiers with a smaller  VC-dimension. This is achieved with a maximum margin algorithm (the  Generalized Portrait). The technique is applicable to a wide variety of  classifiers, including Perceptrons, polynomial classifiers (sigma-pi unit net(cid:173) works) and Radial Basis Functions. The effective number of parameters is  adjusted automatically by the training algorithm to match the complexity  of the problem. It is shown to equal the number of those training patterns  which are closest patterns to the decision boundary (supporting patterns).  Bounds on the generalization error and the speed of convergence of the al(cid:173) gorithm are given. Experimental results on handwritten digit recognition  demonstrate good generalization compared to other algorithms.
Interposing an ontogenetic model between Genetic Algorithms and Neural Networks
Authors: Richard K. Belew
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The relationships between learning, development and evolution in  Nature is taken seriously, to suggest a model of the developmental  process whereby the genotypes manipulated by the Genetic Algo(cid:173) rithm (GA) might be expressed to form phenotypic neural networks  (NNet) that then go on to learn. ONTOL is a grammar for gener(cid:173) ating polynomial NN ets for time-series prediction. Genomes corre(cid:173) spond to an ordered sequence of ONTOL productions and define a  grammar that is expressed to generate a NNet. The NNet's weights  are then modified by learning, and the individual's prediction error  is used to determine GA fitness. A new gene doubling operator  appears critical to the formation of new genetic alternatives in the  preliminary but encouraging results presented.
Bayesian Learning via Stochastic Dynamics
Authors: Radford M. Neal
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  attempt  to  find  a  single  "optimal"  weight  vector  in  conven(cid:173) tional network training can lead to overfitting and poor generaliza(cid:173) tion.  Bayesian methods avoid  this,  without the  need  for  a  valida(cid:173) tion set,  by  averaging the outputs of many  networks with  weights  sampled  from  the  posterior  distribution  given  the  training  data.  This sample can be obtained by simulating a  stochastic dynamical  system that has the posterior as  its stationary distribution. 
1  CONVENTIONAL AND  BAYESIAN  LEARNING 
I view neural networks as probabilistic models, and learning as statistical inference.  Conventional  network  learning  finds  a  single  "optimal"  set  of network  parameter  values,  corresponding  to maximum likelihood or maximum penalized likelihood in(cid:173) ference.  Bayesian  inference  instead  integrates  the  predictions of the  network  over  all  possible  values  of the  network  parameters,  weighting each  parameter set  by  its  posterior probability in light of the training data. 
1.1  NEURAL  NETWORKS  AS  PROBABILISTIC MODELS 
Consider a network taking a vector of real-valued inputs, x, and producing a  vector  of real-valued  outputs,  y,  perhaps  computed  using  hidden  units.  Such  a  network  architecture corresponds  to a function,  I,  with y = I(x, w),  where w  is  a  vector of  connection weights.  If we assume the observed outputs, y, are equal to y  plus Gaus(cid:173) sian noise  of standard deviation  (j,  the network  defines  the conditional probability
Biologically Plausible Local Learning Rules for the Adaptation of the Vestibulo-Ocular Reflex
Authors: Terrence J. Sejnowski, Stephen G. Lisberger Olivier Coenen
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The vestibulo-ocular reflex (VOR) is a compensatory eye movement that  stabilizes images on the retina during head turns. Its magnitude, or gain,  can be modified by visual experience during head movements. Possible  learning mechanisms for this adaptation have been explored in a model  of the oculomotor system based on anatomical and physiological con(cid:173) straints. The local correlational learning rules in our model reproduce the  adaptation and behavior of the VOR under certain parameter conditions.  From these conditions, predictions for the time course of adaptation at  the learning sites are made.
Combining Neural and Symbolic Learning to Revise Probabilistic Rule Bases
Authors: Raymond J. Mooney J. Jeffrey Mahoney
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: a system for  revising probabilis(cid:173)
This paper describes  RAPTURE  - tic  knowledge  bases  that  combines  neural  and  symbolic  learning  methods.  RAPTURE  uses  a  modified  version  of backpropagation  to refine  the  certainty factors  of a  MYCIN-style  rule base  and  uses  ID3's  information gain  heuristic  to  add  new  rules.  Results on  re(cid:173) fining  two  actual  expert  knowledge  bases  demonstrate  that  this  combined approach performs better  than previous methods.
Transient Signal Detection with Neural Networks: The Search for the Desired Signal
Authors: Abir Zahalka José Carlos Príncipe
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Matched  filtering  has  been  one  of the  most  powerful  techniques  employed for transient detection. Here we will show that a dynamic  neural  network  outperforms  the  conventional  approach.  When  the  artificial neural  network (ANN) is  trained with supervised learning  schemes  there  is  a  need  to  supply  the  desired  signal  for  all  time,  although  we  are  only  interested  in  detecting  the  transient.  In  this  paper  we  also  show  the  effects  on  the  detection  agreement  of  different strategies to construct the desired signal. The extension of  the  Bayes  decision  rule  (011  desired  signal),  optimal  in  static  classification,  performs  worse  than  desired  signals  constructed by  random noise or prediction during the background.
Information Theoretic Analysis of Connection Structure from Spike Trains
Authors: Satoshi Yamada, Michio Nakashima, Kenji Matsumoto Satoru Shiono
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have attempted to use information theoretic quantities for  ana(cid:173) lyzing neuronal connection  structure from spike  trains.  Two point  mu tual information  and  its maximum value,  channel capacity,  be(cid:173) tween  a  pair of neurons  were  found  to  be  useful  for  sensitive  de(cid:173) tection  of crosscorrelation  and  for  estimation  of synaptic strength,  respectively.  Three point mutual information among three neurons  could give their interconnection structure.  Therefore, our informa(cid:173) tion  theoretic analysis  was  shown  to be a  very  powerful technique  for  deducing  neuronal  connection  structure.  Some  concrete exam(cid:173) ples  of its  application  to simulated  spike  trains  are presented.
Comparing the Performance of Connectionist and Statistical Classifiers on an Image Segmentation Problem
Authors: W. E. Blanz Sheri L. Gish
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In  this  study,  we  test  the  suitability  of a  connection(cid:173)
In the development  of an image segmentation system for  real time  image processing applications,  we  apply the classical decision anal(cid:173) ysis  paradigm by  viewing image segmentation as a  pixel classifica.(cid:173) tion  task.  We  use  supervised  training  to derive  a  classifier for  our  system  from  a  set  of  examples  of a  particular  pixel  classification  problem.  ist  method  against  two  statistical  methods,  Gaussian  maximum  likelihood  classifier  and first,  second,  and third degree  polynomial  classifiers,  for  the  solution  of a  "real  world"  image  segmentation  problem  taken  from  combustion  research.  Classifiers  are  derived  using  all  three  methods,  and  the  performance  of all  of the  classi(cid:173) fiers  on  the  training  data set  as  well  as  on  3  separate  entire  test  images  is  measured.
Analog Circuits for Constrained Optimization
Authors: John C. Platt
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: This paper  explores  whether  analog  circuitry  can  adequately  per(cid:173) form  constrained  optimization.  Constrained optimization  circuits  are  designed  using  the  differential  multiplier  method.  These  cir(cid:173) cuits fulfill  time-varying constraints correctly.  Example circuits in(cid:173) clude  a  quadratic programming circuit and a  constrained flip-flop.
Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters
Authors: John S. Bridle
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: One  of the  attractions  of neural  network  approaches  to  pattern  recognition  is  the  use  of a  discrimination-based  training  method.  We  show  that once  we  have  modified  the  output  layer  of a  multi(cid:173) layer perceptron to provide mathematically correct  probability dis(cid:173) tributions,  and  replaced  the  usual  squared  error  criterion  with  a  probability-based score,  the  result  is equivalent  to  Maximum  Mu(cid:173) tual  Information training,  which  has  been  used  successfully  to im(cid:173) prove  the  performance  of hidden  Markov models for  speech  recog(cid:173) nition.  If the network is specially constructed to perform the recog(cid:173) nition computations of a  given kind of stochastic model based clas(cid:173) sifier  then we  obtain a  method for  discrimination-based training of  the  parameters  of the  models.  Examples  include  an  HMM-based  word discriminator,  which  we  call an 'Alphanet'.
Can Simple Cells Learn Curves? A Hebbian Model in a Structured Environment
Authors: Daniel M. Kammen William R. Softky
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: In the mammalian visual cortex,  orientation-selective 'simple cells'  which detect  straight lines  may be adapted  to detect  curved  lines  instead.  We  test  a  biologically  plausible,  Hebbian,  single-neuron  model,  which learns oriented receptive fields  upon exposure  to un(cid:173) structured (noise) input and maintains orientation selectivity upon  exposure  to  edges  or  bars  of all  orientations  and  positions.  This  model  can  also  learn  arc-shaped  receptive  fields  upon  exposure  to an environment of only  circular  rings.  Thus,  new  experiments  which try to induce an abnormal (curved) receptive field  may pro(cid:173) vide insight into the plasticity of simple cells.  The model suggests  that exposing  cells  to only  a  single  spatial  frequency  may induce  more  striking spatial frequency  and  orientation  dependent effects  than heretofore observed.
Development and Regeneration of Eye-Brain Maps: A Computational Model
Authors: A. E. Friedman Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We outline a computational model  of the development and regenera(cid:173) tion of specific eye-brain circuits. The model comprises a self-organiz(cid:173) ing map-forming network which uses local Hebb rules. constrained by  molecular markers.  Various  simulations of the development of eye(cid:173) brain maps in fish and frogs are described.
Predicting Weather Using a Genetic Memory: A Combination of Kanerva's Sparse Distributed Memory with Holland's Genetic Algorithms
Authors: David Rogers
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Kanerva's  sparse distributed  memory  (SDM)  is  an  associative-memo(cid:173) ry  model  based  on  the  mathematical  properties  of  high-dimensional  binary address  spaces.  Holland's genetic  algorithms are  a  search  tech(cid:173) nique  for  high-dimensional  spaces  inspired  by  evolutionary  processes  of DNA.  "Genetic  Memory"  is  a  hybrid  of the  above  two  systems,  in  which  the  memory  uses  a  genetic  algorithm  to  dynamically  recon(cid:173) figure  its  physical  storage  locations  to  reflect  correlations  between  the  stored  addresses  and  data.  For  example,  when  presented  with  raw  weather station  data,  the  Genetic  Memory  discovers  specific  fea(cid:173) tures  in  the  weather  data  which  correlate  well  with  upcoming  rain,  and  reconfigures  the  memory  to  utilize  this  information  effectively.  This  architecture  is  designed  to  maximize  the  ability  of the  system  to scale-up to handle real-world problems.
Neural Network Analysis of Distributed Representations of Dynamical Sensory-Motor Transformations in the Leech
Authors: Yan Fang, Terrence J. Sejnowski Shawn R. Lockery
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Interneurons in leech ganglia receive multiple sensory inputs and make  synaptic contacts with many motor neurons. These "hidden" units  coordinate several different behaviors. We used physiological and  anatomical constraints to construct a model of the local bending reflex.  Dynamical networks were trained on experimentally derived input-output  patterns using recurrent back-propagation. Units in the model were  modified to include electrical synapses and multiple synaptic time  constants. The properties of the hidden units that emerged in the  simulations matched those in the leech. The model and data support  distributed rather than localist representations in the local bending reflex.  These results also explain counterintuitive aspects of the local bending  circuitry.
A Computer Modeling Approach to Understanding the Inferior Olive and Its Relationships to the Cerebellar Cortex in Rats
Authors: James M. Bower Maurice Lee
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: This paper presents the results of a simulation of the spatial relationship  between the inferior olivary nucleus and  folium  crus IIA of the lateral  hemisphere  of  the  rat  cerebellum.  The  principal  objective  of  this  modeling effort was to resolve an apparent conflict between a proposed  zonal organization of olivary projections to cerebellar cortex suggested  by  anatomical  tract-tracing  experiments  (Brodal  &  Kawamura  1980;  Campbell & Armstrong  1983) and a more patchy organization apparent  with physiological mapping (Robertson  1987).  The results suggest that  several unique features  of the olivocerebellar circuit may contribute to  the appearance of zonal organization using anatomical  techniques, but  that  the  detailed  patterns  of  patchy  tactile  projections  seen  with  physiological  techniques  are  a  more  accurate  representation  of  the  afferent organization of this region of cortex.
Neural Network Weight Matrix Synthesis Using Optimal Control Techniques
Authors: Amir Dembo, Thomas Kailath O. Farotimi
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: T.  Kailath 
Given a set  of input-output  training samples,  we  describe  a  proce(cid:173) dure  for  determining  the  time sequence  of weights  for  a  dynamic  neural  network  to  model  an  arbitrary  input-output  process.  We  formulate  the  input-output  mapping  problem  as  an  optimal con(cid:173) trol  problem,  defining  a  performance  index  to  be  minimized  as  a  function  of time-varying  weights.  We  solve  the  resulting  nonlin(cid:173) ear two-point-boundary-value problem, and this yields the  training  rule.  For the performance index chosen,  this rule  turns out  to be  a  continuous time generalization of the outer product rule earlier sug(cid:173) gested heuristically by Hopfield for designing associative memories.  Learning curves for  the new  technique are  presented.
Collective Oscillations in the Visual Cortex
Authors: Christof Koch, Philip J. Holmes Daniel M. Kammen
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  firing  patterns  of populations  of cells  in  the  cat  visual  cor(cid:173) tex can  exhibit  oscillatory  responses  in  the  range  of 35  - 85  Hz.  Furthermore,  groups  of neurons  many  mm's  apart  can  be  highly  synchronized  as  long  as  the  cells  have  similar  orientation  tuning.  We investigate two basic network architectures that incorporate ei(cid:173) ther nearest-neighbor or global feedback interactions and conclude  that non-local feedback plays a  fundamental role in the initial syn(cid:173) chronization and dynamic stability of the oscillations.
Neural Networks: The Early Days
Authors: Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A short account is  given of various  investigations of neural  network  properties,  beginning  with  the  classic  work of McCulloch  & Pitts.  Early work on neurodynamics and statistical mechanics, analogies with  magnetic materials, fault tolerance via parallel distributed processing,  memory, learning,  and pattern recognition,  is described.
Operational Fault Tolerance of CMAC Networks
Authors: Franklin J. Rudolph, Adam J. Nucci Michael J. Carter
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The performance sensitivity of Albus' CMAC network was studied for  the scenario in which faults are introduced into the adjustable weights  after training has been accomplished.  It was found that fault sensitivity  was reduced with increased generalization when "loss of weight" faults  were considered,  but sensitivity  was increased for  "saturated weight"  faults.
Effects of Firing Synchrony on Signal Propagation in Layered Networks
Authors: Eberhard E. Fetz, R. D. Puff G. T. Kenyon
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Spiking  neurons  which  integrate  to  threshold  and  fire  were  used  to  study  the  transmission  of  frequency  modulated  (FM)  signals  through  layered networks.  Firing correlations  between cells  in  the  input  layer  were  found  to  modulate  the  transmission  of  FM  sig(cid:173) nals  under  certain dynamical  conditions.  A  tonic  level  of activity  was  maintained  by  providing  each  cell  with  a  source  of Poisson(cid:173) distributed  synaptic  input.  When  the  average  membrane  depo(cid:173) larization  produced  by  the  synaptic  input  was  sufficiently  below  threshold,  the  firing  correlations  between  cells  in  the  input  layer  could greatly amplify the signal present in subsequent layers.  When  the  depolarization was sufficiently close  to threshold,  however,  the  firing  synchrony  between cells  in the  initial layers could  no  longer  effect  the propagation of FM  signals.  In this latter case,  integrate(cid:173) and-fire  neurons  could  be  effectively  modeled  by  simpler  analog  elements governed by a  linear input-output relation.
Acoustic-Imaging Computations by Echolocating Bats: Unification of Diversely-Represented Stimulus Features into Whole Images
Authors: James A. Simmons
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  echolocating  bat,  Eptesicus fuscus,  perceives  the  distance  to  sonar  targets  from  the  delay  of echoes  and  the  shape  of targets  from  the  spectrum  of echoes.  However,  shape  is  perceived  in  terms  of the  target's  range  proftle.  The  time  separation  of echo  components from  parts of the target located at  different  distances  is  reconstructed  from  the  echo  spectrum  and  added  to  the  estimate  of absolute  delay  already  derived  from  the  arrival-time  of  echoes.  The  bat  thus  perceives  the  distance  to  targets  and  range  depth  within  dimension,  which  is  computed.  The  image  corresponds  to  the  crosscorrelation  function  of  echoes.  Fusion  of  physiologically  distinct  time- and  frequency-domain  representations  into  a  fmal,  common  time-domain  image  illustrates  the  binding  of  within(cid:173) modality  features  into  a  unified,  whole  image.  To  support  the  structure  of  images  along  the  dimension  of  range,  bats  can  perceive echo  delay  with a hyperacuity of 10  nanoseconds. 
the  same  psychological
Time Dependent Adaptive Neural Networks
Authors: Fernando J. Pineda
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  comparison  of algorithms  that minimize error functions  to  train  the  trajectories of recurrent networks, reveals how complexity is traded off for  causality.  These  algorithms  are  also  related  to  time-independent  fonnalisms.  It is  suggested  that  causal  and  scalable  algorithms  are  possible  when  the  activation  dynamics  of  adaptive  neurons  is  fast  compared  to  the  behavior  to  be  learned.  Standard  continuous-time  recurrent backpropagation is used in an example.
Neural Network Visualization
Authors: Gerald Tesauro Jakub Wejchert
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  have developed graphics  to visualize  static and dynamic infor(cid:173) mation in layered neural network learning systems.  Emphasis  was  placed  on  creating new  visuals  that  make  use  of spatial arrange(cid:173) ments,  size  information,  animation  and  color.  We  applied  these  tools  to  the  study  of back-propagation learning of simple  Boolean  predicates,  and  have  obtained  new  insights  into  the  dynamics  of  the learning process.
Discovering the Structure of a Reactive Environment by Exploration
Authors: Jonathan Bachrach Michael Mozer
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Consider a robot wandering around an unfamiliar environment. performing ac(cid:173) tions and sensing the resulting environmental states.  The robot's task is to con(cid:173) struct an internal model of its environment. a model that will allow it to predict  the consequences of its actions  and to determine what sequences of actions  to  take  to  reach  particular  goal  states.  Rivest  and  Schapire  (1987&,  1987b;  Schapire.  1988) have studied this problem and have designed a symbolic algo(cid:173) rithm  to  strategically explore  and infer the  structure of "finite state"  environ(cid:173) ments.  The heart of this algorithm is a clever representation of the environment  called an update graph.  We have developed a connectionist implementation of  the update  graph using  a highly-specialized network  architecture.  With  back  propagation learning and a trivial exploration strategy - tions - gorithm on simple problems.  The network has  the  additional  strength  that it  can accommodate stochastic environments.  Perhaps the  greatest virtue of the  connectionist  approach  is  that  it suggests  generalizations of the update graph  representation that do not arise from a traditional, symbolic perspective. 
choosing random ac(cid:173) the connectionist network can outperfonn the Rivest and Schapire al(cid:173)
A Self-organizing Associative Memory System for Control Applications
Authors: Michael Hormel
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The  CHAC  storage  scheme  has  been  used  as  a  basis  for  a  software  implementation  of  an  associative  .emory  system  AHS,  which  itself  is  a  major  part  of  the  learning  control  loop  LERNAS.  A  major  this  CHAC-concept  is  that  the  disadvantage  of  degree  of  local  generalization  (area  of  interpo(cid:173) lation)  is  fixed.  This  paper  deals  with  an  algo(cid:173) rithm  for  self-organizing  variable  generaliza(cid:173) tion  for  the  AKS,  based  on  ideas  of  T.  Kohonen.
Complexity of Finite Precision Neural Network Classifier
Authors: Kai-Yeung Siu, Thomas Kailath Amir Dembo
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A rigorous analysis on the finite  precision computational <)Spects  of  neural  network as  a  pattern  classifier  via a  probabilistic  approach  is  presented.  Even though there exist negative results on  the capa(cid:173) bility of perceptron,  we  show  the following  positive results:  Given  n  pattern vectors each represented by en bits where  e > 1,  that are  uniformly  distributed,  with  high  probability  the  perceptron  can  perform  all  possible  binary  classifications  of the  patterns.  More(cid:173) over, the resulting neural network requires a  vanishingly small pro(cid:173) portion O(log n/n) of the memory that would be required for  com(cid:173) plete  storage  of the  patterns.  Further,  the  perceptron  algorithm  takes  O(n2)  arithmetic operations  with  high  probability,  whereas  other  methods  such  as  linear  programming  takes  O(n3 .5 )  in  the  worst  case.  We also  indicate some mathematical connections  with  VLSI  circuit  testing and the theory  of random matrices.
Dimensionality Reduction and Prior Knowledge in E-Set Recognition
Authors: Geoffrey E. Hinton Kevin J. Lang
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: It is  well known  that  when an  automatic  learning algorithm  is  applied  to a  fixed  corpus  of data,  the size of the corpus  places  an  upper bound  on  the  number  of degrees  of freedom  that  the  model  can  contain  if  it  is  to generalize  well.  Because  the  amount  of hardware  in  a  neural  network  typically  increases  with  the  dimensionality  of  its  inputs,  it  can be challenging to build a high-performance network for classifying  large input patterns.  In this paper, several techniques for addressing this  problem  are  discussed  in  the context of an  isolated  word  recognition  task.
Associative Memory in a Simple Model of Oscillating Cortex
Authors: Bill Baird
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  generic  model  of oscillating  cortex,  which  assumes  "minimal"  coupling justified by known anatomy, is shown to function as an as(cid:173) sociative memory, using previously developed  theory.  The network  has  explicit  excitatory  neurons  with  local  inhibitory  interneuron  feedback  that forms  a  set of nonlinear oscillators coupled  only by  long range excitatofy connections.  Using a local Hebb-like learning  rule for  primary and higher order synapses  at the ends of the long  range  connections,  the  system  learns  to store  the  kinds  of oscil(cid:173) lation amplitude patterns observed  in olfactory  and  visual  cortex.  This  rule  is  derived  from  a  more  general  "projection  algorithm"  for recurrent  analog networks, that analytically guarantees content  addressable  memory  storage  of continuous  periodic  sequences  - capacity:  N /2  Fourier  components for  an  N  node  network  - "spurious"  attractors. 
no
Adjoint Operator Algorithms for Faster Learning in Dynamical Neural Networks
Authors: Nikzad Benny Toomarian, Sandeep Gulati Jacob Barhen
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: A  methodology for  faster supervised learning in  dynamical nonlin(cid:173) ear neural networks is  presented.  It exploits the concept  of adjoint  operntors  to  enable  computation  of changes  in  the  network's  re(cid:173) sponse due to perturbations in all system parameters,  using the so(cid:173) lution of a single set of appropriately constructed  linear equations.  The  lower  bound  on  speedup  per  learning  iteration  over  conven(cid:173) tional methods for  calculating the neuromorphic energy gradient is  O(N2),  where  N  is  the number  of neurons  in  the network.
Computer Simulation of Oscillatory Behavior in Cerebral Cortical Networks
Authors: James M. Bower Matthew A. Wilson
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: It has been known for  many years that specific regions of the work(cid:173) ing cerebral cortex display periodic variations in correlated cellular  activity.  While the olfactory system has  been the focus  of much of  this  work,  similar  behavior  has  recently  been observed  in  primary  visual  cortex.  We  have  developed  models  of  both  the  olfactory  and  visual  cortex which  replicate  the observed  oscillatory  proper(cid:173) ties of these  networks.  Using  these  models  we  have  examined  the  dependence of oscillatory behavior on single cell properties and net(cid:173) work  architectures.  We  discuss the idea that  the oscillatory  events  recorded from  cerebral cortex may  be intrinsic  to the architecture  of cerebral  cortex  as  a  whole,  and  that  these  rhythmic  patterns  may be important in coordinating neuronal activity during sensory  processmg.
Coupled Markov Random Fields and Mean Field Theory
Authors: Federico Girosi Davi Geiger
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Federico  Girosi  Artificial Intelligence  Laboratory,  MIT  545  Tech.  Sq.  #  788  Cambridge, MA 02139 
In recent years many researchers have investigated the use of Markov  Random Fields (MRFs)  for  computer vision.  They can be  applied  for  example  to  reconstruct  surfaces  from  sparse  and  noisy  depth  data  coming from  the  output  of a  visual  process,  or  to  integrate  early vision  processes  to label  physical  discontinuities.  In  this  pa(cid:173) per  we  show  that  by  applying  mean field  theory  to  those  MRFs  models a  class of neural  networks is  obtained.  Those networks can  speed  up  the  solution  for  the  MRFs  models.  The  method  is  not  restricted  to computer vision.
Pulse-Firing Neural Chips for Hundreds of Neurons
Authors: Lionel Tarassenko, Alan F. Murray, Alister Hamilton, Il Song Han, H. Martin Reekie Michael Brownlow
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  announce  new  CMOS  synapse  circuits  using  only  three  and four  MOSFETsisynapse.  Neural states are asynchronous  pulse  streams,  upon  which  arithmetic  is  performed  directly.  Chips  implementing  over  100  fully  programmable  synapses  are  described  and  projections  to  networks  of  hundreds  of  neurons are made. 
1 OVERVIEW OF PULSE FIRING NEURAL  VLSI  The  inspiration  for  the  use  of  pulse  firing  in  silicon  neural  networks  is  clearly  the  electrical/chemical  pulse  mechanism  in  "real"  biological  neurons.  Asynchronous,  digital  voltage  pulses  are  used  to  signal  states  t Si  )  through  synapse  weights  {  Tij  }  to  emulate  neural  dynamics.  Neurons  fire  voltage  pulses  of a frequency  determined  by  their  level  of activity  but of a  constant  magnitude  (usually  5  Volts)  [Murray,1989a].  As  indicated  in  Fig.  1,  to  synapses  perform  arithmetic  directly  on  these  asynchronous  pulses,  increment  or  decrement  the  receiving  neuron's  activity.  The  activity  of  a  receiving  neuron  i,  Xi  is  altered  at  a  frequency  controlled  by  the  sending  neuron  j,  with  state  Sj  by  an  amount  determined  by  the  synapse  weight  (here,  T ij ). 
1 On secondment from  the Korean Telecommunications Authority 
786 
Brownlow, Tarassenko, Murray, Hamilton, Han and Reekie
Rule Representations in a Connectionist Chunker
Authors: Gillette Elvgreen III David S. Touretzky
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  present  two  connectionist  architectures  for  chunking  of symbolic  rewrite rules.  One uses backpropagation learning, the other competitive  learning.  Although  they  were  developed  for  chunking  the  same  sorts  of rules,  the  two  differ  in  their  representational  abilities  and  learning  behaviors.
A Neural Network for Real-Time Signal Processing
Authors: Donald B. Malkoff
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: This paper describes a  neural network algorithm that (1)  performs  temporal pattern matching in real-time, (2)  is trained on-line, with  a single pass,  (3)  requires only a single template for training of each  representative  class,  (4)  is  continuously  adaptable  to  changes  in  background noise, (5) deals with transient signals having low signal(cid:173) to-noise ratios,  (6) works in the presence of non-Gaussian noise,  (7)  makes use of context dependencies and (8) outputs Bayesian proba(cid:173) bility estimates.  The algorithm has been adapted to the problem of  passive sonar signal detection and classification.  It runs on a  Con(cid:173) nection  Machine  and  correctly  classifies,  within  500  ms  of onset,  signals embedded in noise and subject to considerable uncertainty.
Speaker Independent Speech Recognition with Neural Networks and Speech Knowledge
Authors: Renato de Mori, Régis Cardin Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  attempt to  combine neural  networks with  knowledge  from  speech  science to build  a speaker independent speech recogni(cid:173) tion  system.  This  knowledge  is  utilized  in  designing  the  preprocessing,  input coding, output coding,  output supervision  and  architectural  constraints.  To  handle  the  temporal  aspect  of speech we  combine  delays,  copies  of activations  of hidden  and  output  units  at  the  input  level,  and  Back-Propagation  for  Sequences  (BPS),  a  learning algorithm for networks with  local  self-loops.  This  strategy  is  demonstrated  in  several  experi(cid:173) ments,  in  particular  a  nasal  discrimination  task  for  which  the  application  of  a  speech  theory  hypothesis  dramatically  im(cid:173) proved generalization.
Analytic Solutions to the Formation of Feature-Analysing Cells of a Three-Layer Feedforward Visual Information Processing Neural Net
Authors: Dun-Sung Tang
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Analytic  solutions  to  the  information-theoretic  evolution  equa(cid:173) tion of the connection strength of a three-layer feedforward neural  net  for  visual information processing  are  presented.  The results  are  (1)  the  receptive  fields  of the  feature-analysing  cells  corre(cid:173) spond to the eigenvector of the maximum eigenvalue of the Fred(cid:173) holm integral equation of the first  kind derived from the evolution  equation  of  the  connection  strength;  (2)  a  symmetry-breaking  mechanism  (parity-violation)  has  been  identified  to  be  respon(cid:173) sible  for  the  changes  of  the  morphology  of  the  receptive  field;  (3)  the conditions for  the formation of different  morphologies are  explicitly identified.
An Analog VLSI Model of Adaptation in the Vestibulo-Ocular Reflex
Authors: Carver Mead Stephen P. DeWeerth
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: The vestibulo-ocular reflex (VOR)  is  the primary mechanism that  controls the compensatory eye movements that stabilize retinal im(cid:173) ages during rapid head motion.  The primary pathways of this sys(cid:173) tem are feed-forward,  with inputs from  the semicircular canals and  outputs  to  the  oculomotor system.  Since  visual  feedback  is  not  used  directly  in  the  VOR  computation,  the  system  must  exploit  motor learning to perform correctly.  Lisberger(1988) has proposed  a  model for  adapting the  VOR gain  using  image-slip  information  from  the  retina.  We  have  designed  and  tested  analog  very  large(cid:173) scale integrated (VLSI) circuitry that implements a  simplified ver(cid:173) sion of Lisberger's adaptive VOR model.
A Reconfigurable Analog VLSI Neural Network Chip
Authors: Yannis P. Tsividis, Hans Peter Graf Srinagesh Satyanarayana
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: 1024 distributed-neuron synapses have been integrated in an active  area of 6.1mm  x  3.3mm using  a 0.9p.m,  double-metal,  single-poly,  n-well CMOS  technology.  The distributed-neuron synapses are ar(cid:173) ranged in blocks of 16, which we call '4  x  4 tiles'.  Switch matrices  are  interleaved between each of these  tiles  to provide programma(cid:173) bility of interconnections.  With a small area overhead (15  %),  the  1024  units  of the  network can  be rearranged  in various  configura(cid:173) tions.  Some of the possible configurations are, a  12-32-12  network,  a  16-12-12-16  network, two 12-32  networks etc.  (the numbers sep(cid:173) arated  by dashes indicate the number of units per layer, including  the  input  layer).  Weights  are  stored  in  analog  form  on  MaS  ca(cid:173) pacitors.  The synaptic weights  are  usable to a  resolution  of 1 % of  their full  scale  value.  The limitation arises  due  to charge injection  from  the access  switch and  charge leakage.  Other  parameters like  gain and shape of nonlinearity are also programmable.
Handwritten Digit Recognition with a Back-Propagation Network
Authors: Bernhard E. Boser, John S. Denker, Donnie Henderson, R. E. Howard, Wayne E. Hubbard, Lawrence D. Jackel Yann LeCun
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We present an application of back-propagation networks to hand(cid:173) written digit recognition. Minimal preprocessing of the data was  required, but architecture of the network was highly constrained  and specifically designed for the task. The input of the network  consists of normalized images of isolated digits. The method has  1 % error rate and about a 9% reject rate on zipcode digits provided  by the U.S. Postal Service.
Computational Efficiency: A Common Organizing Principle for Parallel Computer Maps and Brain Maps?
Authors: James M. Bower Mark E. Nelson
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: It is  well-known  that  neural  responses  in  particular  brain  regions  are  spatially  organized,  but  no  general  principles  have  been  de(cid:173) veloped  that  relate  the structure of a  brain  map  to  the nature of  the associated computation.  On parallel computers, maps of a sort  quite similar to brain maps arise when a computation is distributed  across  multiple  processors.  In  this  paper  we  will  discuss  the rela(cid:173) tionship  between  maps and computations on these  computers and  suggest how similar considerations might also apply to maps in the  brain.
A Cost Function for Internal Representations
Authors: C. I. Thorbergsson, John A. Hertz Anders Krogh
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: We  introduce  a  cost  function  for  learning  in  feed-forward  neural  networks  which  is  an  explicit  function  of the  internal representa(cid:173) tion  in  addition  to  the  weights.  The  learning  problem  can  then  be formulated  as two simple perceptrons and  a  search for  internal  representations.  Back-propagation  is  recovered  as  a  limit.  The  frequency  of successful  solutions  is  better for  this  algorithm than  for  back-propagation when  weights and hidden  units  are  updated  on the same timescale i.e.  once  every learning step.
HMM Speech Recognition with Neural Net Discrimination
Authors: Richard P Lippmann William Y. Huang
Published in: Advances in Neural Information Processing Systems 2  (NIPS 1989) (Main Conference Track), 1989

Abstract: Two approaches were explored which integrate neural net classifiers  with  Hidden  Markov  Model  (HMM)  speech  recognizers.  Both  at(cid:173) tempt to improve speech pattern discrimination while retaining the  temporal processing advantages of HMMs.  One approach used neu(cid:173) ral nets to provide second-stage discrimination following an HMM  recognizer.  On  a  small  vocabulary  task,  Radial  Basis  Function  (RBF)  and  back-propagation  neural  nets  reduced  the  error  rate  substantially (from 7.9% to 4.2% for the RBF classifier).  In a larger  vocabulary task, neural net classifiers did not reduce the error rate.  They, however,  outperformed Gaussian, Gaussian mixture, and k(cid:173) nearest  neighbor  (KNN)  classifiers.  In  another  approach,  neural  nets  functioned  as  low-level  acoustic-phonetic  feature  extractors.  When  classifying  phonemes  based on single  10  msec.  frames,  dis(cid:173) criminant  RBF  neural  net  classifiers outperformed  Gaussian mix(cid:173) ture  classifiers.  Performance,  however,  differed  little when  classi(cid:173) fying  phones  by  accumulating scores  across  all frames  in phonetic  segments using  a single node HMM  recognizer. 
-This  work  was  sponsored  by  the  Department  of  the  Air  Force  and  the  Air  Force  Office  of 
Scientific Research. 
HMM Speech Recognition with Neural Net Discrimination