mtybilly/Instruct2DS huggingface dataset:
Neural Networks Structured for Control Application to Aircraft Landing
Authors: Yves Chauvin, Van Henkle, Richard Golden Charles Schley
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We present a generic neural network architecture capable of con(cid:173) trolling non-linear plants. The network is composed of dynamic.  parallel, linear maps gated by non-linear switches. Using a recur(cid:173) rent form of the back-propagation algorithm, control is achieved  by optimizing the control gains and task-adapted switch parame(cid:173) ters. A mean quadratic cost function computed across a nominal  plant trajectory is minimized along with performance constraint  penalties. The approach is demonstrated for a control task con(cid:173) sisting of landing a commercial aircraft in difficult wind conditions.  We show that the network yields excellent performance while re(cid:173) maining within acceptable damping response constraints.
Closed-Form Inversion of Backpropagation Networks: Theory and Optimization Issues
Authors: Michael L. Rossen
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe a closed-form technique for mapping the output of a trained  backpropagation network int.o input activity space. The mapping is an in(cid:173) verse mapping in the sense that, when the image of the mapping in input  activity space is propagat.ed forward through the normal network dynam(cid:173) ics, it reproduces the output used to generate that image. When more  than one such inverse mappings exist, our inverse ma.pping is special in  that it has no projection onto the nullspace of the activation flow opera(cid:173) tor for the entire network. An important by-product of our calculation,  when more than one invel'se mappings exist, is an orthogonal basis set of  a significant portion of the activation flow operator nullspace. This basis  set can be used to obtain an alternate inverse mapping that is optimized  for a particular rea.l-world application.
Dynamics of Generalization in Linear Perceptrons
Authors: John A. Hertz Anders Krogh
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We study the evolution of the generalization ability of a  simple linear per(cid:173) ceptron with N  inputs which learns to imitate a  "teacher perceptron".  The  system is  trained  on p  =  aN  binary  example  inputs  and  the  generaliza(cid:173) tion  ability measured by testing for  agreement with  the teacher on all 2N  possible  binary input  patterns.  The dynamics may  be solved  analytically  and  exhibits  a  phase  transition  from  imperfect  to  perfect  generalization  at  a  =  1.  Except  at  this  point  the  generalization  ability  approaches  its  asymptotic  value  exponentially,  with critical slowing  down near  the  tran(cid:173) sition;  the  relaxation  time  is  ex  (1  - y'a)-2.  Right  at  the  critical  point,  1  the  approach  to  perfect  generalization  follows  a  power  law  ex  t - '2.  In  the presence of noise,  the generalization ability is  degraded  by  an amount  ex  (va - 1)-1 just above a  =  1.
Translating Locative Prepositions
Authors: Mary Tabasko Paul W. Munro
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A network was trained by back propagation to map locative expressions  of the form "noun-preposition-noun" to a semantic representation, as in  Cosic  and  Munro  (1988).  The  network's  performance  was  analyzed  over  several  simulations  with  training  sets  in  both  English  and  German.  Translation  of prepositions  was  attempted  by  presenting  a  locative expression to a  network trained in one language to generate a  semantic representation; the semantic representation was  then presented  to the network trained in the other language to generate the appropriate  preposition.
An Analog VLSI Chip for Finding Edges from Zero-crossings
Authors: Christof Koch Wyeth Bair
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  have  designed  and  tested  a  one-dimensional  64  pixel,  analog  CMOS  VLSI chip which localizes intensity edges in real-time.  This device exploits  on-chip photoreceptors and the natural filtering properties of resistive net(cid:173) works to implement a  scheme similar  to  and  motivated  by  the Difference  of Gaussians (DOG) operator proposed by Marr and Hildreth (1980).  Our  chip computes the  zero-crossings associated  with  the difference of two ex(cid:173) ponential weighting functions.  If the  derivative  across  this  zero-crossing  is  above  a  threshold,  an edge  is  reported.  Simulations  indicate  that  this  technique will  extend well to two dimensions.
Comparison of three classification techniques: CART, C4.5 and Multi-Layer Perceptrons
Authors: R. A. Pearson A. C. Tsoi
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: In this paper, after some introductory remarks into the classification prob(cid:173) lem as considered in various research communities, and some discussions  concerning some of the reasons for ascertaining the performances of the  three chosen algorithms, viz., CART (Classification and Regression Tree),  C4.5 (one of the more recent versions of a popular induction tree tech(cid:173) nique known as ID3), and a multi-layer perceptron (MLP), it is proposed  to compare the performances of these algorithms under two criteria: classi(cid:173) fication and generalisation. It is found that, in general, the MLP has better  classification and generalisation accuracies compared with the other two  algorithms.
ART2/BP architecture for adaptive estimation of dynamic processes
Authors: Einar Sørheim
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The goal has been to construct a supervised artificial neural network that  learns incrementally an unknown mapping. As a result a network con(cid:173) sisting of a combination of ART2 and backpropagation is proposed and  is called an "ART2/BP" network. The ART2 network is used to build  and focus a supervised backpropagation network. The ART2/BP network  has the advantage of being able to dynamically expand itself in response  to input patterns containing new information. Simulation results show  that the ART2/BP network outperforms a classical maximum likelihood  method for the estimation of a discrete dynamic and nonlinear transfer  function.
Self-organization of Hebbian Synapses in Hippocampal Neurons
Authors: Zachary F. Mainen, Anthony M. Zador, Brenda J. Claiborne Thomas H. Brown
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We are exploring the significance of biological complexity for neuronal  computation.  Here we demonstrate that Hebbian synapses in realistical(cid:173) ly-modeled  hippocampal  pyramidal  cells  may  give  rise  to  two  novel  forms of self -organization in response to structured synaptic input.  First,  on the basis of the electrotonic relationships between synaptic contacts,  a cell may become tuned to a small subset of its input space.  Second, the  same mechanisms may produce  clusters of potentiated synapses across  the space of the dendrites.  The latter type of self-organization may be  functionally  significant in  the presence of nonlinear dendritic  conduc(cid:173) tances.
Connection Topology and Dynamics in Lateral Inhibition Networks
Authors: F. R. Waugh, R. M. Westervelt C.M Marcus
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We show analytically how the stability of two-dimensional lateral  inhibition neural networks depends on the local connection topology.  For various network topologies, we calculate the critical time delay for  the onset of oscillation in continuous-time networks and present  analytic phase diagrams characterizing the dynamics of discrete-time  networks.
A Delay-Line Based Motion Detection Chip
Authors: John Lazzaro, Andrew Moore, Christof Koch Tim Horiuchi
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Inspired by a visual motion detection model for the ra.bbit retina  and by a computational architecture used for early audition in the  barn owl, we have designed a chip that employs a correlation model  to report the one-dimensional field motion of a scene in real time.  Using subthreshold analog VLSI techniques, we have fabricated and  successfully tested a 8000 transistor chip using a standard MOSIS  process.
Applications of Neural Networks in Video Signal Processing
Authors: Clay D. Spence, Ronald Sverdlove John C. Pearson
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Although color TV is an established technology, there are a number of  longstanding problems for which neural networks may be suited. Impulse  noise is such a problem, and a modular neural network approach is pre(cid:173) sented in this paper. The training and analysis was done on conventional  computers, while real-time simulations were performed on a massively par(cid:173) allel computer called the Princeton Engine. The network approach was  compared to a conventional alternative, a median filter. Real-time simula(cid:173) tions and quantitative analysis demonstrated the technical superiority of  the neural system. Ongoing work is investigating the complexity and cost  of implementing this system in hardware. 
1 THE POTENTIAL FOR NEURAL NETWORKS IN 
CONSUMER ELECTRONICS 
Neural networks are most often considered for application in emerging new tech(cid:173) nologies, such as speech recognition, machine vision, and robotics. The fundamental  ideas behind these technologies are still being developed, and it will be some time  before products containing neural networks are manufactured. As a result, research  in these areas will not drive the development of inexpensive neural network hard(cid:173) ware which could serve as a catalyst for the field of neural networks in general. 
In contrast, neural networks are rarely considered for application in mature tech(cid:173) nologies, such as consumer electronics. These technologies are based on established  principles of information processing and communication, and they are used in mil(cid:173) lions of products per year. The embedding of neural networks within such mass-
Grouping Contours by Iterated Pairing Network
Authors: Shimon Ullman Amnon Shashua
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Shimon Ullman 
We describe in this paper a network that performs grouping of image con(cid:173) tours. The input to the net are fragments of image contours, and the  output is the partitioning of the fragments into groups, together with a  saliency measure for each group. The grouping is based on a measure of  overall length and curvature. The network decomposes the overall opti(cid:173) mization problem into independent optimal pairing problems performed  at each node. The resulting computation maps into a uniform locally  connected network of simple computing elements. 
1 The Problenl: Contour Grouping 
A problem that often arises in visual information processing is the linking of con(cid:173) tour fragments into optimal groups. For example, certain subsets of contours spon(cid:173) taneously form perceptual groups, as illustrated in Fig. 1, and are often detected  immediately without scanning the image in a systematic manner. Grouping process  of this type are likely to play an important role in object recognition by segmenting  the image and selecting image structures that are likely to correspond to objects of  interest in the scene. 
'Ve propose that some form of autonomous grouping is performed at an early stage  based on geometrical characteristics, that are independent of the identity of objects  to be selected. The grouping process is governed by the notion of saliency in a way  that priority is given to forming salient groups at the expense of potentially less  salient ones. This general notion can again be illustrated by Fig. 1; it appears that  certain groups spontaneously emerge, while grouping decisions concerning the less  salient parts of the image may remain unresolved. As we shall see, the computation  below exhibits a similar behavior. 
We define a grouping of the image contours as the formation of a set of disjoint
Simulation of the Neocognitron on a CCD Parallel Processing Architecture
Authors: Alice M. Chiang Michael L. Chuang
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The neocognitron  is  a  neural network for  pattern recognition  and feature  extraction.  An  analog  CCD  parallel  processing  architecture  developed  at Lincoln Laboratory is  particularly  well suited to the computational re(cid:173) quirements of shared-weight networks such as the neocognitron, and imple(cid:173) mentation of the neocognitron using the CCD architecture was simulated.  A  modification  to  the  neocognitron  training  procedure,  which  improves  network performance under the limited arithmetic  precision that would be  imposed  by the CCD  architecture,  is  presented.
Order Reduction for Dynamical Systems Describing the Behavior of Complex Neurons
Authors: L. F. Abbott, Eve Marder Thomas B. Kepler
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We have devised a scheme to reduce the complexity of dynamical  systems belonging to a class that includes most biophysically realistic  neural models. The reduction is based on transformations of variables  and perturbation expansions and it preserves a high level of fidelity to  the original system. The techniques are illustrated by reductions of the  Hodgkin-Huxley system and an augmented Hodgkin-Huxley system.
Note on Learning Rate Schedules for Stochastic Optimization
Authors: John E. Moody Christian Darken
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  present  and  compare  learning  rate  schedules  for  stochastic  gradient  descent,  a  general  algorithm  which  includes  LMS,  on-line  backpropaga(cid:173) tion  and  k-means  clustering  as  special  cases.  We  introduce  "search-then(cid:173) converge"  type  schedules  which  outperform  the  classical  constant  and  "running average"  (1ft) schedules both in speed of convergence and quality  of solution. 
Introduction:  Stochastic  Gradient  Descent 
1  tion G(W).  In  the context  of learning systems  typically G(W) = £x E(W, X),  i.e. 
The  optimization  task  is  to  find  a  parameter  vector  W  which  minimizes  a  func(cid:173)
G  is  the  average  of  an  objective  function  over  the  exemplars,  labeled  E  and  X  respectively.  The stochastic gradient  descent  algorithm is  Ll Wet)  = -1](t)V'w E(W(t), X(t)). 
where  t  is  the  "time",  and  X(t)  is  the  most  recent  independently-chosen  random  exemplar.  For comparison, the  deterministic gradient  descent  algorithm is 
Ll Wet)  =  -1](t)V'w£x E(W(t), X).
Remarks on Interpolation and Recognition Using Neural Nets
Authors: Eduardo D. Sontag
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We consider different  types  of single-hidden-Iayer feedforward  nets:  with  or  without  direct  input  to  output  connections,  and  using  either  thresh(cid:173) old  or  sigmoidal activation functions.  The  main results  show  that  direct  connections in  threshold nets  double  the  recognition  but not  the interpo(cid:173) lation power, while using sigmoids  rather than thresholds allows (at least)  doubling  both.  Various results are also given on VC dimension and  other  measures of recognition capabilities.
Rapidly Adapting Artificial Neural Networks for Autonomous Navigation
Authors: Dean Pomerleau
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The ALVINN (Autonomous Land Vehicle In a Neural Network) project addresses  the problem  of training artificial  neural  networks in  real  time to  perform difficult  perception tasks.  ALVINN ,is  a back-propagation network that uses inputs from  a  video camera and an imaging laser rangefinder to drive the CMU Navlab, a modified  Chevy van.  This paper describes training techniques which allow ALVINN to learn  in under 5 minutes to autonomously control the Navlab by watching a human driver's  response  to  new  situations.  Using  these  techniques,  ALVINN  has  been  trained  to  drive  in  a  variety  of circumstances  including  single-lane  paved  and  unpaved  roads,  multilane  lined  and  unlined  roads,  and  obstacle-ridden  on- and  off-road  environments, at speeds of up to 20 miles per hour.
A Recurrent Neural Network for Word Identification from Continuous Phoneme Strings
Authors: Candace A. Kamm Robert B. Allen
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A  neural  network  architecture  was  designed  for  locating  word  boundaries  and  identifying  words  from  phoneme  sequences.  This  architecture  was  tested  in  three  sets  of  studies.  First,  a  highly  redundant  corpus  with  a  restricted  vocabulary was  generated and the network was trained with a limited number of  phonemic variations for the words  in the corpus.  Tests of network performance  on a transfer set yielded a very low error rate.  In a second study, a network was  trained  to  identify  words  from  expert  transcriptions  of speech.  On a  transfer  test,  error  rate  for  correct  simultaneous  identification  of  words  and  word  boundaries was  18%.  The third study used the output of a phoneme classifier as  the input to the word and  word boundary identification network.  The error rate  on a transfer test set was 49% for this task.  Overall, these studies provide a first  step at identifying words in connected discourse with a neural network.
Adjoint-Functions and Temporal Learning Algorithms in Neural Networks
Authors: J. Barhen N. Toomarian
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The development of learning algorithms is generally based upon the min(cid:173) imization of an energy function. It is a fundamental requirement to com(cid:173) pute the gradient of this energy function with respect to the various pa(cid:173) rameters of the neural architecture, e.g., synaptic weights, neural gain,etc.  In principle, this requires solving a system of nonlinear equations for each  parameter of the model, which is computationally very expensive. A new  methodology for neural learning of time-dependent nonlinear mappings is  presented. It exploits the concept of adjoint operators to enable a fast  global computation of the network's response to perturbations in all the  systems parameters. The importance of the time boundary conditions of  the adjoint functions is discussed. An algorithm is presented in which  the adjoint sensitivity equations are solved simultaneously (Le., forward  in time) along with the nonlinear dynamics of the neural networks. This  methodology makes real-time applications and hardware implementation  of temporal learning feasible.
Language Induction by Phase Transition in Dynamical Recognizers
Authors: Jordan B. Pollack
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A higher order recurrent neural network architecture learns to recognize and  generate languages after being  "trained"  on categorized exemplars.  Studying  these  networks  from  the  perspective  of  dynamical  systems  yields  two  interesting  discoveries:  First,  a  longitudinal  examination  of  the  learning  process  illustrates  a  new  form  of mechanical  inference:  Induction  by  phase  transition.  A  small  weight  adjustment  causes  a  "bifurcation"  in  the  limit  behavior of the network. This phase transition corresponds to the onset of the  network's  capacity  for  generalizing  to  arbitrary-length  strings.  Second,  a  study of the  automata resulting  from  the  acquisition  of previously published  languages  indicates  that  while  the architecture  is  NOT  guaranteed  to  find  a  minimal  finite  automata  consistent  with  the  given  exemplars,  which  is  an  NP-Hard  problem,  the  architecture  does  appear capable  of generating  non(cid:173) regular languages by exploiting fractal and chaotic dynamics. I end the paper  with  a  hypothesis  relating  linguistic  generative  capacity  to  the  behavioral  regimes of non-linear dynamical systems.
Chaitin-Kolmogorov Complexity and Generalization in Neural Networks
Authors: Ronald Rosenfeld Barak A. Pearlmutter
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  present  a  unified  framework  for  a  number  of different  ways  of failing  to  generalize  properly.  During  learning,  sources  of random  information  contaminate  the  network,  effectively  augmenting  the  training  data  with  random information. The complexity of the function computed is therefore  increased,  and generalization is degraded.  We analyze replicated networks,  in  which  a  number of identical networks are independently trained on  the  same data and their results averaged.  We conclude that replication almost  always results in a decrease in the expected complexity of the network, and  that  replication  therefore  increases  expected  generalization.  Simulations  confirming the effect  are  also presented. 
1  BROKEN SYMMETRY CONSIDERED HARMFUL 
Consider  a  one-unit  backpropagation  network  trained  on  exclusive  or.  Without  hidden  units,  the  problem  is  insoluble.  One  point  where  learning  would  stop  is  when  all weights are  zero and the output is  always  ~, resulting in  an mean squared  error  of  ~.  But  this  is  a  saddle  point;  by  placing  the  discrimination  boundary  properly, one point can be gotten correctly, two with errors of ~, and one with error  of i, giving an MSE of i, as shown in figure  1.  Networks are initialized with small random weights, or noise is injected during train(cid:173) ing to break symmetries of this sort.  But in breaking this symmetry, something has  been  lost.  Consider  a  kNN  classifier,  constructed  from  a  kNN  program  and  the  training data.  Anyone  who  has a  copy  of the  kNN  program can construct  an  iden(cid:173) tical classifier  if they  receive  the training data.  Thus,  considering the  classification
VLSI Implementations of Learning and Memory Systems: A Review
Authors: Mark A. Holler
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A large number of VLSI implementations of neural network models  have been  reported. The diversity of these implementations is  noteworthy. This paper attempts to put a group of representative  VLSI implementations in perspective by comparing and contrast(cid:173) ing them. Design trade-offs are discussed and some suggestions forthe  direction of future implementation efforts are made. 
IMPLEMENTATION  Changing the way information is represented can be beneficial. For example a change  of representation can make information more compact for storage and transmission.  Implementation of neural computational models is just the process of changing the  representation of a neural model from mathmatical symbolism to a physical embodi(cid:173) ement for the purpose of shortening the time it takes to process information according  to the neural model. 
FLEXIBIliTY VS. PERFORMANCE 
Today most neural models are already implemented in silicon VLSI, in the form of pro(cid:173) grams running on general purpose digital von Neumann computers. These machines  are available at low cost and are highly flexible. Their flexibility results from the ease  with which their programs can be changed. Maximizing flexibility, however, usually  results in reduced performance. A program will often have to specify several simple op-
Neural Dynamics of Motion Segmentation and Grouping
Authors: Ennio Mingolla
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A  neural  network  model  of motion segmentation  by  visual  cortex  is  de(cid:173) scribed.  The  model  clarifies  how  preprocessing  of  motion  signals  by  a  Motion Oriented Contrast Filter (MOC  Filter)  is joined to long-range co(cid:173) operative motion mechanisms in a  motion Cooperative Competitive Loop  (CC Loop)  to control phenomena such as as induced  motion, motion cap(cid:173) ture,  and motion aftereffects.  The total model system is  a  motion Bound(cid:173) ary Contour System (BCS)  that is computed in parallel with a static BCS  before  both systems cooperate  to generate  a  boundary  representation for  three dimensional visual form perception.  The present investigations clari(cid:173) fy how the static BCS can be modified for use in motion segmentation prob(cid:173) lems, notably for analyzing how ambiguous local movements (the aperture  problem) on a  complex moving shape  are suppressed  and actively reorga(cid:173) nized  into a  coherent global motion signal. 
1 
INTRODUCTION: WHY ARE STATIC AND MOTION  BOUNDARY CONTOUR SYSTEMS  NEEDED? 
Some regions,  notably  MT,  of visual  cortex  are  specialized  for  motion processing.  However, even the earliest stages of visual cortex processing,  such as simple cells in  VI, require stimuli that change through  time for  their  maximal activation and  are  direction-sensitive.  Why has  evolution generated  regions  such  as  MT,  when  even  VI  is  change-sensitive and  direction-sensitive?  What computational properties  are  achieved by MT that are not already available in  VI?
Exploratory Feature Extraction in Speech Signals
Authors: Nathan Intrator
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A  novel  unsupervised  neural  network  for  dimensionality  reduction  which  seeks  directions  emphasizing  multimodality is  presented,  and  its  connec(cid:173) tion  to exploratory projection pursuit  methods is  discussed.  This leads to  a  new  statistical insight  to the  synaptic  modification  equations  governing  learning in  Bienenstock,  Cooper,  and  Munro  (BCM)  neurons  (1982).  The  importance  of a  dimensionality  reduction  principle  based  solely  on  distinguishing  features,  is  demonstrated  using  a  linguistically  motivated  phoneme  recognition  experiment,  and  compared  with  feature  extraction  using  back-propagation network.
Constructing Hidden Units using Examples and Queries
Authors: Kevin J. Lang Eric B. Baum
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: While the network loading problem for 2-layer threshold nets is  NP-hard when learning from examples alone (as with backpropaga(cid:173) tion), (Baum, 91) has now proved that a learner can employ queries  to evade the hidden unit credit assignment problem and PAC-load  nets with up to four hidden units in polynomial time. Empirical  tests show that the method can also learn far more complicated  functions such as randomly generated networks with 200 hidden  units. The algorithm easily approximates Wieland's 2-spirals func(cid:173) tion using a single layer of 50 hidden units, and requires only 30  minutes of CPU time to learn 200-bit parity to 99.7% accuracy.
Analog Neural Networks as Decoders
Authors: Yaser Abu-Mostafa Ruth Erlanson
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Analog neural networks with feedback can be used to implement l((cid:173) Winner-Take-All (KWTA) networks. In turn, KWTA networks can be  used as decoders of a class of nonlinear error-correcting codes. By in(cid:173) terconnecting such KWTA networks, we can construct decoders capable  of decoding more powerful codes. We consider several families of inter(cid:173) connected KWTA networks, analyze their performance in terms of coding  theory metrics, and consider the feasibility of embedding such networks in  VLSI technologies. 
1 
INTRODUCTION: THE K-WINNER-TAKE-ALL  NETWORK 
We have previously demonstrated the use of a continuous Hopfield neural network  as a K-Winner-Take-All (KWTA) network [Majani et al., 1989, Erlanson and Abu(cid:173) Mostafa, 1988}. Given an input of N real numbers, such a network will converge  to a vector of K positive one components and (N - K) negative one components,  with the positive positions indicating the K largest input components. In addition,  we have shown that the (~) such vectors are the only stable states of the system.  One application of the KWTA network is the analog decoding of error-correcting  codes [Majani et al., 1989, Platt and Hopfield, 1986]. Here, a known set of vectors  (the codewords) are transmitted over a noisy channel. At the receiver's end of the  channel, the initial vector must be reconstructed from the noisy vector. 
• currently at: Hughes Network Systems, 10790 Roselle St., San Diego, CA 92121
A Reinforcement Learning Variant for Control Scheduling
Authors: Aloke Guha
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  present  an  algorithm  based  on  reinforcement  and  state  recurrence  learning  techniques  to  solve  control  scheduling  problems.  In  particular,  we  have  devised  a  simple  learning  scheme  called  "handicapped  learning",  in  which  the  weights  of the  associative  search  element  are  reinforced,  either  positively  or negatively,  such  that the  system  is forced  to  move  towards the  desired  setpoint  in  the  shortest possible  trajectory.  To  improve  the  learning  rate,  a  variable  reinforcement  scheme  is  employed:  negative  reinforcement  values are  varied depending  on  whether the failure  occurs in  handicapped or  normal  mode  of operation.  Furthermore,  to  realize  a  simulated  annealing  scheme  for  accelerated  learning,  if the  system  visits  the  same  failed  state  successively,  the  negative  reinforcement  value  is  increased.  In  examples  studied,  these  learning  schemes  have  demonstrated  high  learning  rates,  and  therefore may prove useful  for  in-situ learning.
Kohonen Networks and Clustering: Comparative Performance in Color Clustering
Authors: Daniel Nissman, David Van den Bout, Griff Bilbro Wesley Snyder
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The problem of color clustering is defined and shown to be a problem of  assigning a large number (hundreds of thousands) of 3-vectors to a  small number (256) of clusters. Finding those clusters in such a way that  they best represent a full color image using only 256 distinct colors is a  burdensome computational problem. In this paper, the problem is solved  using "classical" techniques -- k-means clustering, vector quantization  (which turns out to be the same thing in this application), competitive  learning, and Kohonen self-organizing feature maps. Quality of the  result is judged subjectively by how much the pseudo-color result  resembles the true color image, by RMS quantization error, and by run  time. The Kohonen map provides the best solution.
Reconfigurable Neural Net Chip with 32K Connections
Authors: R. Janow, D. Henderson, R. Lee H. P. Graf
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe a CMOS neural net chip with a reconfigurable network archi(cid:173) tecture. It contains 32,768 binary, programmable connections arranged in  256 'building block' neurons. Several 'building blocks' can be connected to  form long neurons with up to 1024 binary connections or to form neurons  with analog connections. Single- or multi-layer networks can be imple(cid:173) mented with this chip. We have integrated this chip into a board system  together with a digital signal processor and fast memory. This system is  currently in use for image processing applications in which the chip extracts  features such as edges and corners from binary and gray-level images.
e-Entropy and the Complexity of Feedforward Neural Networks
Authors: Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We develop a. new feedforward neuralnet.work represent.ation of Lipschitz  functions from [0, p]n into [0,1] ba'3ed on the level sets of the function. We  show that 
~~ + ~€r + ( 1 + h) (:~) n 
is an upper bound on the number of nodes needed to represent f to within  uniform error Cr, where L is the Lipschitz constant. \Ve also show that the  number of bits needed to represent the weights in the network in order to  achieve this approximation is given by 
o (~2;~r (:~) n) . 
\Ve compare this bound with the [-entropy of the functional class under  consideration.
Extensions of a Theory of Networks for Approximation and Learning: Outliers and Negative Examples
Authors: Tomaso Poggio, Bruno Caprile Federico Girosi
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Learning an input-output mapping from a set of examples can be regarded  as synthesizing an approximation of a multi-dimensional function. From  this point of view, this form of learning is closely related to regularization  theory, and we have previously shown (Poggio and Girosi, 1990a, 1990b)  the equivalence between reglilari~at.ioll and a. class of three-layer networks  that we call regularization networks. In this note, we ext.end the theory  by introducing ways of
Stochastic Neurodynamics
Authors: J.D. Cowan
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The main point of this paper is that stochastic neural networks have a  mathematical structure that corresponds quite closely with that of  quantum field theory. Neural network Liouvillians and Lagrangians  can be derived, just as can spin Hamiltonians and Lagrangians in QFf.  It remains to show the efficacy of such a description.
Phonetic Classification and Recognition Using the Multi-Layer Perceptron
Authors: James R. Glass, Michael S. Phillips, Victor W. Zue Hong C. Leung
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: In this paper, we will describe several extensions to our earlier work, utiliz(cid:173) ing a segment-based approach. We will formulate our segmental framework  and report our study on the use of multi-layer perceptrons for detection  and classification of phonemes. We will also examine the outputs of the  network, and compare the network performance with other classifiers. Our  investigation is performed within a set of experiments that attempts to  recognize 38 vowels and consonants in American English independent of  speaker. When evaluated on the TIMIT database, our system achieves an  accuracy of 56%.
Learning Trajectory and Force Control of an Artificial Muscle Arm by Parallel-hierarchical Neural Network Model
Authors: Mitsuo Kawato Masazumi Katayama
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We propose a new parallel-hierarchical neural network model to enable motor  learning for simultaneous control of both trajectory and force. by integrating  Hogan's control method and our previous neural network control model using a  feedback-error-learning scheme. Furthermore. two hierarchical control laws  which apply to the model, are derived by using the Moore-Penrose pseudo(cid:173) inverse matrix. One is related to the minimum muscle-tension-change trajectory  and the other is related to the minimum motor-command-change trajectory. The  human arm is redundant at the dynamics level since joint torque is generated by  agonist and antagonist muscles. Therefore, acquisition of the inverse model is  an ill-posed problem. However. the combination of these control laws and  feedback-error-learning resolve the ill-posed problem. Finally. the efficiency of  the parallel-hierarchical neural network model is shown by learning experiments  using an artificial muscle arm and computer simulations.
Basis-Function Trees as a Generalization of Local Variable Selection Methods for Function Approximation
Authors: Terence D. Sanger
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Local variable selection has proven to be a powerful technique for ap(cid:173) proximating functions in high-dimensional spaces. It is used in several  statistical methods, including CART, ID3, C4, MARS, and others (see the  bibliography for references to these algorithms). In this paper I present  a tree-structured network which is a generalization of these techniques.  The network provides a framework for understanding the behavior of such  algorithms and for modifying them to suit particular applications.
A Short-Term Memory Architecture for the Learning of Morphophonemic Rules
Authors: Chan-Do Lee Michael Gasser
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Despite its successes,  Rumelhart and McClelland's (1986)  well-known ap(cid:173) proach to the learning of morphophonemic rules  suffers from two deficien(cid:173) cies:  (1)  It performs  the  artificial  task  of associating  forms  with  forms  rather  than  perception  or  production.  (2)  It is  not  constrained  in  ways  that humans learners  are.  This paper describes  a  model which  addresses  both objections.  Using  a  simple recurrent  architecture  which  takes  both  forms  and  "meanings"  as  inputs,  the  model  learns  to  generate  verbs  in  one  or  another  "tense",  given  arbitrary  meanings,  and  to  recognize  the  tenses  of verbs.  Furthermore,  it fails  to learn  reversal  processes  unknown  in human language.
Associative Memory in a Network of `Biological' Neurons
Authors: Wulfram Gerstner
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The Hopfield network (Hopfield,  1982,1984) provides a simple model of an  associative memory in  a neuronal structure.  This model, however, is based  on highly artificial assumptions, especially the use of formal-two state neu(cid:173) rons  (Hopfield,  1982) or graded-response  neurons  (Hopfield,  1984).  \Vhat  happens if we  replace  the formal neurons  by 'real' biological neurons?  \Ve  address  this question  in  two steps.  First, we  show  that a simple model of  a  neuron  can  capture  all  relevant features  of neuron  spiking,  i. e., a  wide  range of spiking frequencies  and a realistic distribution of interspike inter(cid:173) vals.  Second, we construct an associative memory by linking these neurons  together.  The analytical solution for  a  large  and fully  connected  network  shows that the  Hopfield solution  is  valid only for  neurons  with  a short re(cid:173) fractory  period.  If the refractory  period  is  longer  than  a  crit.ical  duration  ie,  the  solutions  are  qualitatively different.  The  associative  character  of  the solutions, however,  is  preserved.
Oriented Non-Radial Basis Functions for Image Coding and Analysis
Authors: Jim Christian, Dun-Sung Tang, Wu Chuan-Lin Avijit Saha
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We introduce oriented non-radial basis function networks (ONRBF)  as a generalization of Radial Basis Function networks (RBF)- wherein  the Euclidean distance metric in the exponent of the Gaussian is re(cid:173) placed by a more general polynomial. This permits the definition of  more general regions and in particular- hyper-ellipses with orienta(cid:173) tions. In the case of hyper-surface estimation this scheme requires a  smaller number of hidden units and alleviates the "curse of dimen(cid:173) sionality" associated kernel type approximators.In the case of an im(cid:173) age, the hidden units correspond to features in the image and the  parameters associated with each unit correspond to the rotation, scal(cid:173) ing and translation properties of that particular "feature". In the con(cid:173) text of the ONBF scheme, this means that an image can be  represented by a small number of features. Since, transformation of an  image by rotation, scaling and translation correspond to identical  transformations of the individual features, the ONBF scheme can be  used to considerable advantage for the purposes of image recognition  and analysis.
Evaluation of Adaptive Mixtures of Competing Experts
Authors: Geoffrey E. Hinton Steven J. Nowlan
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  compare  the  performance  of the  modular  architecture,  composed  of  competing  expert  networks,  suggested  by  Jacobs,  Jordan,  Nowlan  and  Hinton  (1991)  to  the  performance  of a  single  back-propagation  network  on  a  complex,  but  low-dimensional,  vowel  recognition  task.  Simulations  reveal that this system is capable of uncovering interesting decompositions  in  a  complex  task.  The  type  of decomposition  is  strongly  influenced  by  the  nature  of the  input  to  the  gating  network  that  decides  which  expert  to  use  for  each  case.  The  modular  architecture  also exhibits consistently  better  generalization on many variations of the  task.
Spoken Letter Recognition
Authors: Ronald Cole Mark Fanty
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Through the use of neural network classifiers and careful feature selection,  we have achieved high-accuracy speaker-independent spoken letter recog(cid:173) nition. For isolated letters, a broad-category segmentation is performed  Location of segment boundaries allows us to measure features at specific  locations in the signal such as vowel onset, where important information  resides. Letter classification is performed with a feed-forward neural net(cid:173) work. Recognition accuracy on a test set of 30 speakers was 96%. Neu(cid:173) ral network classifiers are also used for pitch tracking and broad-category  segmentation of letter strings. Our research has been extended to recog(cid:173) nition of names spelled with pauses between the letters. When searching  a database of 50,000 names, we achieved 95% first choice name retrieval.  Work has begun on a continuous letter classifier which does frame-by-frame  phonetic classification of spoken letters.
Design and Implementation of a High Speed CMAC Neural Network Using Programmable CMOS Logic Cell Arrays
Authors: Brian A. Box, Erich C. Whitney, James M. Glynn W. Thomas Miller III
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A high speed implementation of the CMAC neural network was designed  using dedicated CMOS logic. This technology was then used to implement  two general purpose CMAC associative memory boards for the VME bus.  Each board implements up to 8 independent CMAC networks with a total  of one million adjustable weights. Each CMAC network can be configured  to have from 1 to 512 integer inputs and from 1 to 8 integer outputs.  Response times for typical CMAC networks are well below 1 millisecond,  making the networks sufficiently fast for most robot control problems, and  many pattern recognition and signal processing problems.
Shaping the State Space Landscape in Recurrent Networks
Authors: Jean Pierre Raysz, Bernard Victorri Patrice Simard
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Bernard Victorri  ELSAP  Universite  de  Caen  14032 Caen  Cedex  France 
Fully  recurrent  (asymmetrical)  networks  can  be  thought  of as  dynamic  systems.  The  dynamics  can  be  shaped  to  perform  content  addressable  memories,  recognize  sequences,  or  generate  trajectories.  Unfortunately  several  problems  can  arise:  First,  the  convergence  in  the  state  space  is  not  guaranteed.  Second,  the  learned  fixed  points  or  trajectories  are  not  necessarily stable.  Finally,  there  might exist  spurious fixed  points and/or  spurious  "attracting"  trajectories that do  not correspond  to any patterns.  In this paper,  we  introduce a  new  energy  function  that  presents solutions  to all of these problems.  We present an efficient gradient descent algorithm  which directly acts on the stability of the fixed  points and trajectories and  on the size  and shape of the  corresponding basin and valley of attraction.  The results are illustrated by the simulation of a small content addressable  memory.
EMPATH: Face, Emotion, and Gender Recognition Using Holons
Authors: Janet Metcalfe Garrison W. Cottrell
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The  dimens~onali~y of a  set Off  160 1:~ :a:~s ~~·.10 .  female  subjects  IS  reduced  ........ .  network  The extracted features do not correspond to  in previ~us face  recognition systems (KaR· na~e, 19~;)y' ......••.•..  f~tures we  call  holons.  The  hol.ons  are fV~~ t~!  ..... .  ..  ......\  d'  tances  between  facial  elements.
Interaction Among Ocularity, Retinotopy and On-center/Off-center Pathways During Development
Authors: Shigeru Tanaka
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The development of projections from the retinas to the cortex is  mathematically analyzed according to the previously proposed  thermodynamic formulation of the self-organization of neural networks.  Three types of submodality included in the visual afferent pathways are  assumed in two models: model (A), in which the ocularity and retinotopy  are considered separately, and model (B), in which on-center/off-center  pathways are considered in addition to ocularity and retinotopy. Model (A)  shows striped ocular dominance spatial patterns and, in ocular dominance  histograms, reveals a dip in the binocular bin. Model (B) displays  spatially modulated irregular patterns and shows single-peak behavior in  the histograms. When we compare the simulated results with the observed  results, it is evident that the ocular dominance spatial patterns and  histograms for models (A) and (B) agree very closely with those seen in  monkeys and cats.
Phase-coupling in Two-Dimensional Networks of Interacting Oscillators
Authors: Daniel M. Kammen, Christof Koch, Daniel L. Ruderman, Heinz G. Schuster Ernst Niebur
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Coherent oscillatory activity in large networks of biological or artifi(cid:173) cial neural units may be a useful mechanism for coding information  pertaining to a single perceptual object or for  detailing regularities  within  a  data set.  We  consider  the  dynamics  of a  large  array  of  simple  coupled  oscillators  under  a  variety  of connection  schemes.  Of particular  interest  is  the  rapid  and  robust  phase-locking  that  results  from  a  "sparse"  scheme  where  each  oscillator  is  strongly  coupled  to a  tiny,  randomly selected, subset of its neighbors.