mtybilly/Instruct2DS huggingface dataset:
Order Reduction for Dynamical Systems Describing the Behavior of Complex Neurons
Authors: Thomas B. Kepler, L. F. Abbott, Eve Marder
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We have devised a scheme to reduce the complexity of dynamical  systems belonging to a class that includes most biophysically realistic  neural models. The reduction is based on transformations of variables  and perturbation expansions and it preserves a high level of fidelity to  the original system. The techniques are illustrated by reductions of the  Hodgkin-Huxley system and an augmented Hodgkin-Huxley system.
Note on Learning Rate Schedules for Stochastic Optimization
Authors: Christian Darken, John E. Moody
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  present  and  compare  learning  rate  schedules  for  stochastic  gradient  descent,  a  general  algorithm  which  includes  LMS,  on-line  backpropaga(cid:173) tion  and  k-means  clustering  as  special  cases.  We  introduce  "search-then(cid:173) converge"  type  schedules  which  outperform  the  classical  constant  and  "running average"  (1ft) schedules both in speed of convergence and quality  of solution. 
Introduction:  Stochastic  Gradient  Descent 
1  tion G(W).  In  the context  of learning systems  typically G(W) = £x E(W, X),  i.e. 
The  optimization  task  is  to  find  a  parameter  vector  W  which  minimizes  a  func(cid:173)
G  is  the  average  of  an  objective  function  over  the  exemplars,  labeled  E  and  X  respectively.  The stochastic gradient  descent  algorithm is  Ll Wet)  = -1](t)V'w E(W(t), X(t)). 
where  t  is  the  "time",  and  X(t)  is  the  most  recent  independently-chosen  random  exemplar.  For comparison, the  deterministic gradient  descent  algorithm is 
Ll Wet)  =  -1](t)V'w£x E(W(t), X).
A Recurrent Neural Network for Word Identification from Continuous Phoneme Strings
Authors: Robert B. Allen, Candace A. Kamm
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A  neural  network  architecture  was  designed  for  locating  word  boundaries  and  identifying  words  from  phoneme  sequences.  This  architecture  was  tested  in  three  sets  of  studies.  First,  a  highly  redundant  corpus  with  a  restricted  vocabulary was  generated and the network was trained with a limited number of  phonemic variations for the words  in the corpus.  Tests of network performance  on a transfer set yielded a very low error rate.  In a second study, a network was  trained  to  identify  words  from  expert  transcriptions  of speech.  On a  transfer  test,  error  rate  for  correct  simultaneous  identification  of  words  and  word  boundaries was  18%.  The third study used the output of a phoneme classifier as  the input to the word and  word boundary identification network.  The error rate  on a transfer test set was 49% for this task.  Overall, these studies provide a first  step at identifying words in connected discourse with a neural network.
Adjoint-Functions and Temporal Learning Algorithms in Neural Networks
Authors: N. Toomarian, J. Barhen
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The development of learning algorithms is generally based upon the min(cid:173) imization of an energy function. It is a fundamental requirement to com(cid:173) pute the gradient of this energy function with respect to the various pa(cid:173) rameters of the neural architecture, e.g., synaptic weights, neural gain,etc.  In principle, this requires solving a system of nonlinear equations for each  parameter of the model, which is computationally very expensive. A new  methodology for neural learning of time-dependent nonlinear mappings is  presented. It exploits the concept of adjoint operators to enable a fast  global computation of the network's response to perturbations in all the  systems parameters. The importance of the time boundary conditions of  the adjoint functions is discussed. An algorithm is presented in which  the adjoint sensitivity equations are solved simultaneously (Le., forward  in time) along with the nonlinear dynamics of the neural networks. This  methodology makes real-time applications and hardware implementation  of temporal learning feasible.
Chaitin-Kolmogorov Complexity and Generalization in Neural Networks
Authors: Barak A. Pearlmutter, Ronald Rosenfeld
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  present  a  unified  framework  for  a  number  of different  ways  of failing  to  generalize  properly.  During  learning,  sources  of random  information  contaminate  the  network,  effectively  augmenting  the  training  data  with  random information. The complexity of the function computed is therefore  increased,  and generalization is degraded.  We analyze replicated networks,  in  which  a  number of identical networks are independently trained on  the  same data and their results averaged.  We conclude that replication almost  always results in a decrease in the expected complexity of the network, and  that  replication  therefore  increases  expected  generalization.  Simulations  confirming the effect  are  also presented. 
1  BROKEN SYMMETRY CONSIDERED HARMFUL 
Consider  a  one-unit  backpropagation  network  trained  on  exclusive  or.  Without  hidden  units,  the  problem  is  insoluble.  One  point  where  learning  would  stop  is  when  all weights are  zero and the output is  always  ~, resulting in  an mean squared  error  of  ~.  But  this  is  a  saddle  point;  by  placing  the  discrimination  boundary  properly, one point can be gotten correctly, two with errors of ~, and one with error  of i, giving an MSE of i, as shown in figure  1.  Networks are initialized with small random weights, or noise is injected during train(cid:173) ing to break symmetries of this sort.  But in breaking this symmetry, something has  been  lost.  Consider  a  kNN  classifier,  constructed  from  a  kNN  program  and  the  training data.  Anyone  who  has a  copy  of the  kNN  program can construct  an  iden(cid:173) tical classifier  if they  receive  the training data.  Thus,  considering the  classification
Constructing Hidden Units using Examples and Queries
Authors: Eric B. Baum, Kevin J. Lang
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: While the network loading problem for 2-layer threshold nets is  NP-hard when learning from examples alone (as with backpropaga(cid:173) tion), (Baum, 91) has now proved that a learner can employ queries  to evade the hidden unit credit assignment problem and PAC-load  nets with up to four hidden units in polynomial time. Empirical  tests show that the method can also learn far more complicated  functions such as randomly generated networks with 200 hidden  units. The algorithm easily approximates Wieland's 2-spirals func(cid:173) tion using a single layer of 50 hidden units, and requires only 30  minutes of CPU time to learn 200-bit parity to 99.7% accuracy.
Analog Neural Networks as Decoders
Authors: Ruth Erlanson, Yaser Abu-Mostafa
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Analog neural networks with feedback can be used to implement l((cid:173) Winner-Take-All (KWTA) networks. In turn, KWTA networks can be  used as decoders of a class of nonlinear error-correcting codes. By in(cid:173) terconnecting such KWTA networks, we can construct decoders capable  of decoding more powerful codes. We consider several families of inter(cid:173) connected KWTA networks, analyze their performance in terms of coding  theory metrics, and consider the feasibility of embedding such networks in  VLSI technologies. 
1 
INTRODUCTION: THE K-WINNER-TAKE-ALL  NETWORK 
We have previously demonstrated the use of a continuous Hopfield neural network  as a K-Winner-Take-All (KWTA) network [Majani et al., 1989, Erlanson and Abu(cid:173) Mostafa, 1988}. Given an input of N real numbers, such a network will converge  to a vector of K positive one components and (N - K) negative one components,  with the positive positions indicating the K largest input components. In addition,  we have shown that the (~) such vectors are the only stable states of the system.  One application of the KWTA network is the analog decoding of error-correcting  codes [Majani et al., 1989, Platt and Hopfield, 1986]. Here, a known set of vectors  (the codewords) are transmitted over a noisy channel. At the receiver's end of the  channel, the initial vector must be reconstructed from the noisy vector. 
• currently at: Hughes Network Systems, 10790 Roselle St., San Diego, CA 92121
Kohonen Networks and Clustering: Comparative Performance in Color Clustering
Authors: Wesley Snyder, Daniel Nissman, David Van den Bout, Griff Bilbro
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The problem of color clustering is defined and shown to be a problem of  assigning a large number (hundreds of thousands) of 3-vectors to a  small number (256) of clusters. Finding those clusters in such a way that  they best represent a full color image using only 256 distinct colors is a  burdensome computational problem. In this paper, the problem is solved  using "classical" techniques -- k-means clustering, vector quantization  (which turns out to be the same thing in this application), competitive  learning, and Kohonen self-organizing feature maps. Quality of the  result is judged subjectively by how much the pseudo-color result  resembles the true color image, by RMS quantization error, and by run  time. The Kohonen map provides the best solution.
Reconfigurable Neural Net Chip with 32K Connections
Authors: H. P. Graf, R. Janow, D. Henderson, R. Lee
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe a CMOS neural net chip with a reconfigurable network archi(cid:173) tecture. It contains 32,768 binary, programmable connections arranged in  256 'building block' neurons. Several 'building blocks' can be connected to  form long neurons with up to 1024 binary connections or to form neurons  with analog connections. Single- or multi-layer networks can be imple(cid:173) mented with this chip. We have integrated this chip into a board system  together with a digital signal processor and fast memory. This system is  currently in use for image processing applications in which the chip extracts  features such as edges and corners from binary and gray-level images.
Extensions of a Theory of Networks for Approximation and Learning: Outliers and Negative Examples
Authors: Federico Girosi, Tomaso Poggio, Bruno Caprile
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Learning an input-output mapping from a set of examples can be regarded  as synthesizing an approximation of a multi-dimensional function. From  this point of view, this form of learning is closely related to regularization  theory, and we have previously shown (Poggio and Girosi, 1990a, 1990b)  the equivalence between reglilari~at.ioll and a. class of three-layer networks  that we call regularization networks. In this note, we ext.end the theory  by introducing ways of
Phonetic Classification and Recognition Using the Multi-Layer Perceptron
Authors: Hong C. Leung, James R. Glass, Michael S. Phillips, Victor W. Zue
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: In this paper, we will describe several extensions to our earlier work, utiliz(cid:173) ing a segment-based approach. We will formulate our segmental framework  and report our study on the use of multi-layer perceptrons for detection  and classification of phonemes. We will also examine the outputs of the  network, and compare the network performance with other classifiers. Our  investigation is performed within a set of experiments that attempts to  recognize 38 vowels and consonants in American English independent of  speaker. When evaluated on the TIMIT database, our system achieves an  accuracy of 56%.
Learning Trajectory and Force Control of an Artificial Muscle Arm by Parallel-hierarchical Neural Network Model
Authors: Masazumi Katayama, Mitsuo Kawato
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We propose a new parallel-hierarchical neural network model to enable motor  learning for simultaneous control of both trajectory and force. by integrating  Hogan's control method and our previous neural network control model using a  feedback-error-learning scheme. Furthermore. two hierarchical control laws  which apply to the model, are derived by using the Moore-Penrose pseudo(cid:173) inverse matrix. One is related to the minimum muscle-tension-change trajectory  and the other is related to the minimum motor-command-change trajectory. The  human arm is redundant at the dynamics level since joint torque is generated by  agonist and antagonist muscles. Therefore, acquisition of the inverse model is  an ill-posed problem. However. the combination of these control laws and  feedback-error-learning resolve the ill-posed problem. Finally. the efficiency of  the parallel-hierarchical neural network model is shown by learning experiments  using an artificial muscle arm and computer simulations.
A Short-Term Memory Architecture for the Learning of Morphophonemic Rules
Authors: Michael Gasser, Chan-Do Lee
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Despite its successes,  Rumelhart and McClelland's (1986)  well-known ap(cid:173) proach to the learning of morphophonemic rules  suffers from two deficien(cid:173) cies:  (1)  It performs  the  artificial  task  of associating  forms  with  forms  rather  than  perception  or  production.  (2)  It is  not  constrained  in  ways  that humans learners  are.  This paper describes  a  model which  addresses  both objections.  Using  a  simple recurrent  architecture  which  takes  both  forms  and  "meanings"  as  inputs,  the  model  learns  to  generate  verbs  in  one  or  another  "tense",  given  arbitrary  meanings,  and  to  recognize  the  tenses  of verbs.  Furthermore,  it fails  to learn  reversal  processes  unknown  in human language.
Oriented Non-Radial Basis Functions for Image Coding and Analysis
Authors: Avijit Saha, Jim Christian, Dun-Sung Tang, Wu Chuan-Lin
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We introduce oriented non-radial basis function networks (ONRBF)  as a generalization of Radial Basis Function networks (RBF)- wherein  the Euclidean distance metric in the exponent of the Gaussian is re(cid:173) placed by a more general polynomial. This permits the definition of  more general regions and in particular- hyper-ellipses with orienta(cid:173) tions. In the case of hyper-surface estimation this scheme requires a  smaller number of hidden units and alleviates the "curse of dimen(cid:173) sionality" associated kernel type approximators.In the case of an im(cid:173) age, the hidden units correspond to features in the image and the  parameters associated with each unit correspond to the rotation, scal(cid:173) ing and translation properties of that particular "feature". In the con(cid:173) text of the ONBF scheme, this means that an image can be  represented by a small number of features. Since, transformation of an  image by rotation, scaling and translation correspond to identical  transformations of the individual features, the ONBF scheme can be  used to considerable advantage for the purposes of image recognition  and analysis.
Evaluation of Adaptive Mixtures of Competing Experts
Authors: Steven J. Nowlan, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  compare  the  performance  of the  modular  architecture,  composed  of  competing  expert  networks,  suggested  by  Jacobs,  Jordan,  Nowlan  and  Hinton  (1991)  to  the  performance  of a  single  back-propagation  network  on  a  complex,  but  low-dimensional,  vowel  recognition  task.  Simulations  reveal that this system is capable of uncovering interesting decompositions  in  a  complex  task.  The  type  of decomposition  is  strongly  influenced  by  the  nature  of the  input  to  the  gating  network  that  decides  which  expert  to  use  for  each  case.  The  modular  architecture  also exhibits consistently  better  generalization on many variations of the  task.
Spoken Letter Recognition
Authors: Mark Fanty, Ronald Cole
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Through the use of neural network classifiers and careful feature selection,  we have achieved high-accuracy speaker-independent spoken letter recog(cid:173) nition. For isolated letters, a broad-category segmentation is performed  Location of segment boundaries allows us to measure features at specific  locations in the signal such as vowel onset, where important information  resides. Letter classification is performed with a feed-forward neural net(cid:173) work. Recognition accuracy on a test set of 30 speakers was 96%. Neu(cid:173) ral network classifiers are also used for pitch tracking and broad-category  segmentation of letter strings. Our research has been extended to recog(cid:173) nition of names spelled with pauses between the letters. When searching  a database of 50,000 names, we achieved 95% first choice name retrieval.  Work has begun on a continuous letter classifier which does frame-by-frame  phonetic classification of spoken letters.
Design and Implementation of a High Speed CMAC Neural Network Using Programmable CMOS Logic Cell Arrays
Authors: W. Thomas Miller III, Brian A. Box, Erich C. Whitney, James M. Glynn
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A high speed implementation of the CMAC neural network was designed  using dedicated CMOS logic. This technology was then used to implement  two general purpose CMAC associative memory boards for the VME bus.  Each board implements up to 8 independent CMAC networks with a total  of one million adjustable weights. Each CMAC network can be configured  to have from 1 to 512 integer inputs and from 1 to 8 integer outputs.  Response times for typical CMAC networks are well below 1 millisecond,  making the networks sufficiently fast for most robot control problems, and  many pattern recognition and signal processing problems.
Shaping the State Space Landscape in Recurrent Networks
Authors: Patrice Simard, Jean Pierre Raysz, Bernard Victorri
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Bernard Victorri  ELSAP  Universite  de  Caen  14032 Caen  Cedex  France 
Fully  recurrent  (asymmetrical)  networks  can  be  thought  of as  dynamic  systems.  The  dynamics  can  be  shaped  to  perform  content  addressable  memories,  recognize  sequences,  or  generate  trajectories.  Unfortunately  several  problems  can  arise:  First,  the  convergence  in  the  state  space  is  not  guaranteed.  Second,  the  learned  fixed  points  or  trajectories  are  not  necessarily stable.  Finally,  there  might exist  spurious fixed  points and/or  spurious  "attracting"  trajectories that do  not correspond  to any patterns.  In this paper,  we  introduce a  new  energy  function  that  presents solutions  to all of these problems.  We present an efficient gradient descent algorithm  which directly acts on the stability of the fixed  points and trajectories and  on the size  and shape of the  corresponding basin and valley of attraction.  The results are illustrated by the simulation of a small content addressable  memory.
EMPATH: Face, Emotion, and Gender Recognition Using Holons
Authors: Garrison W. Cottrell, Janet Metcalfe
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The  dimens~onali~y of a  set Off  160 1:~ :a:~s ~~·.10 .  female  subjects  IS  reduced  ........ .  network  The extracted features do not correspond to  in previ~us face  recognition systems (KaR· na~e, 19~;)y' ......••.•..  f~tures we  call  holons.  The  hol.ons  are fV~~ t~!  ..... .  ..  ......\  d'  tances  between  facial  elements.
Phase-coupling in Two-Dimensional Networks of Interacting Oscillators
Authors: Ernst Niebur, Daniel M. Kammen, Christof Koch, Daniel L. Ruderman, Heinz G. Schuster
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Coherent oscillatory activity in large networks of biological or artifi(cid:173) cial neural units may be a useful mechanism for coding information  pertaining to a single perceptual object or for  detailing regularities  within  a  data set.  We  consider  the  dynamics  of a  large  array  of  simple  coupled  oscillators  under  a  variety  of connection  schemes.  Of particular  interest  is  the  rapid  and  robust  phase-locking  that  results  from  a  "sparse"  scheme  where  each  oscillator  is  strongly  coupled  to a  tiny,  randomly selected, subset of its neighbors.
An Analog VLSI Splining Network
Authors: Daniel B. Schwartz, Vijay K. Samalam
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  have produced  a  VLSI  circuit capable of learning  to approximate ar(cid:173) bitrary  smooth  of a  single  variable  using  a  technique  closely  related  to  splines.  The circuit effectively  has  512 knots space on  a  uniform grid and  has full support for  learning.  The circuit  also can be used  to approximate  multi-variable functions as sum of splines. 
An  interesting,  and  as  of yet,  nearly  untapped set of applications for  VLSI  imple(cid:173) mentation of neural network learning systems can be found in adaptive control and  non-linear signal processing.  In  most such  applications,  the  learning  task  consists  of approximating a  real  function  of a  small number  of continuous  variables  from  discrete  data points.  Special purpose hardware is especially  interesting for  applica(cid:173) tions of this  type since  they  generally  require  real  time on-line learning  and  there  can  be stiff constraints on  the power  budget and size  of the hardware.  Frequently,  the already difficult  learning problem is made more complex by the non-stationary  nature of the underlying process.  Conventional feed-forward  networks  with sigmoidal units are clearly  inappropriate  for applications of this type.  Although they have exhibited remarkable performance  in some types of time series  prediction  problems (for example, Wiegend,  1990  and  Atlas, 1990), their learning rates in general are too slow for on-line learning.  On-line  performance can be improved most easily by using networks with more constrained  architecture, effectively making the learning problem easier by giving the network a  hint about the learning task.  Networks that build local representations of the data,  such  as  radial  basis  functions,  are excellent  candidates for  these  type of problems.  One  great  advantage  of such  networks  is  that  they  require  only  a  single  layer  of  units.  If the position and width of the units are fixed,  the learning problem is linear
Learning to See Rotation and Dilation with a Hebb Rule
Authors: Martin I. Sereno, Margaret E. Sereno
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Previous work (M.I. Sereno, 1989; cf. M.E. Sereno, 1987) showed that a  feedforward network with area VI-like input-layer units and a Hebb rule  can develop area MT-like second layer units that solve the aperture  problem for pattern motion. The present study extends this earlier work  to more complex motions. Saito et al. (1986) showed that neurons with  large receptive fields in macaque visual area MST are sensitive to  different senses of rotation and dilation, irrespective of the receptive field  location of the movement singularity. A network with an MT-like  second layer was trained and tested on combinations of rotating, dilating,  and translating patterns. Third-layer units learn to detect specific senses  of rotation or dilation in a position-independent fashion, despite having  position-dependent direction selectivity within their receptive fields.
A Comparative Study of the Practical Characteristics of Neural Network and Conventional Pattern Classifiers
Authors: Kenney Ng, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Seven different pattern classifiers were implemented on a serial computer  and compared using artificial and speech recognition tasks. Two neural  network (radial basis function and high order polynomial GMDH network)  and five conventional classifiers (Gaussian mixture, linear tree, K nearest  neighbor, KD-tree, and condensed K nearest neighbor) were evaluated.  Classifiers were chosen to be representative of different approaches to pat(cid:173) tern classification and to complement and extend those evaluated in a  previous study (Lee and Lippmann, 1989). This and the previous study  both demonstrate that classification error rates can be equivalent across  different classifiers when they are powerful enough to form minimum er(cid:173) ror decision regions, when they are properly tuned, and when sufficient  training data is available. Practical characteristics such as training time,  classification time, and memory requirements, however, can differ by or(cid:173) ders of magnitude. These results suggest that the selection of a classifier  for a particular task should be guided not so much by small differences in  error rate, but by practical considerations concerning memory usage, com(cid:173) putational resources, ease of implementation, and restrictions on training  and classification times.
Real-time autonomous robot navigation using VLSI neural networks
Authors: Lionel Tarassenko, Michael Brownlow, Gillian Marshall, Jan Tombs, Alan Murray
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe a real time robot navigation system based on three VLSI  neural network modules. These are a resistive grid for path planning, a  nearest-neighbour classifier for localization using range data from a time(cid:173) of-flight infra-red sensor and a sensory-motor associative network for dy(cid:173) namic obstacle avoidance .
CAM Storage of Analog Patterns and Continuous Sequences with 3N2 Weights
Authors: Bill Baird, Frank Eeckman
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A simple architecture and algorithm for analytically guaranteed associa(cid:173) tive memory storage of analog patterns, continuous sequences, and chaotic  attractors in the same network is described. A matrix inversion determines  network weights, given prototype patterns to be stored. There are N units  of capacity in an N node network with 3N 2 weights. It costs one unit per  static attractor, two per Fourier component of each sequence, and four per  chaotic attractor. There are no spurious attractors, and there is a Lia(cid:173) punov function in a special coordinate system which governs the approach  of transient states to stored trajectories. Unsupervised or supervised incre(cid:173) mental learning algorithms for pattern classification, such as competitive  learning or bootstrap Widrow-Hoff can easily be implemented. The archi(cid:173) tecture can be "folded" into a recurrent network with higher order weights  that can be used as a model of cortex that stores oscillatory and chaotic  attractors by a Hebb rule. Hierarchical sensory-motor control networks  may be constructed of interconnected "cortical patches" of these network  modules. Network performance is being investigated by application to the  problem of real time handwritten digit recognition.
Generalization Properties of Radial Basis Functions
Authors: Sherif M. Botros, Christopher G. Atkeson
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We examine the ability of radial basis functions  (RBFs) to generalize.  We  compare the performance of several types of RBFs.  We use the inverse dy(cid:173) namics of an idealized  two-joint  arm as  a  test case.  We find  that without  a  proper  choice of a  norm for  the inputs,  RBFs have  poor  generalization  properties.  A simple global scaling of the input variables greatly improves  performance.  We suggest some efficient methods to approximate this dis(cid:173) tance  metric.
Learning Time-varying Concepts
Authors: Anthony Kuh, Thomas Petsche, Ronald L. Rivest
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: This work extends computational learning theory to situations in which concepts  vary over time, e.g., system identification of a time-varying plant. We have  extended formal definitions of concepts and learning to provide a framework  in which an algorithm can track a concept as it evolves over time. Given  this framework and focusing on memory-based algorithms, we have derived  some PAC-style sample complexity results that determine, for example, when  tracking is feasible. We have also used a similar framework and focused on  incremental tracking algorithms for which we have derived some bounds on  the mistake or error rates for some specific concept classes.
A Theory for Neural Networks with Time Delays
Authors: Bert de Vries, José Carlos Príncipe
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We present a new neural network model for processing of temporal  patterns.  This  model,  the  gamma  neural model,  is as  general  as  a  convolution  delay  model  with  arbitrary  weight  kernels  w(t).  We  show  that  the  gamma  model  can  be  formulated  as  a  (partially  prewired)  additive  model.  A  temporal  hebbian  learning  rule  is  derived  and  we  establish  links  to  related  existing  models  for  temporal processing.
Natural Dolphin Echo Recognition Using an Integrator Gateway Network
Authors: Herbert L. Roitblat, Patrick W. B. Moore, Paul E. Nachtigall, Ralph H. Penner
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We have been studying the performance of a bottlenosed dolphin on  a delayed matching-to-sample task to gain insight into the processes and  mechanisms that the animal uses during echolocation. The dolphin  recognizes targets by emitting natural sonar signals and listening to the  echoes that return. This paper describes a novel neural network  architecture, called an integrator gateway network, that we have de(cid:173) veloped to account for this performance. The integrator gateway  network combines information from multiple echoes to classify targets  with about 90% accuracy. In contrast, a standard backpropagation  network performed with only about 63% accuracy.
Compact EEPROM-based Weight Functions
Authors: A. Kramer, C. K. Sin, R. Chu, P. K. Ko
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We are focusing on the development of a  highly compact neural net weight  function based on the use of EEPROM devices.  These devices have already  proven  useful  for  analog  weight storage,  but  existing  designs  rely  on  the  use  of conventional voltage multiplication as the weight function,  requiring  additional  transistors  per  synapse.  A  parasitic  capacitance  between  the  floating gate and the drain of the EEPROM structure leads to an unusual  J-V characteristic  which can  be used  to advantage in  designing a compact  synapse.  This  novel  behavior  is  well  characterized  by  a  model  we  have  developed.  A single-device circuit results in a  1-quadrant synapse function  which  is  nonlinear,  though  monotonic.  A  simple  extension  employing  2  EEPROMs  results  in  a  2  quadrant  function  which  is  much  more  linear.  This approach offers  the  potential for  more than a  ten-fold  increase  in  the  density of neural net  implementations. 
1 
INTRODUCTION - ANALOG  WEIGHTING 
The recent surge of interest in neural networks and parallel analog computation has  motivated  the  need  for  compact  analog computing blocks.  Analog  weighting is  an  important computational function  of this class.  Analog  weighting is  the combining  of two analog values, one  of which  is  typically varying (the input) and one of which  is  typically fixed  (the  weight)  or  at  least  varying  more  slowly.  The  varying  value  is  "weighted"  by  the  fixed  value  through  the  "weighting function",  typically  mul(cid:173) tiplication.  Analog  weighting is  most  interesting  when  the  overall computational  task  involves  computing  the  "weighted  sum  of the  inputs."  That  is,  to  compute 
2:7=1  t(lOj, Vi)  where  to  is  the  weighting function  and  ~v =  {lOb W2,  ... , wn}  and
Discovering Viewpoint-Invariant Relationships That Characterize Objects
Authors: Richard S. Zemel, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Using  an  unsupervised  learning  procedure,  a  network  is  trained on  an en(cid:173) semble of images of the same two-dimensional object at different positions,  orientations  and  sizes.  Each  half of  the  network  "sees"  one  fragment  of  the object, and  tries to produce  as  output a set of 4 parameters that have  high mutual information with the 4 parameters output by  the other half of  the network.  Given the ensemble of training patterns, the 4 parameters on  which the two halves of the network can agree are the position, orientation,  and  size  of the  whole  object,  or  some  recoding  of them.  After  training,  the network can reject  instances of other shapes by  using the fact  that the  predictions  made  by  its  two  halves  disagree.  If two  competing  networks  are trained on  an unlabelled mixture of images of two objects, they  cluster  the training cases on the basis of the objects' shapes,  independently of the  position, orientation, and size.
Second Order Properties of Error Surfaces: Learning Time and Generalization
Authors: Yann LeCun, Ido Kanter, Sara A. Solla
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The learning time of a simple neural network model is obtained through an  analytic computation of the eigenvalue spectrum for the Hessian matrix,  which describes the second order properties of the cost function in the  space of coupling coefficients. The form of the eigenvalue distribution  suggests new techniques for accelerating the learning process, and provides  a theoretical justification for the choice of centered versus biased state  variables.
Connectionist Music Composition Based on Melodic and Stylistic Constraints
Authors: Michael Mozer, Todd Soukup
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe  a  recurrent  connectionist  network,  called CONCERT,  that  uses  a  set  of  melodies  written  in  a  given  style  to  compose  new  melodies  in  that  style. CONCERT  is  an extension of a traditional algorithmic composition tech(cid:173) nique  in  which  transition  tables specify  the  probability  of the  next  note  as  a  function of previous context.  A central ingredient of CONCERT  is the use of a  psychologically-grounded representation of pitch.
Transforming Neural-Net Output Levels to Probability Distributions
Authors: John S. Denker, Yann LeCun
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: (1)  The  outputs  of a  typical  multi-output  classification  network  do  not  satisfy the axioms of probability; probabilities should be positive and sum  to one.  This problem  can  be solved  by  treating  the trained  network  as  a  preprocessor that produces  a  feature  vector that can be further  processed,  for instance by classical statistical estimation techniques.  (2) We present a  method for computing the first two moments ofthe probability distribution  indicating the range of outputs that are  consistent  with the input and the  training  data.  It is  particularly  useful  to  combine  these  two  ideas:  we  implement the  ideas  of section  1 using  Parzen  windows,  where  the  shape  and relative size  of each  window  is  computed  using the ideas of section  2.  This  allows  us  to make  contact  between  important  theoretical ideas  (e.g.  the  ensemble  formalism)  and  practical  techniques  (e.g.  back-prop).  Our  results  also  shed  new  light  on  and  generalize  the  well-known  "soft max"  scheme. 
1  Distribution of Categories in Output Space 
In  many  neural-net  applications,  it  is  crucial  to  produce  a  set  of C  numbers  that  serve  as estimates of the probability of C  mutually exclusive outcomes.  For exam(cid:173) ple,  in  speech  recognition,  these  numbers  represent  the  probability of C  different  phonemes;  the probabilities of successive segments can be combined using a  Hidden  Markov  Model.  Similarly,  in  an  Optical  Character  Recognition  ("OCR")  applica(cid:173) tion,  the numbers  represent  C  possible characters.  Probability information for  the  "best  guess"  category  (and  probable runner-up  categories)  is  combined  with  con(cid:173) text,  cost  information, etcetera,  to produce  recognition of multi-character strings.
Can neural networks do better than the Vapnik-Chervonenkis bounds?
Authors: David Cohn, Gerald Tesauro
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: \Ve describe a series of careful llumerical experiments which measure the  average generalization capability of neural networks trained on a variety of  simple functions. These experiments are designed to test whether average  generalization performance can surpass the worst-case bounds obtained  from formal learning theory using the Vapnik-Chervonenkis dimension  (Blumer et al., 1989). We indeed find that, in some cases, the average  generalization is significantly better than the VC bound: the approach to  perfect performance is exponential in the number of examples m, rather  than the 11m result of the bound. In other cases, we do find the 11m  behavior of the VC bound, and in these cases, the numerical prefactor is  closely related to prefactor contained in the bound.
Distributed Recursive Structure Processing
Authors: Geraldine Legendre, Yoshiro Miyata, Paul Smolensky
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Harmonic grammar (Legendre,  et al., 1990) is a connectionist theory of lin(cid:173) guistic  well-formed ness  based on the assumption  that the well-formedness  of a  sentence  can  be  measured  by  the  harmony  (negative  energy)  of the  corresponding  connectionist  state.  Assuming  a  lower-level  connectionist  network that obeys a  few  general connectionist  principles  but is  otherwise  unspecified,  we  construct  a  higher-level  network  with  an  equivalent  har(cid:173) mony function  that captures the most linguistically relevant global aspects  of the  lower  level  network.  In  this  paper,  we  extend  the  tensor  product  representation  (Smolensky  1990)  to fully  recursive  representations  of re(cid:173) cursively  structured objects like  sentences  in  the lower-level  network.  We  show  theoretically  and  with  an  example  the  power  of the  new  technique  for  parallel distributed  structure  processing.
On the Circuit Complexity of Neural Networks
Authors: V. P. Roychowdhury, K. Y. Siu, A. Orlitsky, T. Kailath
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: '~le introduce a geometric approach for investigating the power of threshold  circuits. Viewing n-variable boolean functions as vectors in 'R'2", we invoke  tools from linear algebra and linear programming to derive new results on  the realizability of boolean functions using threshold gat.es.  Using this approach, one can obtain: (1) upper-bounds on the number of  spurious memories in HopfielJ networks, and on the number of functions  implementable by a depth-d threshold circuit; (2) a lower bound on the  number of ort.hogonal input. functions required to implement. a threshold  function; (3) a necessary condit.ion for an arbit.rary set of input. functions to  implement a threshold function; (4) a lower bound on the error introduced  in approximating boolean functions using sparse polynomials; (5) a limit  on the effectiveness of the only known lower-bound technique (based on  computing correlations among boolean functions) for the depth of thresh(cid:173) old circuit.s implement.ing boolean functions, and (6) a constructive proof  that every boolean function f of n input variables is a threshold function  of polynomially many input functions, none of which is significantly cor(cid:173) related with f. Some of these results lead t.o genera.lizations of key results  concerning threshold circuit complexity, particularly t.hose that are based  on the so-called spectral or Ha.rmonic analysis approach. Moreover, our  geometric approach yields simple proofs, based on elementary results from  linear algebra, for many of these earlier results.
An Attractor Neural Network Model of Recall and Recognition
Authors: Eytan Ruppin, Yehezkel Yeshurun
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: This  work  presents  an  Attractor  Neural  Network  (ANN)  model  of Re(cid:173) call  and  Recognition.  It is  shown  that  an  ANN  model  can  qualitatively  account  for  a  wide  range  of experimental  psychological  data  pertaining  to  the  these  two  main  aspects  of memory  access.  Certain  psychological  phenomena  are  accounted  for,  including  the  effects  of list-length,  word(cid:173) frequency,  presentation  time,  context  shift,  and  aging.  Thereafter,  the  probabilities of successful  Recall  and  Recognition  are  estimated,  in  order  to  possibly enable further  quantitative examination of the model.
Training Knowledge-Based Neural Networks to Recognize Genes in DNA Sequences
Authors: Michiel O. Noordewier, Geoffrey G. Towell, Jude W. Shavlik
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We describe the application of a hybrid symbolic/connectionist machine  learning algorithm to the task of recognizing important genetic sequences.  The symbolic portion of the KBANN system utilizes inference rules that  provide a roughly-correct method for recognizing a class of DNA sequences  known as eukaryotic splice-junctions. We then map this "domain theory"  into a neural network and provide training examples. Using the samples,  the neural network's learning algorithm adjusts the domain theory so that  it properly classifies these DNA sequences. Our procedure constitutes  a general method for incorporating preexisting knowledge into artificial  neural networks. We present an experiment in molecular genetics that  demonstrates the value of doing so.
Multi-Layer Perceptrons with B-Spline Receptive Field Functions
Authors: Stephen H. Lane, Marshall Flax, David Handelman, Jack Gelfand
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Multi-layer perceptrons are often slow to learn nonlinear functions  with complex local structure due to the global nature of their function  approximations. It is shown that standard multi-layer perceptrons are  actually a special case of a more general network formulation that  incorporates B-splines into the node computations. This allows novel  spline network architectures to be developed that can combine the  generalization capabilities and scaling properties of global multi-layer  feedforward networks with the computational efficiency and learning  speed of local computational paradigms. Simulation results are  presented for the well known spiral problem of Weiland and of Lang  and Witbrock to show the effectiveness of the Spline Net approach. 
1.
Planning with an Adaptive World Model
Authors: Sebastian Thrun, Knut Möller, Alexander Linden
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We present a new connectionist planning method [TML90].  By interaction  with  an  unknown  environment,  a  world  model  is  progressively  construc(cid:173) ted  using  gradient  descent.  For  deriving  optimal actions  with  respect  to  future  reinforcement,  planning is  applied in two steps:  an experience  net(cid:173) work proposes a  plan which is subsequently optimized by gradient descent  with  a  chain of world  models,  so  that  an  optimal reinforcement  may be  obtained  when  it  is  actually  run.  The  appropriateness  of this  method  is  demonstrated by a  robotics  application and a  pole balancing task.
Simple Spin Models for the Development of Ocular Dominance Columns and Iso-Orientation Patches
Authors: J.D. Cowan, A. E. Friedman
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Simple classical spin models well-known to physicists as the ANNNI  and Heisenberg XY Models. in which long-range interactions occur in  a pattern given by the Mexican Hat operator. can generate many of the  structural properties characteristic of the ocular dominance columns  and iso-orientation patches seen in cat and primate visual cortex.