mtybilly/Instruct2DS huggingface dataset:
Prediction of Protein Topologies Using Generalized IOHMMs and RNNs
Authors: Gianluca Pollastri, Pierre Baldi, Alessandro Vullo, Paolo Frasconi
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We develop and test new machine learning methods for the predic- tion of topological representations of protein structures in the form of coarse- or (cid:12)ne-grained contact or distance maps that are transla- tion and rotation invariant. The methods are based on generalized input-output hidden Markov models (GIOHMMs) and generalized recursive neural networks (GRNNs). The methods are used to pre- dict topology directly in the (cid:12)ne-grained case and, in the coarse- grained case, indirectly by (cid:12)rst learning how to score candidate graphs and then using the scoring function to search the space of possible con(cid:12)gurations. Computer simulations show that the pre- dictors achieve state-of-the-art performance.
1
Introduction: Protein Topology Prediction
Predicting the 3D structure of protein chains from the linear sequence of amino acids is a fundamental open problem in computational molecular biology [1]. Any approach to the problem must deal with the basic fact that protein structures are translation and rotation invariant. To address this invariance, we have proposed a machine learning approach to protein structure prediction [4] based on the predic- tion of topological representations of proteins, in the form of contact or distance maps. The contact or distance map is a 2D representation of neighborhood rela- tionships consisting of an adjacency matrix at some distance cuto(cid:11) (typically in the range of 6 to 12 (cid:23)A), or a matrix of pairwise Euclidean distances. Fine-grained maps are derived at the amino acid or even atomic level. Coarse maps are obtained by looking at secondary structure elements, such as helices, and the distance between their centers of gravity or, as in the simulations below, the minimal distances be- tween their C(cid:11) atoms. Reasonable methods for reconstructing 3D coordinates from contact/distance maps have been developed in the NMR literature and elsewhere
Neural Control of Sensory Acquisition: The Vestibulo-Ocular Reflex
Authors: Michael G. Paulin, Mark E. Nelson, James M. Bower
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present a new hypothesis that the cerebellum plays a key role in ac(cid:173) tively controlling the acquisition of sensory infonnation by the nervous  system.  In this paper we explore this idea by examining the function of  a  simple  cerebellar-related  behavior,  the  vestibula-ocular  reflex  or  VOR, in  which  eye movements  are generated to minimize image slip  on  the  retina  during  rapid  head  movements.  Considering  this  system  from  the point of view of statistical estimation theory, our results  sug(cid:173) gest that the transfer function of the VOR, often regarded as a static or  slowly  modifiable  feature  of the  system,  should  actually  be  continu(cid:173) ously and rapidly changed during head movements. We further suggest  that these changes are under the direct control of the cerebellar cortex  and propose experiments to test this hypothesis.
Modeling the Olfactory Bulb - Coupled Nonlinear Oscillators
Authors: Zhaoping Li, John J. Hopfield
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The olfactory  bulb of mammals  aids  in  the  discrimination  of  odors.  A  mathematical  model  based  on  the  bulbar  anatomy  and  electrophysiology  is  described.  Simulations  produce  a  35-60  Hz  modulated activity coherent across the bulb, mimicing the observed  field  potentials.  The  decision  states  (for  the  odor  information)  here  can  be  thought  of as  stable  cycles,  rather  than  point  stable  states  typical  of simpler  neuro-computing  models.  Analysis  and  simulations show that a  group of coupled non-linear oscillators are  responsible for the oscillatory activities determined by the odor in(cid:173) put, and that the bulb, with appropriate inputs from higher centers,  can  enhance  or suppress  the  sensitivity  to  partiCUlar  odors.  The  model provides a framework  in which to understand the transform  between odor input and the bulbar output to olfactory cortex.
Efficient Parallel Learning Algorithms for Neural Networks
Authors: Alan H. Kramer, Alberto Sangiovanni-Vincentelli
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Parallelizable optimization techniques are applied to the problem of  learning in feedforward neural networks. In addition to having supe(cid:173) rior convergence properties, optimization techniques such as the Polak(cid:173) Ribiere method are also significantly more efficient than the Back(cid:173) propagation algorithm. These results are based on experiments per(cid:173) formed on small boolean learning problems and the noisy real-valued  learning problem of hand-written character recognition.
The Boltzmann Perceptron Network: A Multi-Layered Feed-Forward Network Equivalent to the Boltzmann Machine
Authors: Eyal Yair, Allen Gersho
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The  concept  of  the  stochastic  Boltzmann  machine  (BM)  is  auractive  for  decision  making  and  pattern  classification  purposes  since  the  probability  of  attaining  the network  states  is a  function  of the network energy.  Hence,  the  probability of attaining particular energy minima  may be associated  with  the  probabilities  of  making  certain  decisions  (or  classifications).  However,  because of its stochastic  nature,  the complexity of the BM is fairly  high and  therefore  such  networks  are  not  very  likely  to  be  used  in  practice.  In  this  paper  we  suggest  a  way  to  alleviate  this  drawback  by  converting  the  sto(cid:173) chastic  BM into  a  deterministic  network  which  we  call  the  Boltzmann  Per(cid:173) ceptron  Network  (BPN).  The BPN is functionally  equivalent  to  the  BM but  has  a  feed-forward  structure  and  low  complexity.  No annealing  is required.  The  conditions  under  which  such  a  convmion  is  feasible  are  given.  A  learning  algorithm  for  the  BPN based  on  the  conjugate  gradient  method  is  also provided which is somewhat akin  to the backpropagation algorithm.
Links Between Markov Models and Multilayer Perceptrons
Authors: Hervé Bourlard, C. J. Wellekens
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Hidden Markov models are widely used for automatic speech recog(cid:173) nition. They inherently incorporate the sequential character of the  speech signal and are statistically trained. However, the a-priori  choice of the model topology limits their flexibility. Another draw(cid:173) back of these models is their weak discriminating power. Multilayer  perceptrons are now promising tools in the connectionist approach  for classification problems and have already been successfully tested  on speech recognition problems. However, the sequential nature of  the speech signal remains difficult to handle in that kind of ma(cid:173) chine. In this paper, a discriminant hidden Markov model is de(cid:173) fined and it is shown how a particular multilayer perceptron with  contextual and extra feedback input units can be considered as a  general form of such Markov models.
Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment
Authors: Michael Mozer, Paul Smolensky
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper proposes a means of using the knowledge in a network to  determine the functionality or relevance of individual units, both for  the purpose of understanding the network's behavior and improving its  performance. The basic idea is to iteratively train the network to a cer(cid:173) tain performance criterion, compute a measure of relevance that identi(cid:173) fies which input or hidden units are most critical to performance, and  automatically trim the least relevant units. This skeletonization tech(cid:173) nique can be used to simplify networks by eliminating units that con(cid:173) vey redundant information; to improve learning performance by first  learning with spare hidden units and then trimming the unnecessary  ones away, thereby constraining generalization; and to understand the  behavior of networks in terms of minimal "rules."
A Passive Shared Element Analog Electrical Cochlea
Authors: David Feld, Joe Eisenberg, Edwin Lewis
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present a  simplified model  of the  micromechanics of the human  cochlea,  realized  with  electrical  elements.  Simulation  of the  model  shows that it retains four signal processing features whose importance  we argue on the basis of engineering logic and evolutionary evidence.  Furthermore, just as  the cochlea does,  the  model  achieves  massively  parallel signal processing in a structurally economic way, by means of  shared elements.  By extracting what we believe are the five essential  features of the cochlea, we hope to design a useful  front-end  filter to  process  acoustic  images and to  obtain  a  better understanding  of the  auditory system.
Speech Production Using A Neural Network with a Cooperative Learning Mechanism
Authors: Mitsuo Komura, Akio Tanaka
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  propose  a  new  neural  network  model  and  its  learning  algorithm. The proposed neural network consists of four layers  - input, hidden, output and final output layers. The hidden and  output layers are multiple.  Using the proposed  SICL(Spread  Pattern Information and Cooperative  Learning)  algorithm,  it  is  possible  to  learn  analog  data  accurately  and  to  obtain  smooth outputs. Using this neural network, we have developed  a  speech production system consisting of a  phonemic  symbol  production  subsystem  and  a  speech  parameter  production  subsystem.  We  have  succeeded  in  producing  natural  speech  waves with high accuracy.
Comparing Biases for Minimal Network Construction with Back-Propagation
Authors: Stephen Jose Hanson, Lorien Y. Pratt
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: learning 
representations during 
Rumelhart (1987). has proposed a method for choosing minimal or  "simple"  in Back-propagation  networks. This approach can be used to (a) dynamically select the  number of hidden units. (b) construct a representation that is  appropriate for the problem and (c) thus improve the generalization  ability of Back-propagation networks. The method Rumelhart suggests  involves adding penalty terms to the usual error function. In this paper  we introduce Rumelhart·s minimal networks idea and compare two  possible biases on the weight search space. These biases are compared  in both simple counting problems and a speech recognition problem.  In general. the constrained search does seem to minimize the number of  hidden units required with an expected increase in local minima.
What Size Net Gives Valid Generalization?
Authors: Eric B. Baum, David Haussler
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  address  the  question  of when  a  network  can  be  expected  to  generalize from m  random training examples chosen from some ar(cid:173) bitrary probability distribution, assuming that future test examples  are drawn from  the same  distribution.  Among  our  results are  the  following  bounds on appropriate sample vs.  network size.  Assume  o <  £  $  1/8.  We  show  that  if m  >  O( ~log~) random  exam(cid:173) ples  can  be  loaded  on  a  feedforward  network  of linear  threshold  functions  with N  nodes and W  weights,  so that at least a  fraction  1 - t of the examples  are  correctly  classified,  then one  has confi(cid:173) dence approaching certainty that the network will correctly classify  a  fraction  1 - £  of future  test  examples drawn from  the same  dis(cid:173) tribution.  Conversely,  for  fully-connected  feedforward  nets  with  one  hidden  layer,  any learning  algorithm  using  fewer  than  O( '!')  random training examples  will,  for  some distributions of examples  consistent  with  an  appropriate  weight  choice,  fail  at  least  some  fixed fraction of the time to find  a  weight choice that will correctly  classify more  than a 1 - £  fraction of the future  test examples.
A Low-Power CMOS Circuit Which Emulates Temporal Electrical Properties of Neurons
Authors: Jack L. Meador, Clint S. Cole
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This  paper  describes  a  CMOS  artificial  neuron.  The  circuit  is 
directly  derived  from  the  voltage-gated  channel  model  of  neural 
membrane,  has  low  power  dissipation,  and  small  layout  geometry. 
The principal motivations behind this work include a desire for high 
performance,  more  accurate  neuron  emulation,  and  the  need  for 
higher density in practical neural network implementations.
Applications of Error Back-Propagation to Phonetic Classification
Authors: Hong C. Leung, Victor W. Zue
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper is concerced with the use of error back-propagation  in phonetic classification. Our objective is to investigate the ba(cid:173) sic characteristics of back-propagation, and study how the frame(cid:173) work of multi-layer perceptrons can be exploited in phonetic recog(cid:173) nition. We explore issues such as integration of heterogeneous  sources of information, conditioll~ that can affect performance of  phonetic classification, internal representations, comparisons with  traditional pattern classification techniques, comparisons of differ(cid:173) ent error metrics, and initialization of the network. Our investiga(cid:173) tion is performed within a set of experiments that attempts to rec(cid:173) ognize the 16 vowels in American English independent of speaker.  Our results are comparable to human performance. 
Early approaches in phonetic recognition fall into two major extremes: heuristic  and algorithmic. Both approaches have their own merits and shortcomings. The  heuristic approach has the intuitive appeal that it focuses on the linguistic informa(cid:173) tion in the speech signal and exploits acoustic-phonetic knowledge. HO'fever, the  weak control strategy used for utilizing our knowledge has been grossly inadequate.  At the other extreme, the algorithmic approach relies primarily on the powerful con(cid:173) trol strategy offered by well-formulated pattern recognition techniques. However,  relatively little is known about how our speech knowledge accumulated over the  past few decades can be incorporated into the well-formulated algorithms. We feel  that artificial neural networks (ANN) have some characteristics that can potentially  enable them to bridge the gap between these two extremes. On the one hand, our  speech knowledge can provide guidance to the structure and design of the network.  On the other hand, the self-organizing mechanism of ANN can provide a control  strategy for utilizing our knowledge. 
In this paper, we extend our earlier work on the use of artificial neural networks  for phonetic recognition [2]. Specifically, we focus our investigation on the following  sets of issues. First, we describe the use of the network to integrate heterogeneous  sources of information. We will see how classification performance improves as more 
Error Back-Propagation to Phonetic Classification 
207 
information is available. Second, we discuss several important factors that can sub(cid:173) stantially affect the performance of phonetic classification. Third, we examine the  internal representation of the network. Fourth, we compare the network with two  traditional classification techniques: K-nearest neighbor and Gaussian classifica(cid:173) tion. Finally, we discuss our specific implementations of back-propagation that  yield improved performance and more efficient learning time.
An Analog Self-Organizing Neural Network Chip
Authors: James R. Mann, Sheldon Gilbert
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A design for a fully analog version of a self-organizing feature map neural  network has been completed. Several parts of this design are in fabrication.  The feature map algorithm was modified to accommodate circuit solutions  to the various computations required. Performance effects were measured  by simulating the design as part of a frontend for a speech recognition  system. Circuits are included to implement both activation computations and  weight adaption 'or learning. External access to the analog weight values is  provided to facilitate weight initialization, testing and static storage. This  fully analog implementation requires an order of magnitude less area than  a comparable digital/analog hybrid version developed earlier.
Heterogeneous Neural Networks for Adaptive Behavior in Dynamic Environments
Authors: Randall D. Beer, Hillel J. Chiel, Leon S. Sterling
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Research  in artificial neural networks has genera1ly emphasized  homogeneous architectures. In contrast, the nervous systems of natural  animals exhibit great heterogeneity in both their elements and patterns  of interconnection. This heterogeneity is crucial to the flexible  generation of behavior which is essential for survival in a complex,  dynamic environment. It may also provide powerful insights into the  design of artificial neural networks.  In this paper, we describe a  heterogeneous neural network for controlling  the wa1king of a  simulated insect. This controller is inspired by the neuroethological  It exhibits a  and neurobiological literature on insect locomotion.  variety of statically stable gaits at different speeds simply by varying  the tonic activity of a single cell. It can also adapt to perturbations as a  natural consequence of its design.
Programmable Analog Pulse-Firing Neural Networks
Authors: Alister Hamilton, Alan F. Murray, Lionel Tarassenko
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  describe  pulse  - stream  firing  integrated  circuits  that  imple(cid:173) ment asynchronous analog neural networks.  Synaptic weights are  stored  dynamically,  and  weighting  uses  time-division  of  the  neural  pulses  from  a  signalling  neuron  to  a  receiving  neuron.  MOS  transistors  in  their  "ON"  state  act  as  variable  resistors  to  control  a  capacitive  discharge,  and  time-division  is  thus  achieved  by  a  small  synapse  circuit  cell.  The  VLSI  chip  set  design  uses  2.5J.1.m  CMOS technology.
Simulation and Measurement of the Electric Fields Generated by Weakly Electric Fish
Authors: Brian Rasnow, Christopher Assad, Mark E. Nelson, James M. Bower
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The weakly electric fish, Gnathonemus peters;;, explores its environment by gener(cid:173) ating pulsed elecbic fields  and detecting small pertwbations in the fields  resulting from  nearby objects.  Accordingly, the fISh  detects and discriminates objects on  the basis of a  sequence of elecbic "images" whose temporal and spatial properties depend on  the  tim(cid:173) ing of the fish's electric organ discharge and its body position relative to objects in its en(cid:173) vironmenl  We are interested in investigating how these fish utilize timing and body-po(cid:173) sition during exploration to aid in object discrimination.  We have developed a fmite-ele(cid:173) ment simulation of the fish's self-generated electric  fields  so as  to  reconstruct the elec(cid:173) trosensory consequences of body position and electric organ discharge timing in the fish.  This paper describes this finite-element simulation system and presents preliminary elec(cid:173) tric field measurements which are being used to tune the simulation.
Training Multilayer Perceptrons with the Extended Kalman Algorithm
Authors: Sharad Singhal, Lance Wu
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: trained with 
A large fraction of recent work in artificial neural nets uses  multilayer perceptrons  the back-propagation  algorithm described by Rumelhart et. a1. This algorithm  converges slowly for large or complex problems such as  speech recognition, where thousands of iterations may be  needed for convergence even with small data sets. In this  paper, we show that training multilayer perceptrons is an  identification problem for a nonlinear dynamic system which  can be solved using  the Extended Kalman Algorithm.  Although computationally complex, the Kalman algorithm  usually converges in a few  the  algorithm and compare it with back-propagation using two(cid:173) dimensional examples. 
iterations. We describe
An Electronic Photoreceptor Sensitive to Small Changes in Intensity
Authors: Tobi Delbrück, C. A. Mead
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We describe an electronic photoreceptor circuit that is sensitive to  small changes in incident light intensity. The sensitivity to change8  in the intensity is achieved by feeding back to the input a filtered  version of the output. The feedback loop includes a hysteretic el(cid:173) ement. The circuit behaves in a manner reminiscent of the gain  control properties and temporal responses of a variety of retinal  cells, particularly retinal bipolar cells. We compare the thresholds  for detection of intensity increments by a human and by the cir(cid:173) cuit. Both obey Weber's law and for both the temporal contrast  sensitivities are nearly identical. 
We previously described an electronic photoreceptor that outputs a voltage that is  logarithmic in the light intensity (Mead, 1985). This report describes an extension  of this circuit which was based on a suggestion by Frank Werblin that biological  retinas may achieve greater sensitivity to change8 in the illumination by feeding  back a filtered version of the output. 
OPERATION OF THE CIRCUIT 
The circuit (Figure 1) consists of a phototransistor (P), exponential feedback to  P (Ql, Q2, and Q3), a transconductance amplifier (A), and the hysteretic element  (Q4 and Qs). In general terms the operation of the circuit consists of two stages of  amplification with hysteresis in the feedback loop. The light falls on the parasitic  bipolar transistor P. (The rest of the circuit is shielded by metal.) P's collector  is the substrate and the base is an isolated well. P and Ql form the first stage of  amplification. The light produces a base current Is for P. The emitter current IE  is PIs, neglecting collector resistance for now. P is typically a few hundred. The  feedback current IQl is set by the gate voltage on QdQ2' which is set by the current  through Q3, which is set by the feedback voltage Vjb. In equilibrium Vjb will be  such that IQl = IE and some voltage Vp will be the output of the first stage. The 
An Electronic Photoreceptor Sensitive to Small Changes 
721 
negative feedback through the transconductance amplifier A will make Vp ~ V/ b•  This voltage is logarithmic in the light intensity, since in subthreshold operation  the currents through Q2 and Q3 are exponential in their gate to source voltages.  The DC output of the circuit will be Vout ~ V/b = Vdd - (2kT /q) log IE, neglecting  the back-gate effect for Q2. Figure Sa (DC output) shows that the assumption of  subthreshold operation is valid over about 4 orders of magnitude.
Training a 3-Node Neural Network is NP-Complete
Authors: Avrim Blum, Ronald L. Rivest
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We consider  a  2-layer,  3-node,  n-input neural network whose  nodes  compute linear threshold functions  of their inputs.  We  show  that it  is NP-complete to decide whether there exist weights and thresholds  for the three nodes of this network so that it will produce output con(cid:173) sistent  with  a  given set of training examples.  We  extend  the result  to other simple networks.  This result suggests that those looking for  perfect  training  algorithms  cannot  escape  inherent  computational  difficulties just by  considering only simple or very  regular networks.  It also suggests the importance, given a training problem, of finding  an  appropriate network  and input encoding for  that problem.  It is  left as an open problem to extend our result to nodes with non-linear  functions such as  sigmoids.
Adaptive Neural Net Preprocessing for Signal Detection in Non-Gaussian Noise
Authors: Richard P Lippmann, Paul Beckman
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  nonlinearity  is  required  before  matched  filtering  in  mInimum  error  receivers  when  additive  noise  is  present  which  is  impulsive  and  highly  non-Gaussian.  Experiments  were  performed  to  determine  whether  the  correct clipping  nonlinearity  could  be provided  by  a  single-input  single(cid:173) output  multi-layer  perceptron  trained  with  back  propagation.  It  was  found  that a  multi-layer perceptron with one input and output node,  20  nodes  in  the  first  hidden  layer,  and  5  nodes  in  the  second  hidden  layer  could be trained to provide a  clipping nonlinearity with fewer than 5,000  presentations  of noiseless  and  corrupted  waveform  samples.  A  network  trained  at  a  relatively  high  signal-to-noise  (SIN)  ratio  and  then  used  as  a  front  end  for  a  linear  matched  filter  detector greatly  reduced  the  probability  of error.  The  clipping  nonlinearity  formed  by  this  network  was similar to that used in current receivers designed for impulsive  noise  and  provided  similar substantial  improvements in  performance.
GENESIS: A System for Simulating Neural Networks
Authors: Matthew A. Wilson, Upinder S. Bhalla, John D. Uhley, James M. Bower
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: support  simulations  at  many  it  is 
We  have  developed  a  graphically  oriented,  general  purpose  simulation  system  to  facilitate  the  modeling  of  neural  networks.  The  simulator  is  implemented  under  UNIX  and  X-windows  and  is  levels  of  detail.  designed  to  in  both  applied  network  Specifically,  modeling  and  in  the  simulation  of  detailed,  realistic,  biologically(cid:173) based  models.  Examples  of  current  models  developed  under  this  system  include  mammalian  olfactory  bulb  and  cortex,  invertebrate  central  pattern  generators,  as  well  as  more  abstract  connectionist  simulations. 
intended  for  use
Learning by Choice of Internal Representations
Authors: Tal Grossman, Ronny Meir, Eytan Domany
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  introduce  a  learning algorithm for  multilayer neural  net(cid:173) works  composed of binary linear threshold elements.  Whereas ex(cid:173) isting algorithms reduce  the learning process  to minimizing a  cost  function  over  the  weights,  our  method  treats  the  internal  repre(cid:173) sentations  as  the fundamental entities  to  be  determined.  Once  a  correct set of internal representations is  arrived at, the weights are  found  by  the  local  aild  biologically plausible Perceptron  Learning  Rule  (PLR).  We tested  our  learning algorithm on  four  problems:  adjacency, symmetry, parity and  combined symmetry-parity.
Adaptive Neural Networks Using MOS Charge Storage
Authors: Daniel B. Schwartz, R. E. Howard, Wayne E. Hubbard
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: MOS charge storage has been demonstrated as an effective method to store  the  weights  in  VLSI  implementations  of  neural  network  models  by  several  workers  2 .  However,  to  achieve  the  full  power  of a  VLSI  implementation  of  an adaptive algorithm, the learning operation must built into the circuit.  We  have  fabricated  and  tested  a  circuit  ideal  for  this  purpose  by  connecting  a  pair of capacitors with  a  CCD like structure, allowing for  variable size  weight  changes  as  well  as  a  weight  decay  operation.  A  2.51-'  CMOS  version  achieves  better  than  10  bits  of dynamic  range  in  a  140/'  X  3501-'  area.  A  1.25/'  chip  based  upon  the  same  cell  has  1104  weights  on  a  3.5mm  x  6.0mm  die  and  is  capable of peak learning rates  of at least  2  x  109  weight  changes  per second. 
1  Adaptive  Networks 
Much  of the  recent  excitement  about  neural  network  models  of computation  has  been  driven  by  the  prospect  of new  architectures  for  fine  grained  parallel  compu(cid:173) tation using analog VLSI.  Adaptive systems are espescially good targets for  analog  VLSI because the ada.ptive process  can compensate for  the inaccuracy of individual  devices  as easily as for  the variability of the signal.  However, silicon  VLSI  does  not  provide  us  with  an  ideal solution  for  weight  storage.  Among  the  properties  of an  ideal storage technology for  analog VLSI  adaptive systems  are: 
•  The minimum available weight change ~w must be small.  The simplest adap(cid:173)
tive  algorithms optimize  the  weights  by  minimizing  the  output  error  with  a  steepest  descent  search  in weight space  [1].  Iterative improvement algorithms  such  as  steepest  descent  are  based  on  the  heuristic  assumption  of  'better'  weights being found  in  the neighborhood of 'good' ones;  a  heuristic that fails  when  the granularity of the weights is  not fine  enough.  In the worst  case,  the  resolution required just to represent  a  function  can  grow exponentially in  the  dimension of the input space . 
•  The  weights  must be able  to represent  both  positive and negative values  and  the  changes  must  be easily  reversible.  Frequently,  the  weights  may cycle  up  and down while the adaptive process is converging and millions of incremental  changes  during  a  single  training session  is  not  unreasonable.  If the  weights  cannot easily  follow  all  of these  changes,  then  the  learning  must  be  done  off  chip. 
1 Now  at GTE Laboratories, 40 Sylvan Rd., Waltham, Mass 02254  dbs@gte.com%relay.cs.net  2For example, see  the  papers by Mann and Gilbert, Walker and Akers,  and Murray  et.  al.  in
Backpropagation and Its Application to Handwritten Signature Verification
Authors: Timothy S. Wilkinson, Dorothy A. Mighell, Joseph W. Goodman
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  pool  of handwritten  signatures  is  used  to  train  a  neural  net(cid:173) work for the task of deciding whether or not a  given signature is a  forgery.  The network is  a feedforward  net, with a binary image as  input.  There is a hidden layer, with a single unit output layer.  The  weights are  adjusted according to the backpropagation algorithm.  The signatures are entered into a C  software program through the  use of a Datacopy Electronic Digitizing Camera.  The binary signa(cid:173) tures  are normalized  and centered.  The performance is  examined  as  a  function of the training set  and network structure.  The  best  scores  are  on  the  order of 2%  true signature  rejection  with  2-4%  false  signature acceptance.
Dynamics of Analog Neural Networks with Time Delay
Authors: Charles M. Marcus, R. M. Westervelt
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A time delay in the response of the neurons in a network can  induce sustained oscillation and chaos. We present a stability  criterion based on local stability analysis to prevent sustained  oscillation  in  symmetric  delay  networks,  and  show  an  example  of chaotic  dynamics  in  a  non-symmetric  delay  network.
On the K-Winners-Take-All Network
Authors: E. Majani, Ruth Erlanson, Yaser S. Abu-Mostafa
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present  and  rigorously  analyze a generalization of the Winner(cid:173) Take-All  Network:  the  K-Winners-Take-All  Network.  This  net(cid:173) work  identifies  the  K  largest  of a  set  of N  real  numbers.  The  network  model used  is  the continuous Hopfield  model. 
I  - INTRODUCTION 
The Winner-Take-All  Network  is  a  network  which  identifies  the  largest  of N  real  numbers.  Winner-Take-All  Networks  have  been  developed  using  various  neural  networks models (Grossberg-73, Lippman-87, Feldman-82, Lazzaro-89).  We present  here  a  generalization  of the  Winner-Take-All  Network:  the  K-Winners-Take-All  (KWTA) Network.  The KWTA Network identifies the K  largest of N  real numbers.  The neural network model we  use throughout the paper is  the continuous Hopfield  network  model  (Hopfield-84).  If the states of the  N  nodes are initialized  to the N  real numbers, then, if the gain of the sigmoid is large enough, the network converges  to the state with  K  positive real numbers in  the positions of the nodes with  the K  largest  initial states, and  N  - K  negative  real  numbers everywhere else.  Consider  the  following  example:  N  = 4,  K  = 2.  There  are  6  =  (~)  stable  states:(++-)T, (++)T, (+--+)T, ( _ ++)T, (++)T, and (++)T.  If the initial  state of the  network  is  (0.3,  -0.4,  0.7,  O.l)T,  then  the network  will  converge to (Vi,V2,V3,v4)T  where Vi> 0,  V2  < 0,  V3  > 0,  V4  < 0  ((+ _ +_)T).  In Section II, we  define  the KWTA  Network (connection weights, external inputs).  In Section III,  we  analyze  the equilibrium states and  in  Section  IV,  we  identify  all  the stable equilibrium states of the KWTA Network.  In Section V, we  describe the  dynamics  of the  KWTA  Network.  In  Section  VI,  we  give  two  important examples  of the KWTA  Network and comment on an alternate implementation of the KWTA  Network. 
On the K-Winners-Take-All Network 
635 
II - THE K-WINNERS-TAKE-ALL  NETWORK 
The continuous Hopfield network model (Hopfield-84)  (also known as the Grossberg  additive model  (Grossberg-88)), is characterized by a system of first order differen(cid:173) tial equations which governs the evolution of the state of the network (i = 1, .. . , N) : 
The  sigmoid  function  g(u)  is  defined  by:  g(u)  = f(G·  u),  where  G  >  0  is  the  gain  of  the  sigmoid,  and  f(u)  is  defined  by:  1.  "f/u,  0  <  f'(u)  <  f'(O)  = 1,  2.  limu .... +oo  f( u) = 1,  3.  limu .... -oo f( u) = -l.  The KWTA Network is characterized by mutually inhibitory interconnections Taj  =  -1  for  i  ¥=  j, a self connection Tai  = a,  (Ial < 1) and'an external input (identical  for  every node) which  depends on the number  K  of winners desired and the size of  the network  N  : ti = 2K - N.  The differential equations for  the KWTA  Network are therefore: 
for  all  i,  Cd~i = -Aui + (a + l)g(ui) - (tg(u j )  - t) ,
Use of Multi-Layered Networks for Coding Speech with Phonetic Features
Authors: Yoshua Bengio, Régis Cardin, Renato de Mori, Piero Cosi
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Preliminary  results  on  speaker-independant  speech  recognition  are  reported.  A method  that combines  expertise  on  neural  networks  with  expertise  on  speech  recognition  is  used  to  build  the  recognition  systems.  For  transient  sounds,  event(cid:173) driven  property  extractors  with  variable  resolution  in  the  time  and  frequency  domains  are  used.  For  sonorant  speech,  a  model  of the  human  auditory  system  is  preferred  to  FFT  as  a  front-end  module.
A Programmable Analog Neural Computer and Simulator
Authors: Paul Mueller, Jan Van der Spiegel, David Blackman, Timothy Chiu, Thomas Clare, Joseph Dao, Christopher Donham, Tzu-pu Hsieh, Marc Loinaz
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This  report describes  the  design  of a  programmable general  purpose analog neural computer and simulator.  It is intended  primarily  for  real-world  real-time  computations  such  as  analysis  of visual  or  acoustical patterns, robotics  and the development of  special purpose  neural nets.  The machine is scalable and  composed of interconnected  modules containing arrays of neurons, modifiable synapses and switches.  It runs  entirely in analog  mode but connection architecture, synaptic  gains and time  constants as well as neuron parameters are set digitally.  Each  neuron has a limited number of inputs and can be connected to any  but not all other neurons. For the determination of synaptic gains and the  implementation  of  learning  algorithms  the  neuron  outputs  are  multiplexed, AID  converted and stored in digital  memory.  Even at  moderate size of 1()3 to IDS neurons  computational speed is expected to  exceed that of any current  digital computer.
An Information Theoretic Approach to Rule-Based Connectionist Expert Systems
Authors: Rodney M. Goodman, John W. Miller, Padhraic Smyth
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We discuss in this paper architectures for executing probabilistic rule-bases in a par(cid:173) allel manner,  using  as  a theoretical basis recently introduced information-theoretic  models.  We will begin by describing our (non-neural) learning algorithm and theory  of quantitative rule  modelling, followed  by  a discussion on  the exact nature of two  particular models.  Finally we work through an example of our approach, going from  database to rules to inference network, and compare the network's performance with  the theoretical limits for  specific  problems.
A Model for Resolution Enhancement (Hyperacuity) in Sensory Representation
Authors: Jun Zhang, John P. Miller
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Heiligenberg (1987)  recently proposed a  model  to explain how sen(cid:173)
sory  maps  could  enhance  resolution  through  orderly  arrangement  of 
broadly tuned receptors.  We  have  extended  this  model  to the general 
case  of polynomial  weighting  schemes  and  proved  that  the  response 
function  is  also  a  polynomial  of the  same  order.  We  further  demon(cid:173)
strated  that  the  Hermitian  polynomials  are  eigenfunctions  of the  sys(cid:173)
tem.  Finally we suggested  a  biologically  plausible mechanism for  sen(cid:173)
sory representation of external stimuli with resolution far exceeding the 
inter-receptor separation.
Neural Network Star Pattern Recognition for Spacecraft Attitude Determination and Control
Authors: Phillip Alvelda, A. Miguel San Martin
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: computational  bottlenecks 
Currently,  the  most  complex  spacecraft  attitude  determination  and  control  tasks  are  ultimately  governed  by  ground-based  systems  and  personnel.  Conventional  on-board  systems  face  severe  serial  microprocessors operating on inherently parallel problems.  New  computer architectures based on the anatomy of the human brain  seem  to  promise  high  speed  and  fault-tolerant  solutions  to  the  limitations  of serial  processing.  This  paper  discusses  the  latest  applications of artificial neural networks to the  problem of star  pattern recognition  for spacecraft attitude determination.
Neural Net Receivers in Multiple Access-Communications
Authors: Bernd-Peter Paris, Geoffrey Orsak, Mahesh Varanasi, Behnaam Aazhang
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The application of neural networks to the demodulation of  spread-spectrum signals in a multiple-access environment is  considered. This study is motivated in large part by the fact  that, in a multiuser system, the conventional (matched fil(cid:173) ter) receiver suffers severe performance degradation as the  relative powers of the interfering signals become large (the  "near-far" problem). Furthermore, the optimum receiver,  which alleviates the near-far problem, is too complex to be  of practical use. Receivers based on multi-layer perceptrons  are considered as a simple and robust alternative to the opti(cid:173) mum solution. The optimum receiver is used to benchmark  the performance of the neural net receiver; in particular, it is  proven to be instrumental in identifying the decision regions  of the neural networks. The back-propagation algorithm and  a modified version of it are used to train the neural net. An  importance sampling technique is introduced to reduce the  number of simulations necessary to evaluate the performance  of neural nets. In all examples considered the proposed neu(cid:173) ral ~et receiver significantly outperforms the conventional  recelver.
A Computationally Robust Anatomical Model for Retinal Directional Selectivity
Authors: Norberto M. Grzywacz, Franklin R. Amthor
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We analyze a mathematical model for  retinal directionally selective  cells  based  on  recent  electrophysiological  data,  and  show  that  its  computation of motion direction is  robust against  noise  and speed.
Learning Sequential Structure in Simple Recurrent Networks
Authors: David Servan-Schreiber, Axel Cleeremans, James L. McClelland
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We explore a network architecture introduced by Elman (1988) for  predicting successive elements of a sequence. The network uses the  pattern of activation over a set of hidden units from time-step t-l,  together with element t, to predict element t+ 1. When the network is  trained with strings from a particular finite-state grammar, it can learn  to be a perfect finite-state recognizer for the grammar. Cluster analyses  of the hidden-layer patterns of activation showed that they encode  prediction-relevant information about the entire path traversed through  the network. We illustrate the phases of learning with cluster analyses  performed at different points during training. 
Several connectionist architectures that are explicitly constrained to capture  sequential infonnation have been developed. Examples are Time Delay  Networks (e.g. Sejnowski & Rosenberg. 1986) -- also called 'moving  window' paradigms -- or algorithms such as back-propagation in time  (Rumelhart. Hinton & Williams. 1986), Such architectures use explicit  representations of several consecutive events. if not of the entire history of  past inputs. Recently. Elman (1988) has introduced a simple recurrent  network (SRN) that has the potential to master an infinite corpus of  sequences with the limited means of a learning procedure that is completely  local in time (see Figure I.).
GEMINI: Gradient Estimation Through Matrix Inversion After Noise Injection
Authors: Yann Le Cun, Conrad C. Galland, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Learning procedures that measure how random perturbations of unit ac(cid:173) tivities correlate with changes in reinforcement are inefficient but simple  to implement in hardware. Procedures like back-propagation (Rumelhart,  Hinton and Williams, 1986) which compute how changes in activities af(cid:173) fect the output error are much more efficient, but require more complex  hardware. GEMINI is a hybrid procedure for multilayer networks, which  shares many of the implementation advantages of correlational reinforce(cid:173) ment procedures but is more efficient. GEMINI injects noise only at the  first hidden layer and measures the resultant effect on the output error.  A linear network associated with each hidden layer iteratively inverts the  matrix which relates the noise to the error change, thereby obtaining  the error-derivatives. No back-propagation is involved, thus allowing un(cid:173) known non-linearities in the system. Two simulations demonstrate the  effectiveness of GEMINI.
Neural Networks for Model Matching and Perceptual Organization
Authors: Eric Mjolsness, Gene Gindi, P. Anandan
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We introduce an optimization approach for solving problems in com(cid:173)
puter vision that involve multiple levels of abstraction. Our objective  functions include compositional and specialization hierarchies. We cast  vision problems as inexact graph matching problems, formulate graph  matching in terms of constrained optimization, and use analog neural  networks to perform the optimization. The method is applicable to per(cid:173) ceptual grouping and model matching. Preliminary experimental results  are shown.
Neuronal Maps for Sensory-Motor Control in the Barn Owl
Authors: Clay D. Spence, John C. Pearson, J. J. Gelfand, R. M. Peterson, W. E. Sullivan
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The bam owl has fused visual/auditory/motor representations of  space in its midbrain which are used to orient the head so that visu(cid:173) al or auditory stimuli are centered in the visual field of view. We  present models and computer simulations of these structures which  address various problems, inclu
Performance of Synthetic Neural Network Classification of Noisy Radar Signals
Authors: Stanley C. Ahalt, F. D. Garber, I. Jouny, Ashok K. Krishnamurthy
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This study evaluates the performance of the multilayer-perceptron  and  the frequency-sensitive  competitive  learning network  in  iden(cid:173) tifying  five  commercial  aircraft  from  radar  backscatter  measure(cid:173) ments.  The performance  of the neural network  classifiers  is  com(cid:173) pared  with  that of the  nearest-neighbor  and  maximum-likelihood  classifiers.  Our  results  indicate  that  for  this  problem,  the  neural  network  classifiers  are  relatively  insensitive  to  changes  in  the  net(cid:173) work  topology,  and  to the noise  level  in  the  training data.  While,  for  this  problem,  the traditional algorithms outperform these sim(cid:173) ple neural classifiers,  we  feel  that neural networks show  the poten(cid:173) tial for  improved performance.