mtybilly/Instruct2DS huggingface dataset:
Prediction of Protein Topologies Using Generalized IOHMMs and RNNs
Authors: Pierre Baldi, Alessandro Vullo, Paolo Frasconi Gianluca Pollastri
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We develop and test new machine learning methods for the predic- tion of topological representations of protein structures in the form of coarse- or (cid:12)ne-grained contact or distance maps that are transla- tion and rotation invariant. The methods are based on generalized input-output hidden Markov models (GIOHMMs) and generalized recursive neural networks (GRNNs). The methods are used to pre- dict topology directly in the (cid:12)ne-grained case and, in the coarse- grained case, indirectly by (cid:12)rst learning how to score candidate graphs and then using the scoring function to search the space of possible con(cid:12)gurations. Computer simulations show that the pre- dictors achieve state-of-the-art performance.
1
Introduction: Protein Topology Prediction
Predicting the 3D structure of protein chains from the linear sequence of amino acids is a fundamental open problem in computational molecular biology [1]. Any approach to the problem must deal with the basic fact that protein structures are translation and rotation invariant. To address this invariance, we have proposed a machine learning approach to protein structure prediction [4] based on the predic- tion of topological representations of proteins, in the form of contact or distance maps. The contact or distance map is a 2D representation of neighborhood rela- tionships consisting of an adjacency matrix at some distance cuto(cid:11) (typically in the range of 6 to 12 (cid:23)A), or a matrix of pairwise Euclidean distances. Fine-grained maps are derived at the amino acid or even atomic level. Coarse maps are obtained by looking at secondary structure elements, such as helices, and the distance between their centers of gravity or, as in the simulations below, the minimal distances be- tween their C(cid:11) atoms. Reasonable methods for reconstructing 3D coordinates from contact/distance maps have been developed in the NMR literature and elsewhere
Neural Control of Sensory Acquisition: The Vestibulo-Ocular Reflex
Authors: Mark E. Nelson, James M. Bower Michael G. Paulin
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present a new hypothesis that the cerebellum plays a key role in ac(cid:173) tively controlling the acquisition of sensory infonnation by the nervous  system.  In this paper we explore this idea by examining the function of  a  simple  cerebellar-related  behavior,  the  vestibula-ocular  reflex  or  VOR, in  which  eye movements  are generated to minimize image slip  on  the  retina  during  rapid  head  movements.  Considering  this  system  from  the point of view of statistical estimation theory, our results  sug(cid:173) gest that the transfer function of the VOR, often regarded as a static or  slowly  modifiable  feature  of the  system,  should  actually  be  continu(cid:173) ously and rapidly changed during head movements. We further suggest  that these changes are under the direct control of the cerebellar cortex  and propose experiments to test this hypothesis.
Modeling the Olfactory Bulb - Coupled Nonlinear Oscillators
Authors: John J. Hopfield Zhaoping Li
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The olfactory  bulb of mammals  aids  in  the  discrimination  of  odors.  A  mathematical  model  based  on  the  bulbar  anatomy  and  electrophysiology  is  described.  Simulations  produce  a  35-60  Hz  modulated activity coherent across the bulb, mimicing the observed  field  potentials.  The  decision  states  (for  the  odor  information)  here  can  be  thought  of as  stable  cycles,  rather  than  point  stable  states  typical  of simpler  neuro-computing  models.  Analysis  and  simulations show that a  group of coupled non-linear oscillators are  responsible for the oscillatory activities determined by the odor in(cid:173) put, and that the bulb, with appropriate inputs from higher centers,  can  enhance  or suppress  the  sensitivity  to  partiCUlar  odors.  The  model provides a framework  in which to understand the transform  between odor input and the bulbar output to olfactory cortex.
Efficient Parallel Learning Algorithms for Neural Networks
Authors: Alberto Sangiovanni-Vincentelli Alan H. Kramer
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Parallelizable optimization techniques are applied to the problem of  learning in feedforward neural networks. In addition to having supe(cid:173) rior convergence properties, optimization techniques such as the Polak(cid:173) Ribiere method are also significantly more efficient than the Back(cid:173) propagation algorithm. These results are based on experiments per(cid:173) formed on small boolean learning problems and the noisy real-valued  learning problem of hand-written character recognition.
The Boltzmann Perceptron Network: A Multi-Layered Feed-Forward Network Equivalent to the Boltzmann Machine
Authors: Allen Gersho Eyal Yair
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The  concept  of  the  stochastic  Boltzmann  machine  (BM)  is  auractive  for  decision  making  and  pattern  classification  purposes  since  the  probability  of  attaining  the network  states  is a  function  of the network energy.  Hence,  the  probability of attaining particular energy minima  may be associated  with  the  probabilities  of  making  certain  decisions  (or  classifications).  However,  because of its stochastic  nature,  the complexity of the BM is fairly  high and  therefore  such  networks  are  not  very  likely  to  be  used  in  practice.  In  this  paper  we  suggest  a  way  to  alleviate  this  drawback  by  converting  the  sto(cid:173) chastic  BM into  a  deterministic  network  which  we  call  the  Boltzmann  Per(cid:173) ceptron  Network  (BPN).  The BPN is functionally  equivalent  to  the  BM but  has  a  feed-forward  structure  and  low  complexity.  No annealing  is required.  The  conditions  under  which  such  a  convmion  is  feasible  are  given.  A  learning  algorithm  for  the  BPN based  on  the  conjugate  gradient  method  is  also provided which is somewhat akin  to the backpropagation algorithm.
Links Between Markov Models and Multilayer Perceptrons
Authors: C. J. Wellekens Hervé Bourlard
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Hidden Markov models are widely used for automatic speech recog(cid:173) nition. They inherently incorporate the sequential character of the  speech signal and are statistically trained. However, the a-priori  choice of the model topology limits their flexibility. Another draw(cid:173) back of these models is their weak discriminating power. Multilayer  perceptrons are now promising tools in the connectionist approach  for classification problems and have already been successfully tested  on speech recognition problems. However, the sequential nature of  the speech signal remains difficult to handle in that kind of ma(cid:173) chine. In this paper, a discriminant hidden Markov model is de(cid:173) fined and it is shown how a particular multilayer perceptron with  contextual and extra feedback input units can be considered as a  general form of such Markov models.
Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment
Authors: Paul Smolensky Michael Mozer
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper proposes a means of using the knowledge in a network to  determine the functionality or relevance of individual units, both for  the purpose of understanding the network's behavior and improving its  performance. The basic idea is to iteratively train the network to a cer(cid:173) tain performance criterion, compute a measure of relevance that identi(cid:173) fies which input or hidden units are most critical to performance, and  automatically trim the least relevant units. This skeletonization tech(cid:173) nique can be used to simplify networks by eliminating units that con(cid:173) vey redundant information; to improve learning performance by first  learning with spare hidden units and then trimming the unnecessary  ones away, thereby constraining generalization; and to understand the  behavior of networks in terms of minimal "rules."
A Passive Shared Element Analog Electrical Cochlea
Authors: Joe Eisenberg, Edwin Lewis David Feld
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present a  simplified model  of the  micromechanics of the human  cochlea,  realized  with  electrical  elements.  Simulation  of the  model  shows that it retains four signal processing features whose importance  we argue on the basis of engineering logic and evolutionary evidence.  Furthermore, just as  the cochlea does,  the  model  achieves  massively  parallel signal processing in a structurally economic way, by means of  shared elements.  By extracting what we believe are the five essential  features of the cochlea, we hope to design a useful  front-end  filter to  process  acoustic  images and to  obtain  a  better understanding  of the  auditory system.
Speech Production Using A Neural Network with a Cooperative Learning Mechanism
Authors: Akio Tanaka Mitsuo Komura
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  propose  a  new  neural  network  model  and  its  learning  algorithm. The proposed neural network consists of four layers  - input, hidden, output and final output layers. The hidden and  output layers are multiple.  Using the proposed  SICL(Spread  Pattern Information and Cooperative  Learning)  algorithm,  it  is  possible  to  learn  analog  data  accurately  and  to  obtain  smooth outputs. Using this neural network, we have developed  a  speech production system consisting of a  phonemic  symbol  production  subsystem  and  a  speech  parameter  production  subsystem.  We  have  succeeded  in  producing  natural  speech  waves with high accuracy.
Associative Learning via Inhibitory Search
Authors: David H. Ackley
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: ALVIS is  a  reinforcement-based  connectionist  architecture  that  learns  associative  maps  in  continuous  multidimensional  environ(cid:173) ments.  The  discovered  locations  of  positive  and  negative  rein(cid:173) forcements  are  recorded  in  "do be"  and  "don't  be"  subnetworks,  respectively.  The outputs of the subnetworks relevant  to the cur(cid:173) rent goal are combined and compared with the current location to  produce  an  error  vector.  This  vector  is  backpropagated  through  a  motor-perceptual  mapping  network.  to  produce  an  action  vec(cid:173) tor that leads the system towards do-be locations  and  away from  don 't-be locations.  AL VIS is  demonstrated with a simulated robot  posed a  target-seeking task.
Comparing Biases for Minimal Network Construction with Back-Propagation
Authors: Lorien Y. Pratt Stephen Jose Hanson
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: learning 
representations during 
Rumelhart (1987). has proposed a method for choosing minimal or  "simple"  in Back-propagation  networks. This approach can be used to (a) dynamically select the  number of hidden units. (b) construct a representation that is  appropriate for the problem and (c) thus improve the generalization  ability of Back-propagation networks. The method Rumelhart suggests  involves adding penalty terms to the usual error function. In this paper  we introduce Rumelhart·s minimal networks idea and compare two  possible biases on the weight search space. These biases are compared  in both simple counting problems and a speech recognition problem.  In general. the constrained search does seem to minimize the number of  hidden units required with an expected increase in local minima.
What Size Net Gives Valid Generalization?
Authors: David Haussler Eric B. Baum
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  address  the  question  of when  a  network  can  be  expected  to  generalize from m  random training examples chosen from some ar(cid:173) bitrary probability distribution, assuming that future test examples  are drawn from  the same  distribution.  Among  our  results are  the  following  bounds on appropriate sample vs.  network size.  Assume  o <  £  $  1/8.  We  show  that  if m  >  O( ~log~) random  exam(cid:173) ples  can  be  loaded  on  a  feedforward  network  of linear  threshold  functions  with N  nodes and W  weights,  so that at least a  fraction  1 - t of the examples  are  correctly  classified,  then one  has confi(cid:173) dence approaching certainty that the network will correctly classify  a  fraction  1 - £  of future  test  examples drawn from  the same  dis(cid:173) tribution.  Conversely,  for  fully-connected  feedforward  nets  with  one  hidden  layer,  any learning  algorithm  using  fewer  than  O( '!')  random training examples  will,  for  some distributions of examples  consistent  with  an  appropriate  weight  choice,  fail  at  least  some  fixed fraction of the time to find  a  weight choice that will correctly  classify more  than a 1 - £  fraction of the future  test examples.
A Low-Power CMOS Circuit Which Emulates Temporal Electrical Properties of Neurons
Authors: Clint S. Cole Jack L. Meador
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This  paper  describes  a  CMOS  artificial  neuron.  The  circuit  is 
directly  derived  from  the  voltage-gated  channel  model  of  neural 
membrane,  has  low  power  dissipation,  and  small  layout  geometry. 
The principal motivations behind this work include a desire for high 
performance,  more  accurate  neuron  emulation,  and  the  need  for 
higher density in practical neural network implementations.
Applications of Error Back-Propagation to Phonetic Classification
Authors: Victor W. Zue Hong C. Leung
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper is concerced with the use of error back-propagation  in phonetic classification. Our objective is to investigate the ba(cid:173) sic characteristics of back-propagation, and study how the frame(cid:173) work of multi-layer perceptrons can be exploited in phonetic recog(cid:173) nition. We explore issues such as integration of heterogeneous  sources of information, conditioll~ that can affect performance of  phonetic classification, internal representations, comparisons with  traditional pattern classification techniques, comparisons of differ(cid:173) ent error metrics, and initialization of the network. Our investiga(cid:173) tion is performed within a set of experiments that attempts to rec(cid:173) ognize the 16 vowels in American English independent of speaker.  Our results are comparable to human performance. 
Early approaches in phonetic recognition fall into two major extremes: heuristic  and algorithmic. Both approaches have their own merits and shortcomings. The  heuristic approach has the intuitive appeal that it focuses on the linguistic informa(cid:173) tion in the speech signal and exploits acoustic-phonetic knowledge. HO'fever, the  weak control strategy used for utilizing our knowledge has been grossly inadequate.  At the other extreme, the algorithmic approach relies primarily on the powerful con(cid:173) trol strategy offered by well-formulated pattern recognition techniques. However,  relatively little is known about how our speech knowledge accumulated over the  past few decades can be incorporated into the well-formulated algorithms. We feel  that artificial neural networks (ANN) have some characteristics that can potentially  enable them to bridge the gap between these two extremes. On the one hand, our  speech knowledge can provide guidance to the structure and design of the network.  On the other hand, the self-organizing mechanism of ANN can provide a control  strategy for utilizing our knowledge. 
In this paper, we extend our earlier work on the use of artificial neural networks  for phonetic recognition [2]. Specifically, we focus our investigation on the following  sets of issues. First, we describe the use of the network to integrate heterogeneous  sources of information. We will see how classification performance improves as more 
Error Back-Propagation to Phonetic Classification 
207 
information is available. Second, we discuss several important factors that can sub(cid:173) stantially affect the performance of phonetic classification. Third, we examine the  internal representation of the network. Fourth, we compare the network with two  traditional classification techniques: K-nearest neighbor and Gaussian classifica(cid:173) tion. Finally, we discuss our specific implementations of back-propagation that  yield improved performance and more efficient learning time.
An Analog Self-Organizing Neural Network Chip
Authors: Sheldon Gilbert James R. Mann
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A design for a fully analog version of a self-organizing feature map neural  network has been completed. Several parts of this design are in fabrication.  The feature map algorithm was modified to accommodate circuit solutions  to the various computations required. Performance effects were measured  by simulating the design as part of a frontend for a speech recognition  system. Circuits are included to implement both activation computations and  weight adaption 'or learning. External access to the analog weight values is  provided to facilitate weight initialization, testing and static storage. This  fully analog implementation requires an order of magnitude less area than  a comparable digital/analog hybrid version developed earlier.
A Bifurcation Theory Approach to the Programming of Periodic Attractors in Network Models of Olfactory Cortex
Authors: Bill Baird
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  new  learning  algorithm  for  the  storage  of  static  and  periodic  attractors  in  biologically  inspired  recurrent  analog  neural  networks  is  introduced.  For  a  network  of  n  nodes,  n  static  or  n/2  periodic  attractors  may  be  stored.  The  algorithm  allows  programming  of  the  network  vector  field  indepen(cid:173) dent  of  the  patterns  to  be  stored.  Stability  of  patterns,  basin  geometry,  and  rates  of  convergence  may  be  controlled.  For  orthonormal  patterns,  the  l~grning operation  reduces  to  a  kind  of  periodic  outer  product  rule  that  allows  local,  additive,  commutative,  incremental  learning.  Standing  or  traveling  wave  cycles  may  be  stored  to  mimic  the  kind  of  oscillating  spatial  patterns  that  appear  in  the  neural  activity  of  the  olfactory  bulb  and  prepyriform  cortex  during  inspiration  and  suffice,  in  the  bulb,  to  predict  the  pattern  recognition  behavior  of  rabbits  in  classical  conditioning  ex(cid:173) periments.  These  attractors  arise,  during  simulat(cid:173) ed  inspiration,  through  a  multiple  Hopf  bifurca(cid:173) tion,  which  can  act  as  a  critical  "decision  pOint"  for  their  selection  by  a  very  small  input  pattern.
Heterogeneous Neural Networks for Adaptive Behavior in Dynamic Environments
Authors: Hillel J. Chiel, Leon S. Sterling Randall D. Beer
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Research  in artificial neural networks has genera1ly emphasized  homogeneous architectures. In contrast, the nervous systems of natural  animals exhibit great heterogeneity in both their elements and patterns  of interconnection. This heterogeneity is crucial to the flexible  generation of behavior which is essential for survival in a complex,  dynamic environment. It may also provide powerful insights into the  design of artificial neural networks.  In this paper, we describe a  heterogeneous neural network for controlling  the wa1king of a  simulated insect. This controller is inspired by the neuroethological  It exhibits a  and neurobiological literature on insect locomotion.  variety of statically stable gaits at different speeds simply by varying  the tonic activity of a single cell. It can also adapt to perturbations as a  natural consequence of its design.
Programmable Analog Pulse-Firing Neural Networks
Authors: Alan F. Murray, Lionel Tarassenko Alister Hamilton
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  describe  pulse  - stream  firing  integrated  circuits  that  imple(cid:173) ment asynchronous analog neural networks.  Synaptic weights are  stored  dynamically,  and  weighting  uses  time-division  of  the  neural  pulses  from  a  signalling  neuron  to  a  receiving  neuron.  MOS  transistors  in  their  "ON"  state  act  as  variable  resistors  to  control  a  capacitive  discharge,  and  time-division  is  thus  achieved  by  a  small  synapse  circuit  cell.  The  VLSI  chip  set  design  uses  2.5J.1.m  CMOS technology.
An Analog VLSI Chip for Thin-Plate Surface Interpolation
Authors: John G. Harris
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Reconstructing a surface from sparse sensory data is a well-known  problem iIi computer vision. This paper describes an experimental  analog VLSI chip for smooth surface interpolation from sparse depth  data. An eight-node ID network was designed in 3J.lm CMOS and  successfully tested. The network minimizes a second-order or "thin(cid:173) plate" energy of the surface. The circuit directly implements the cou(cid:173) pled depth/slope model of surface reconstruction (Harris, 1987). In  addition, this chip can provide Gaussian-like smoothing of images.
Simulation and Measurement of the Electric Fields Generated by Weakly Electric Fish
Authors: Christopher Assad, Mark E. Nelson, James M. Bower Brian Rasnow
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The weakly electric fish, Gnathonemus peters;;, explores its environment by gener(cid:173) ating pulsed elecbic fields  and detecting small pertwbations in the fields  resulting from  nearby objects.  Accordingly, the fISh  detects and discriminates objects on  the basis of a  sequence of elecbic "images" whose temporal and spatial properties depend on  the  tim(cid:173) ing of the fish's electric organ discharge and its body position relative to objects in its en(cid:173) vironmenl  We are interested in investigating how these fish utilize timing and body-po(cid:173) sition during exploration to aid in object discrimination.  We have developed a fmite-ele(cid:173) ment simulation of the fish's self-generated electric  fields  so as  to  reconstruct the elec(cid:173) trosensory consequences of body position and electric organ discharge timing in the fish.  This paper describes this finite-element simulation system and presents preliminary elec(cid:173) tric field measurements which are being used to tune the simulation.
Training Multilayer Perceptrons with the Extended Kalman Algorithm
Authors: Lance Wu Sharad Singhal
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: trained with 
A large fraction of recent work in artificial neural nets uses  multilayer perceptrons  the back-propagation  algorithm described by Rumelhart et. a1. This algorithm  converges slowly for large or complex problems such as  speech recognition, where thousands of iterations may be  needed for convergence even with small data sets. In this  paper, we show that training multilayer perceptrons is an  identification problem for a nonlinear dynamic system which  can be solved using  the Extended Kalman Algorithm.  Although computationally complex, the Kalman algorithm  usually converges in a few  the  algorithm and compare it with back-propagation using two(cid:173) dimensional examples. 
iterations. We describe
An Electronic Photoreceptor Sensitive to Small Changes in Intensity
Authors: C. A. Mead Tobi Delbrück
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We describe an electronic photoreceptor circuit that is sensitive to  small changes in incident light intensity. The sensitivity to change8  in the intensity is achieved by feeding back to the input a filtered  version of the output. The feedback loop includes a hysteretic el(cid:173) ement. The circuit behaves in a manner reminiscent of the gain  control properties and temporal responses of a variety of retinal  cells, particularly retinal bipolar cells. We compare the thresholds  for detection of intensity increments by a human and by the cir(cid:173) cuit. Both obey Weber's law and for both the temporal contrast  sensitivities are nearly identical. 
We previously described an electronic photoreceptor that outputs a voltage that is  logarithmic in the light intensity (Mead, 1985). This report describes an extension  of this circuit which was based on a suggestion by Frank Werblin that biological  retinas may achieve greater sensitivity to change8 in the illumination by feeding  back a filtered version of the output. 
OPERATION OF THE CIRCUIT 
The circuit (Figure 1) consists of a phototransistor (P), exponential feedback to  P (Ql, Q2, and Q3), a transconductance amplifier (A), and the hysteretic element  (Q4 and Qs). In general terms the operation of the circuit consists of two stages of  amplification with hysteresis in the feedback loop. The light falls on the parasitic  bipolar transistor P. (The rest of the circuit is shielded by metal.) P's collector  is the substrate and the base is an isolated well. P and Ql form the first stage of  amplification. The light produces a base current Is for P. The emitter current IE  is PIs, neglecting collector resistance for now. P is typically a few hundred. The  feedback current IQl is set by the gate voltage on QdQ2' which is set by the current  through Q3, which is set by the feedback voltage Vjb. In equilibrium Vjb will be  such that IQl = IE and some voltage Vp will be the output of the first stage. The 
An Electronic Photoreceptor Sensitive to Small Changes 
721 
negative feedback through the transconductance amplifier A will make Vp ~ V/ b•  This voltage is logarithmic in the light intensity, since in subthreshold operation  the currents through Q2 and Q3 are exponential in their gate to source voltages.  The DC output of the circuit will be Vout ~ V/b = Vdd - (2kT /q) log IE, neglecting  the back-gate effect for Q2. Figure Sa (DC output) shows that the assumption of  subthreshold operation is valid over about 4 orders of magnitude.
Training a 3-Node Neural Network is NP-Complete
Authors: Ronald L. Rivest Avrim Blum
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We consider  a  2-layer,  3-node,  n-input neural network whose  nodes  compute linear threshold functions  of their inputs.  We  show  that it  is NP-complete to decide whether there exist weights and thresholds  for the three nodes of this network so that it will produce output con(cid:173) sistent  with  a  given set of training examples.  We  extend  the result  to other simple networks.  This result suggests that those looking for  perfect  training  algorithms  cannot  escape  inherent  computational  difficulties just by  considering only simple or very  regular networks.  It also suggests the importance, given a training problem, of finding  an  appropriate network  and input encoding for  that problem.  It is  left as an open problem to extend our result to nodes with non-linear  functions such as  sigmoids.
Automatic Local Annealing
Authors: Jared Leinbach
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This research involves a method for finding global maxima  in  constraint  satisfaction  networks.  It  is  an  annealing  process  butt  unlike  most  otherst  requires  no  annealing  schedule.  Temperature  is  instead  determined  locally  by  units at each updatet and thus all processing is done at the  unit  level.  There  are  two  major  practical  benefits  to  processing  this  way:  1)  processing  can continue  in  'bad t  areas of the networkt while 'good t areas remain stablet and  2)  processing  continues  in  the  'bad t  areast as  long  as  the  constraints  remain  poorly  satisfied  (i.e.  it  does  not  stop  after  some  predetermined  number of cycles).  As a  resultt  this  method  not  only  avoids  the  kludge  of requiring  an  externally determined annealing schedulet but it also finds  global  maxima  more  quickly  and  consistently  than  externally  scheduled  systems  the  to  Boltzmann machine (Ackley et alt 1985) is made).  FinallYt  implementation of this method is computationally trivial.
Adaptive Neural Net Preprocessing for Signal Detection in Non-Gaussian Noise
Authors: Paul Beckman Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  nonlinearity  is  required  before  matched  filtering  in  mInimum  error  receivers  when  additive  noise  is  present  which  is  impulsive  and  highly  non-Gaussian.  Experiments  were  performed  to  determine  whether  the  correct clipping  nonlinearity  could  be provided  by  a  single-input  single(cid:173) output  multi-layer  perceptron  trained  with  back  propagation.  It  was  found  that a  multi-layer perceptron with one input and output node,  20  nodes  in  the  first  hidden  layer,  and  5  nodes  in  the  second  hidden  layer  could be trained to provide a  clipping nonlinearity with fewer than 5,000  presentations  of noiseless  and  corrupted  waveform  samples.  A  network  trained  at  a  relatively  high  signal-to-noise  (SIN)  ratio  and  then  used  as  a  front  end  for  a  linear  matched  filter  detector greatly  reduced  the  probability  of error.  The  clipping  nonlinearity  formed  by  this  network  was similar to that used in current receivers designed for impulsive  noise  and  provided  similar substantial  improvements in  performance.
Learning the Solution to the Aperture Problem for Pattern Motion with a Hebb Rule
Authors: Martin I. Sereno
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: to 
The  primate  visual  system  learns  to  recognize  the  true  direction  of  pattern  motion  using  local  detectors  only  capable  of  detecting  the  component  of  motion  perpendicular  the  orientation  of  the  moving  edge.  A  multilayer  feedforward  network  model  similar  to  Linsker's  model  was  presented  with  input  patterns  each  consisting  of  randomly  oriented  contours  moving  in  a  particular  direction.  Input  layer  units  are  granted  component  direction  and  speed  tuning  curves  similar  to  those  recorded  from  neurons  in  primate  visual  area  VI  that  project  to  area  MT.  The  network  is  trained  on  many  such  patterns  until  most  weights  saturate.  A  proportion  of  the  units  in  the  second  layer  solve  the  aperture  problem  (e.g.,  show  the  to  gratings),  same  direction-tuning  curve  peak  resembling  pattern-direction  selective  neurons,  which  ftrst  appear  inareaMT.
GENESIS: A System for Simulating Neural Networks
Authors: Upinder S. Bhalla, John D. Uhley, James M. Bower Matthew A. Wilson
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: support  simulations  at  many  it  is 
We  have  developed  a  graphically  oriented,  general  purpose  simulation  system  to  facilitate  the  modeling  of  neural  networks.  The  simulator  is  implemented  under  UNIX  and  X-windows  and  is  levels  of  detail.  designed  to  in  both  applied  network  Specifically,  modeling  and  in  the  simulation  of  detailed,  realistic,  biologically(cid:173) based  models.  Examples  of  current  models  developed  under  this  system  include  mammalian  olfactory  bulb  and  cortex,  invertebrate  central  pattern  generators,  as  well  as  more  abstract  connectionist  simulations. 
intended  for  use
Learning by Choice of Internal Representations
Authors: Ronny Meir, Eytan Domany Tal Grossman
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  introduce  a  learning algorithm for  multilayer neural  net(cid:173) works  composed of binary linear threshold elements.  Whereas ex(cid:173) isting algorithms reduce  the learning process  to minimizing a  cost  function  over  the  weights,  our  method  treats  the  internal  repre(cid:173) sentations  as  the fundamental entities  to  be  determined.  Once  a  correct set of internal representations is  arrived at, the weights are  found  by  the  local  aild  biologically plausible Perceptron  Learning  Rule  (PLR).  We tested  our  learning algorithm on  four  problems:  adjacency, symmetry, parity and  combined symmetry-parity.
Adaptive Neural Networks Using MOS Charge Storage
Authors: R. E. Howard, Wayne E. Hubbard Daniel B. Schwartz
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: MOS charge storage has been demonstrated as an effective method to store  the  weights  in  VLSI  implementations  of  neural  network  models  by  several  workers  2 .  However,  to  achieve  the  full  power  of a  VLSI  implementation  of  an adaptive algorithm, the learning operation must built into the circuit.  We  have  fabricated  and  tested  a  circuit  ideal  for  this  purpose  by  connecting  a  pair of capacitors with  a  CCD like structure, allowing for  variable size  weight  changes  as  well  as  a  weight  decay  operation.  A  2.51-'  CMOS  version  achieves  better  than  10  bits  of dynamic  range  in  a  140/'  X  3501-'  area.  A  1.25/'  chip  based  upon  the  same  cell  has  1104  weights  on  a  3.5mm  x  6.0mm  die  and  is  capable of peak learning rates  of at least  2  x  109  weight  changes  per second. 
1  Adaptive  Networks 
Much  of the  recent  excitement  about  neural  network  models  of computation  has  been  driven  by  the  prospect  of new  architectures  for  fine  grained  parallel  compu(cid:173) tation using analog VLSI.  Adaptive systems are espescially good targets for  analog  VLSI because the ada.ptive process  can compensate for  the inaccuracy of individual  devices  as easily as for  the variability of the signal.  However, silicon  VLSI  does  not  provide  us  with  an  ideal solution  for  weight  storage.  Among  the  properties  of an  ideal storage technology for  analog VLSI  adaptive systems  are: 
•  The minimum available weight change ~w must be small.  The simplest adap(cid:173)
tive  algorithms optimize  the  weights  by  minimizing  the  output  error  with  a  steepest  descent  search  in weight space  [1].  Iterative improvement algorithms  such  as  steepest  descent  are  based  on  the  heuristic  assumption  of  'better'  weights being found  in  the neighborhood of 'good' ones;  a  heuristic that fails  when  the granularity of the weights is  not fine  enough.  In the worst  case,  the  resolution required just to represent  a  function  can  grow exponentially in  the  dimension of the input space . 
•  The  weights  must be able  to represent  both  positive and negative values  and  the  changes  must  be easily  reversible.  Frequently,  the  weights  may cycle  up  and down while the adaptive process is converging and millions of incremental  changes  during  a  single  training session  is  not  unreasonable.  If the  weights  cannot easily  follow  all  of these  changes,  then  the  learning  must  be  done  off  chip. 
1 Now  at GTE Laboratories, 40 Sylvan Rd., Waltham, Mass 02254  dbs@gte.com%relay.cs.net  2For example, see  the  papers by Mann and Gilbert, Walker and Akers,  and Murray  et.  al.  in
Implications of Recursive Distributed Representations
Authors: Jordan B. Pollack
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: I  will  describe  my  recent  results  on  the  automatic  development  of fixed(cid:173) width recursive  distributed representations  of variable-sized  hierarchal data  structures.  One  implication  of this  wolk  is  that  certain  types  of AI-style  data-structures can now be represented in fixed-width analog vectors. Simple  inferences  can  be  perfonned  using  the  type  of pattern  associations  that  neural  networks excel  at  Another implication arises from  noting that these  representations  become  self-similar in  the  limit Once  this door to  chaos is  opened.  many  interesting new  questions  about  the  representational  basis  of  intelligence emerge, and can (and will) be discussed.
Backpropagation and Its Application to Handwritten Signature Verification
Authors: Dorothy A. Mighell, Joseph W. Goodman Timothy S. Wilkinson
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  pool  of handwritten  signatures  is  used  to  train  a  neural  net(cid:173) work for the task of deciding whether or not a  given signature is a  forgery.  The network is  a feedforward  net, with a binary image as  input.  There is a hidden layer, with a single unit output layer.  The  weights are  adjusted according to the backpropagation algorithm.  The signatures are entered into a C  software program through the  use of a Datacopy Electronic Digitizing Camera.  The binary signa(cid:173) tures  are normalized  and centered.  The performance is  examined  as  a  function of the training set  and network structure.  The  best  scores  are  on  the  order of 2%  true signature  rejection  with  2-4%  false  signature acceptance.
Theory of Self-Organization of Cortical Maps
Authors: Shigeru Tanaka
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  have  mathematically  shown  that  cortical  maps  in  the  primary sensory  cortices  can  be  reproduced  by  using  three  hypotheses  which  have  physiological  basis  and  meaning.  Here, our main focus is on ocular.dominance column formation  in the primary visual cortex.  Monte Carlo simulations on the  segregation of ipsilateral and contralateral afferent terminals  are carried out.  Based on  these,  we  show that almost all  the  physiological  experimental  results  concerning  the  ocular  dominance patterns of cats and monkeys reared under normal  or various abnormal visual conditions can be explained from a  viewpoint of the phase transition phenomena. 
ROUGH SKETCH OF OUR THEORY 
In order to describe the use-dependent self-organization of neural connections  {Singer,1987 and Frank,1987},  we  have  proposed  a  set of coupled  equations  involving  the  electrical  activities  and  neural  connection  density  {Tanaka,  1988}, by using the following physiologically based hypotheses: (1) Modifiable  synapses grow or collapse due to  the competition among themselves for  some  trophic factors,  which are secreted retrogradely from  the postsynaptic side to  the  presynaptic  side.  (2)  Synapses  also  sprout or  retract  according  to  the  concurrence  of presynaptic  spike  activity  and  postsynaptic  local  membrane  depolarization.  (3)  There already exist lateral connections within the  layer,  into  which  the  modifiable  nerve  fibers  are  destined  to  project,  before  the  synaptic modification begins.  Considering this set of equations, we  find  that  the  time  scale  of electrical  activities  is  much  smaller  than  time  course  necessary  for  synapses  to  grow  or  retract.  So  we  can  apply  the  adiabatic  approximation to the equations.  Furthermore, we identify the input electrical  activities,  i.e.,  the  firing  frequency  elicited  from  neurons  in  the  projecting  neuronal layer, with the stochastic process which is specialized by the spatial  correlation function  Ckp;k'  p'.  Here,  k  and  k'  represent  the  positions  of the  neurons  in  the  projecting  layer.  II  stands  for  different  pathways  such  as  ipsilateral  or  contralateral,  on-center  or  off-center,  colour  specific  or  nonspecific  and  so  on.  From  these  approximations,  we  have  a  nonlinear
An Adaptive Network That Learns Sequences of Transitions
Authors: C. L. Winter
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We describe  an  adaptive  network,  TIN2,  that learns  the  transition  function of a sequential system from  observations of its behavior.  It  integrates two subnets, TIN-I  (Winter, Ryan and Turner,  1987) and  TIN-2.  TIN-2  constructs  state  representations  from  examples  of  system behavior, and  its  dynamics are the main  topics of the paper.  TIN-I abstracts transition functions from  noisy state representations  and environmental data during training, while in operation it produces  sequences of transitions in response to variations in input.  Dynamics  of both nets are based on the Adaptive Resonance Theory of Carpenter  and Grossberg (1987).  We give results from an experiment in which  TIN2 learned the behavior of a system that recognizes strings with an  even number of l's .
Dynamics of Analog Neural Networks with Time Delay
Authors: R. M. Westervelt Charles M. Marcus
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A time delay in the response of the neurons in a network can  induce sustained oscillation and chaos. We present a stability  criterion based on local stability analysis to prevent sustained  oscillation  in  symmetric  delay  networks,  and  show  an  example  of chaotic  dynamics  in  a  non-symmetric  delay  network.
On the K-Winners-Take-All Network
Authors: Ruth Erlanson, Yaser S. Abu-Mostafa E. Majani
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We present  and  rigorously  analyze a generalization of the Winner(cid:173) Take-All  Network:  the  K-Winners-Take-All  Network.  This  net(cid:173) work  identifies  the  K  largest  of a  set  of N  real  numbers.  The  network  model used  is  the continuous Hopfield  model. 
I  - INTRODUCTION 
The Winner-Take-All  Network  is  a  network  which  identifies  the  largest  of N  real  numbers.  Winner-Take-All  Networks  have  been  developed  using  various  neural  networks models (Grossberg-73, Lippman-87, Feldman-82, Lazzaro-89).  We present  here  a  generalization  of the  Winner-Take-All  Network:  the  K-Winners-Take-All  (KWTA) Network.  The KWTA Network identifies the K  largest of N  real numbers.  The neural network model we  use throughout the paper is  the continuous Hopfield  network  model  (Hopfield-84).  If the states of the  N  nodes are initialized  to the N  real numbers, then, if the gain of the sigmoid is large enough, the network converges  to the state with  K  positive real numbers in  the positions of the nodes with  the K  largest  initial states, and  N  - K  negative  real  numbers everywhere else.  Consider  the  following  example:  N  = 4,  K  = 2.  There  are  6  =  (~)  stable  states:(++-)T, (++)T, (+--+)T, ( _ ++)T, (++)T, and (++)T.  If the initial  state of the  network  is  (0.3,  -0.4,  0.7,  O.l)T,  then  the network  will  converge to (Vi,V2,V3,v4)T  where Vi> 0,  V2  < 0,  V3  > 0,  V4  < 0  ((+ _ +_)T).  In Section II, we  define  the KWTA  Network (connection weights, external inputs).  In Section III,  we  analyze  the equilibrium states and  in  Section  IV,  we  identify  all  the stable equilibrium states of the KWTA Network.  In Section V, we  describe the  dynamics  of the  KWTA  Network.  In  Section  VI,  we  give  two  important examples  of the KWTA  Network and comment on an alternate implementation of the KWTA  Network. 
On the K-Winners-Take-All Network 
635 
II - THE K-WINNERS-TAKE-ALL  NETWORK 
The continuous Hopfield network model (Hopfield-84)  (also known as the Grossberg  additive model  (Grossberg-88)), is characterized by a system of first order differen(cid:173) tial equations which governs the evolution of the state of the network (i = 1, .. . , N) : 
The  sigmoid  function  g(u)  is  defined  by:  g(u)  = f(G·  u),  where  G  >  0  is  the  gain  of  the  sigmoid,  and  f(u)  is  defined  by:  1.  "f/u,  0  <  f'(u)  <  f'(O)  = 1,  2.  limu .... +oo  f( u) = 1,  3.  limu .... -oo f( u) = -l.  The KWTA Network is characterized by mutually inhibitory interconnections Taj  =  -1  for  i  ¥=  j, a self connection Tai  = a,  (Ial < 1) and'an external input (identical  for  every node) which  depends on the number  K  of winners desired and the size of  the network  N  : ti = 2K - N.  The differential equations for  the KWTA  Network are therefore: 
for  all  i,  Cd~i = -Aui + (a + l)g(ui) - (tg(u j )  - t) ,
Does the Neuron "Learn" like the Synapse?
Authors: Raoul Tawel
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: An improved learning paradigm that offers a significant reduction in com(cid:173)
putation time during the supervised learning phase is described. 
It is based on 
extending the role that the neuron plays in artificial neural systems. Prior work 
has regarded the neuron as a strictly passive, non-linear processing element, and 
the synapse on the other hand as the primary source of information processing and 
knowledge retention. In this work, the role of the neuron is extended insofar as allow(cid:173)
ing its parameters to adaptively participate in the learning phase. The temperature 
of the sigmoid function is an example of such a parameter. During learning, both the 
synaptic interconnection weights w[j and the neuronal temperatures Tr are opti(cid:173)
mized so as to capture the knowledge contained within the training set. The method 
allows each neuron to possess and update its own characteristic local temperature. 
This algorithm has been applied to logic type of problems such as the XOR or parity 
problem, resulting in a significant decrease in the required number of training cycles.
Use of Multi-Layered Networks for Coding Speech with Phonetic Features
Authors: Régis Cardin, Renato de Mori, Piero Cosi Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Preliminary  results  on  speaker-independant  speech  recognition  are  reported.  A method  that combines  expertise  on  neural  networks  with  expertise  on  speech  recognition  is  used  to  build  the  recognition  systems.  For  transient  sounds,  event(cid:173) driven  property  extractors  with  variable  resolution  in  the  time  and  frequency  domains  are  used.  For  sonorant  speech,  a  model  of the  human  auditory  system  is  preferred  to  FFT  as  a  front-end  module.