mtybilly/Instruct2DS huggingface dataset:
PCA-Pyramids for Image Compression
Authors: Horst Bischof, Kurt Hornik
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper presents a new  method for image compression by neural  networks.  First, we  show  that we  can use neural networks in  a  py(cid:173) ramidal framework, yielding the so-called PCA pyramids.  Then we  present an image compression method based on the PCA pyramid,  which  is  similar  to  the  Laplace  pyramid  and  wavelet  transform.  Some experimental results  with  real images are reported.  Finally,  we  present  a  method  to  combine  the  quantization  step  with  the  learning of the PCA pyramid.
Morphogenesis of the Lateral Geniculate Nucleus: How Singularities Affect Global Structure
Authors: Svilen Tzonev, Klaus Schulten, Joseph G. Malpeli
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The macaque lateral geniculate nucleus (LGN) exhibits an intricate  lamination pattern, which changes midway through the nucleus at a  point coincident with small gaps due to the blind spot in the retina.  We  present a  three-dimensional model of morphogenesis in  which  local cell interactions cause a  wave  of development of neuronal re(cid:173) ceptive fields  to propagate through the nucleus  and establish two  distinct lamination patterns.  We  examine the interactions between  the wave  and the localized singularities due  to the gaps,  and find  that the gaps induce the change in lamination pattern.  We  explore  critical factors which  determine general LGN organization.
Reinforcement Learning Predicts the Site of Plasticity for Auditory Remapping in the Barn Owl
Authors: Alexandre Pouget, Cedric Deffayet, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The auditory system of the barn owl  contains several spatial maps.  In young barn owls raised with optical prisms over their eyes,  these  auditory  maps are  shifted  to stay  in  register  with the  visual  map,  suggesting  that  the  visual  input  imposes  a  frame  of reference  on  the  auditory  maps.  However,  the  optic  tectum,  the  first  site  of  convergence  of visual  with  auditory information, is  not the  site  of  plasticity for  the  shift  of the  auditory  maps;  the  plasticity occurs  instead  in  the  inferior  colliculus, which  contains  an  auditory  map  and projects into the optic tectum.  We explored a model of the owl  remapping in which  a global reinforcement signal whose  delivery  is  controlled  by  visual foveation.  A hebb  learning rule gated by  rein(cid:173) forcement  learned  to appropriately adjust auditory  maps.  In addi(cid:173) tion,  reinforcement  learning  preferentially  adjusted  the  weights  in  the inferior colliculus,  as in the owl brain, even  though the weights  were  allowed  to  change  throughout  the auditory system.  This ob(cid:173) servation  raises  the  possibility  that  the  site  of learning  does  not  have  to  be  genetically specified,  but  could  be  determined  by  how  the learning procedure  interacts with the network  architecture. 
126 
Alexandre Pouget,  Cedric Deffayet,  Te"ence J.  Sejnowski 
c:::======:::::»  •
A Lagrangian Formulation For Optical Backpropagation Training In Kerr-Type Optical Networks
Authors: James Edward Steck, Steven R. Skinner, Alvaro A. Cruz-Cabrara, Elizabeth C. Behrman
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A training method based on a form of continuous spatially distributed  optical error back-propagation is presented for an all optical network  composed of nondiscrete neurons and weighted interconnections. The all  optical network is feed-forward and is composed of thin layers of a Kerr(cid:173) type self focusing/defocusing nonlinear optical material. The training  method is derived from a Lagrangian formulation of the constrained  minimization of the network error at the output. This leads to a  formulation that describes training as a calculation of the distributed error  of the optical signal at the output which is then reflected back through the  device to assign a spatially distributed error to the internal layers. This  error is then used to modify the internal weighting values. Results from  several computer simulations of the training are presented, and a simple  optical table demonstration of the network is discussed. 
772 
Elizabeth C. Behrman 
1 KERR TYPE MATERIALS 
Kerr-type optical networks utilize thin layers of Kerr-type nonlinear materials, in which the  index of refraction can vary within the material and depends on the amount of light striking  the material at a given location. The material index of refraction can be described by:  n(x)=no+nzI(x), where 110 is the linear index of refraction, ~ is the nonlinear coefficient, and  I(x) is the irradiance of a applied optical field as a function of position x across the material  layer (Armstrong, 1962). This means that a beam of light (a signal beam carrying  information perhaps) passing through a layer of Kerr-type material can be steered or  controlled by another beam of light which applies a spatially varying pattern of intensity  onto the Kerr-type material. Steering of light with a glass lens (having constance index of  refraction) is done by varying the thickness of the lens (the amount of material present) as  a function of position. Thus the Kerr effect can be loosely thought of as a glass lens whose  geometry and therefore focusing ability could be dynamically controlled as a function of  position across the lens. Steering in the Kerr material is accomplished by a gradient or  change in the material index of refraction which is created by a gradient in applied light  intensity. This is illustrated by the simple experiment in Figure 1 where a small weak probe  beam is steered away from a straight path by the intensity gradient of a more powerful pump  beam.
Nonlinear Image Interpolation using Manifold Learning
Authors: Christoph Bregler, Stephen M. Omohundro
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The problem of interpolating between specified images in an image  sequence  is  a  simple,  but  important  task  in  model-based  vision.  We  describe  an  approach based  on  the abstract  task of  "manifold  learning"  and present  results  on  both synthetic and real image se(cid:173) quences.  This  problem  arose  in  the  development  of a  combined  lip-reading and speech  recognition system.
Interior Point Implementations of Alternating Minimization Training
Authors: Michael Lemmon, Peter T. Szymanski
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper presents an alternating minimization (AM) algorithm  used in the training of radial basis function and linear regressor  networks. The algorithm is a modification of a small-step interior  point method used in solving primal linear programs. The algo(cid:173) rithm has a convergence rate of O( fo,L) iterations where n is a  measure of the network size and L is a measure of the resulting  solution's accuracy. Two results are presented that specify how  aggressively the two steps of the AM may be pursued to ensure  convergence of each step of the alternating minimization.
A Convolutional Neural Network Hand Tracker
Authors: Steven J. Nowlan, John C. Platt
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe  a system that can track  a hand in a sequence of video  frames  and recognize hand gestures  in  a user-independent  manner.  The system  locates  the  hand in  each  video  frame  and  determines  if the hand is open or closed.  The tracking system is  able to track  the  hand  to  within  ±10  pixels  of its  correct  location  in  99.7%  of  the frames  from a  test set  containing video sequences  from  18  dif(cid:173) ferent  individuals captured in 18  different  room environments.  The  gesture  recognition network correctly  determines if the hand being  tracked  is  open  or  closed  in  99.1 % of the  frames  in this  test  set .  The system has been designed to operate in real time with existing  hardware.
Temporal Dynamics of Generalization in Neural Networks
Authors: Changfeng Wang, Santosh S. Venkatesh
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper  presents  a  rigorous  characterization  of how  a  general  nonlinear learning machine generalizes  during  the training process  when  it  is  trained  on  a  random  sample  using  a  gradient  descent  algorithm  based  on  reduction  of  training  error.  It is  shown,  in  particular, that best generalization performance occurs,  in general,  before  the global  minimum of the  training error  is  achieved.  The  different  roles  played  by  the  complexity of the  machine  class  and  the  complexity of the  specific  machine in the class  during learning  are  also precisely  demarcated.
Learning Stochastic Perceptrons Under k-Blocking Distributions
Authors: Mario Marchand, Saeed Hadjifaradji
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  statistical  method  that  PAC  learns  the  class  of  stochastic  perceptrons  with  arbitrary  monotonic  activation  func(cid:173) tion and weights Wi  E  {-I, 0, + I} when the probability distribution  that  generates  the input examples  is  member of a  family  that we  call k-blocking distributions.  Such distributions represent an impor(cid:173) tant step beyond the case where each input variable is statistically  independent  since the 2k-blocking  family  contains  all  the  Markov  distributions of order k.  By stochastic  percept ron we  mean  a  per(cid:173) ceptron which,  upon presentation of input vector x, outputs 1 with  probability  fCLJi WiXi  - B).  Because the same algorithm works  for  any  monotonic  (nondecreasing  or  nonincreasing)  activation  func(cid:173) tion  f  on  Boolean  domain,  it  handles  the  well  studied  cases  of  sigmolds  and the  "usual"  radial basis functions.
Unsupervised Classification of 3D Objects from 2D Views
Authors: Satoshi Suzuki, Hiroshi Ando
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper presents an unsupervised learning scheme for categorizing  3D  objects  from  their  2D  projected images.  The  scheme  exploits  an  auto-associative network's ability to encode each view of a single object  into a representation that indicates its view direction.  We propose two  models that employ different classification mechanisms; the first model  selects an auto-associative network whose recovered view best matches  the input view, and the second model is based on a modular architecture  whose  additional  network classifies  the  views  by  splitting  the  input  space  nonlinearly.  We demonstrate  the  effectiveness  of the  proposed  classification models through simulations using 3D wire-frame objects.
Hyperparameters Evidence and Generalisation for an Unrealisable Rule
Authors: Glenn Marion, David Saad
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Using a statistical mechanical formalism we  calculate the evidence,  generalisation  error  and  consistency  measure  for  a  linear  percep(cid:173) tron  trained  and  tested  on  a  set  of examples  generated  by  a  non  linear  teacher.  The teacher  is  said  to be  unrealisable  because  the  student  can  never  model it without error.  Our model allows  us  to  interpolate between the known case of a linear teacher,  and an un(cid:173) realisable, nonlinear teacher.  A comparison of the hyperparameters  which  maximise the evidence  with  those  that optimise the perfor(cid:173) mance measures  reveals  that,  in  the  non-linear  case,  the  evidence  procedure is a misleading guide to optimising performance.  Finally,  we explore the extent to which the evidence procedure is  unreliable  and find  that, despite being sub-optimal, in  some circumstances  it  might be  a  useful  method for  fixing  the hyperparameters.
A Model for Chemosensory Reception
Authors: Rainer Malaka, Thomas Ragg, Martin Hammer
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A new  model for chemosensory reception is presented.  It models reacti(cid:173) ons between odor molecules and receptor proteins and the activation of  second  messenger  by receptor proteins.  The mathematical  formulation  of the  reaction  kinetics  is  transformed  into an  artificial  neural  network  (ANN). The resulting feed-forward network provides a powerful means  for parameter fitting by applying learning algorithms. The weights of the  network corresponding to chemical parameters can be trained by presen(cid:173) ting experimental data.  We demonstrate the simulation capabilities of the  model with experimental data from honey bee chemosensory neurons.  It  can be shown that our model is sufficient to rebuild the observed data and  that simpler models are not able to do this task.
A Real Time Clustering CMOS Neural Engine
Authors: Teresa Serrano-Gotarredona, Bernabé Linares-Barranco, José Luis Huertas
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe  an  analog  VLSI  implementation  of the ARTI  algorithm  (Carpenter, 1987). A prototype chip has been fabricated in a standard low  cost  1.5~m double-metal single-poly CMOS process. It has a die area of  lcm2  and  is  mounted in  a  12O-pins PGA package.  The chip realizes  a  modified version of the original ARTI  architecture.  Such modification  has been shown to preserve all computational properties of the original  algorithm  (Serrano,  1994a),  while  being  more  appropriate  for  VLSI  realizations. The chip implements an ARTI network with 100 F 1 nodes  and 18 F2 nodes. It can therefore cluster 100 binary pixels input patterns  into up to 18 different categories. Modular expansibility of the system is  possible  by  assembling  an  NxM  array  of  chips  without  any  extra  interfacing circuitry, resulting in an F 1 layer with  l00xN nodes, and an  F2  layer  with  18xM nodes.  Pattern  classification  is  performed  in  less  than  1.8~s, which  means  an  equivalent  computing  power of 2.2x109  connections and connection-updates per second. Although internally the  chip is analog in nature, it interfaces to the outside world through digital  signals, thus having a true asynchrounous digital behavior. Experimental  chip  test  results  are  available,  which  have  been  obtained  through  test  equipments for digital chips.
Learning from queries for maximum information gain in imperfectly learnable problems
Authors: Peter Sollich, David Saad
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In  supervised  learning,  learning  from  queries  rather  than  from  random  examples  can  improve  generalization  performance  signif(cid:173) icantly.  We  study  the  performance of query  learning for  problems  where  the  student  cannot learn  the  teacher  perfectly,  which  occur  frequently  in  practice.  As  a  prototypical scenario of this  kind,  we  consider  a  linear  perceptron  student  learning a  binary  perceptron  teacher.  Two kinds of queries for  maximum information gain,  i.e.,  minimum entropy,  are  investigated:  Minimum  student  space  en(cid:173) tropy  (MSSE)  queries,  which  are  appropriate  if the  teacher  space  is  unknown,  and  minimum teacher  space entropy  (MTSE)  queries,  which can be used  if the teacher space is  assumed to be known, but  a student of a simpler form  has deliberately  been  chosen.  We  find  that  for  MSSE  queries,  the  structure  of the  student  space  deter(cid:173) mines  the  efficacy  of query  learning,  whereas  MTSE  queries  lead  to  a  higher  generalization  error  than  random examples,  due  to  a  lack of feedback about the progress of the student in the way queries  are  selected.
Factorial Learning by Clustering Features
Authors: Joshua B. Tenenbaum, Emanuel V. Todorov
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We introduce a novel algorithm for factorial learning, motivated  by segmentation problems in computational vision, in which the  underlying factors correspond to clusters of highly correlated input  features. The algorithm derives from a new kind of competitive  clustering model, in which the cluster generators compete to ex(cid:173) plain each feature of the data set and cooperate to explain each  input example, rather than competing for examples and cooper(cid:173) ating on features, as in traditional clustering algorithms. A natu(cid:173) ral extension of the algorithm recovers hierarchical models of data  generated from multiple unknown categories, each with a differ(cid:173) ent, multiple causal structure. Several simulations demonstrate  the power of this approach.
Generalization in Reinforcement Learning: Safely Approximating the Value Function
Authors: Justin A. Boyan, Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  straightforward  approach  to  the  curse  of  dimensionality  in  re(cid:173) inforcement  learning  and  dynamic  programming  is  to  replace  the  lookup table with a generalizing function approximator such as a neu(cid:173) ral net.  Although this has been successful in the domain of backgam(cid:173) mon,  there  is  no  guarantee  of convergence.  In  this  paper,  we  show  that the combination of dynamic programming and function approx(cid:173) imation  is  not  robust,  and  in  even  very  benign  cases,  may produce  an  entirely  wrong  policy.  We  then  introduce  Grow-Support,  a  new  algorithm which is safe from divergence yet can still reap the benefits  of successful  generalization .
A Rapid Graph-based Method for Arbitrary Transformation-Invariant Pattern Classification
Authors: Alessandro Sperduti, David G. Stork
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  graph-based  method  for  rapid,  accurate  search  through prototypes for  transformation-invariant pattern classifica(cid:173) tion.  Our method has in theory the same recognition accuracy as  other recent  methods  based  on  ''tangent  distance"  [Simard et al.,  1994],  since it uses the same categorization rule.  Nevertheless ours  is  significantly  faster  during  classification  because  far  fewer  tan(cid:173) gent  distances  need  be  computed.  Crucial  to  the success  of our  system  are  1)  a  novel  graph  architecture in  which  transformation  constraints and geometric  relationships  among  prototypes are en(cid:173) coded  during learning,  and 2)  an improved graph search criterion,  used during classification.  These architectural insights are applica(cid:173) ble to a wide range of problem domains.  Here we demonstrate that  on  a  handwriting  recognition task,  a  basic implementation of our  system  requires  less  than  half the  computation  of the  Euclidean  sorting method.
Coarse-to-Fine Image Search Using Neural Networks
Authors: Clay Spence, John C. Pearson, Jim Bergen
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The  efficiency  of image  search  can  be  greatly  improved  by  using  a  coarse-to-fine search strategy with a  multi-resolution image representa(cid:173) tion. However,  if the resolution is so low that the objects have few  dis(cid:173) tinguishing  features,  search  becomes  difficult.  We  show  that  the  performance of search at such low resolutions can be improved by using  context information, i.e., objects visible at low-resolution which are not  the objects of interest but are associated with them. The networks can be  given explicit context information as inputs, or they can learn to detect  the context objects, in which case the user does not have to be aware of  their existence. We also use Integrated Feature Pyramids, which repre(cid:173) sent high-frequency  information  at low  resolutions.  The use of multi(cid:173) resolution search techniques allows us to combine information about the  appearance of the objects on many scales in an efficient way.  A natural  fOlm  of exemplar selection also arises from these  techniques.  We illus(cid:173) trate these ideas by training hierarchical systems of neural  networks to  find clusters of buildings in aerial photographs of farmland.
ICEG Morphology Classification using an Analogue VLSI Neural Network
Authors: Richard Coggins, Marwan A. Jabri, Barry Flower, Stephen Pickard
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: An  analogue  VLSI  neural  network  has  been  designed  and  tested  to perform cardiac morphology classification tasks.  Analogue tech(cid:173) niques were chosen to meet the strict power and area requirements  of an Implantable Cardioverter Defibrillator (ICD) system.  The ro(cid:173) bustness  of the neural network  architecture  reduces  the impact of  noise,  drift  and offsets  inherent  in  analogue approaches.  The net(cid:173) work  is a  10:6:3 multi-layer percept ron  with on  chip  digital weight  storage,  a  bucket  brigade input  to feed  the  Intracardiac  Electro(cid:173) gram  (ICEG)  to  the  network  and  has  a  winner  take  all  circuit  at  the  output.  The  network  was  trained  in  loop  and  included  a  commercial ICD in the signal processing path.  The system has suc(cid:173) cessfully distinguished arrhythmia for different patients with better  than 90%  true positive and true negative detections for  dangerous  rhythms which cannot be detected  by present  ICDs.  The chip  was  implemented in 1.2um CMOS and consumes less than 200n W max(cid:173) imum average power in an area of 2.2  x 2.2mm2.
Prior Knowledge in Support Vector Kernels
Authors: Bernhard Schölkopf, Patrice Simard, Alex J. Smola, Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We explore methods for incorporating prior knowledge about a problem  at hand in Support Vector learning machines.  We  show that both invari(cid:173) ances under group transfonnations and prior knowledge about locality in  images can be incorporated by constructing appropriate kernel functions.
A Revolution: Belief Propagation in Graphs with Cycles
Authors: Brendan J. Frey, David J. C. MacKay
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Until recently, artificial intelligence researchers have frowned  upon  the  application of probability propagation in  Bayesian  belief net(cid:173) works  that  have cycles.  The probability propagation algorithm is  only exact in networks that are cycle-free.  However, it has recently  been  discovered  that the  two  best  error-correcting decoding  algo(cid:173) rithms  are  actually  performing  probability  propagation  in  belief  networks with cycles. 
1  Communicating over a  noisy channel 
Our increasingly wired world demands efficient  methods for  communicating bits of  information  over  physical  channels  that  introduce  errors.  Examples  of real-world  channels  include  twisted-pair telephone  wires,  shielded  cable-TV  wire,  fiber-optic  cable,  deep-space  radio,  terrestrial  radio,  and  indoor  radio.  Engineers  attempt  to  correct  the  errors  introduced  by  the  noise  in  these  channels  through  the  use  of  channel  coding  which  adds  protection  to the information  source,  so  that  some  channel errors  can  be  corrected.  A  popular  model  of a  physical  channel  is  shown  in  Fig.  1.  A vector of K  information bits u  =  (Ut, ... ,UK), Uk  E {O, I}  is  encoded,  and a  vector of N  codeword bits x  =  (Xl! ... ,XN)  is  transmitted into the channel.  Independent  Gaussian noise  with  variance  (12  is  then  added to each codeword bit, 
.. Brendan Frey  is  currently a  Beckman Fellow  at the Beckman Institute for  Advanced 
Science and Technology, University of Illinois  at Urbana-Champaign. 
480 
B.  J  Frey and D.  J.  C. MacKay 
Gaussian noise  with variance (J2 
--U--'~~I~ _ E _ ncoder __ ~1  x  ~ y 
.I~ _ Decoder _ ~--U~.~ 
Figure 1:  A communication system with a channel that adds Gaussian noise to the  transmitted discrete-time sequence. 
producing  the  real-valued  channel  output  vector  y  =  (Y!, ... ,YN).  The  decoder  must  then use  this  received  vector  to  make  a  guess  U at the  original  information  vector.  The  probability  P" (e)  of  bit  error  is  minimized  by  choosing  the  Uk  that  maximizes  P(ukly)  for  k  = 1, ... , K.  The  rate  K/N of a  code  is  the  number  of  information  bits  communicated  per  codeword  bit.  We  will  consider  rate  ~ 1/2  systems in this paper,  where  N  ==  2K.  The simplest rate 1/2 encoder duplicates each information hit:  X2k-l  =  X2k  =  Uk,  k  =  1, ... , K.  The optimal decoder for this repetition code  simply averages together  pairs of noisy channel outputs and then applies  a threshold:
Dynamic Stochastic Synapses as Computational Units
Authors: Wolfgang Maass, Anthony M. Zador
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In  most  neural network models, synapses are treated as static weights  that  change only on  the slow  time scales of learning.  In  fact,  however, synapses  are highly  dynamic,  and show  use-dependent  plasticity over a  wide  range  of time scales.  Moreover,  synaptic transmission is  an  inherently stochastic  process:  a  spike  arriving  at  a  presynaptic  terminal  triggers  release  of a  vesicle  of neurotransmitter from  a  release  site  with  a  probability that can  be  much  less  than one.  Changes in  release probability represent one of the  main mechanisms by which synaptic efficacy is modulated in neural circuits.  We propose and investigate a simple model for dynamic stochastic synapses  that can  easily be  integrated into common models  for  neural computation.  We  show  through  computer  simulations  and  rigorous  theoretical  analysis  that  this  model  for  a  dynamic  stochastic synapse increases  computational  power in a nontrivial way.  Our results may have implications for the process(cid:173) ing of time-varying signals by both biological and artificial neural networks. 
A synapse 8  carries out computations on spike trains, more precisely on  trains of spikes  from  the  presynaptic  neuron.  Each  spike  from  the  presynaptic  neuron  mayor may  not  trigger the release  of a  neurotransmitter-filled  vesicle  at the synapse.  The  probability of a  vesicle  release ranges from  about  0.01  to almost  1.  Furthermore this  release probability is  known  to  be strongly  "history dependent"  [Dobrunz and  Stevens, 1997].  A spike causes an  excitatory or inhibitory potential  (EPSP or IPSP, respectively)  in  the postsynaptic  neuron  only when a  vesicle  is  released. 
A spike  train is  represented  as a sequence 1 of firing  times,  i.e.  as increasing sequences  of numbers tl < t2  < ... from  R+ := {z E R: z ~ O}  . For each  spike train 1 the output of  synapse  8  consists of the  sequence 8W  of those  ti  E 10n which  vesicles  are  "released"  by  8  , i.e.  of those t, E 1 which cause an excitatory or inhibitory postsynaptic potential (EPSP  or IPSP,  respectively).  The map 1 -+  8(1)  may  be  viewed  as  a  stochastic function  that is  computed by  synapse S.  Alternatively one can  characterize  the  output  SW  of a  synapse  8  through its  release  pattern q = qlq2 ... E  {R, F}·  , where  R  stands for  release and  F  for  failure  of release.  For each  t, E 1 one sets q,  =  R  if ti  E 8(1)  , and qi  =  F  if ti  ¢ 8W  . 
Dynamic Stochastic Synapses as Computational Units 
195
Self-similarity Properties of Natural Images
Authors: Antonio Turiel, Germán Mato, Néstor Parga, Jean-Pierre Nadal
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Scale  invariance  is  a  fundamental  property  of ensembles  of  nat(cid:173) ural  images  [1].  Their  non  Gaussian  properties  [15,  16]  are  less  well  understood,  but  they  indicate  the  existence  of  a  rich  statis(cid:173) tical  structure.  In  this  work  we  present  a  detailed  study  of the  marginal statistics of a  variable related to the edges in the images.  A numerical analysis shows that it exhibits extended self-similarity  [3,  4,  5].  This  is  a  scaling  property  stronger  than  self-similarity:  all its moments can be expressed as  a power of any given moment.  More  interesting,  all  the  exponents  can  be  predicted  in  terms  of  a  multiplicative  log-Poisson process.  This  is  the very  same model  that  was  used  very  recently  to  predict  the  correct  exponents  of  the structure functions  of turbulent flows  [6].  These  results  allow  us to study the underlying multifractal singularities.  In  particular  we  find  that the most singular structures are one-dimensional:  the  most singular manifold  consists of sharp edges. 
Category:  Visual Processing.
Multiple Threshold Neural Logic
Authors: Vasken Bohossian, Jehoshua Bruck
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We introduce a new Boolean computing element related to the Lin(cid:173) ear Threshold element, which is the Boolean version of the neuron.  Instead of the sign function,  it computes an arbitrary  (with poly(cid:173) nornialy many transitions) Boolean function of the weighted sum of  its inputs.  We  call the new computing element  an LT M  element,  which stands for  Linear Threshold with Multiple transitions.  The  paper consists  of the following  main  contributions related to  our study of LTM circuits:  (i)  the creation  of efficient  designs  of  LTM circuits for  the addition of a multiple number of integers and  the product of two integers.  In particular, we show how to compute  the  addition  of m  integers  with  a  single  layer  of  LT M  elements.  (ii)  a proof that the area of the VLSI layout is reduced from O(n2 )  in  LT circuits  to  O(n)  in  LTM circuits,  for  n  inputs  symmetric  Boolean functions,  and  (iii)  the characterization of the computing  power  of LT M  relative to LT circuits.
A General Purpose Image Processing Chip: Orientation Detection
Authors: Ralph Etienne-Cummings, Donghui Cai
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The  generalization  ability  of a  neural  network  can  sometimes  be  improved dramatically by regularization.  To  analyze the improve(cid:173) ment  one  needs  more  refined  results  than  the  asymptotic  distri(cid:173) bution  of  the  weight  vector.  Here  we  study  the  simple  case  of  one-dimensional  linear  regression  under  quadratic  regularization,  i.e.,  ridge  regression.  We  study  the  random  design,  misspecified  case, where we derive expansions for  the optimal regularization pa(cid:173) rameter and  the ensuing improvement.  It is  possible  to construct  examples  where it  is  best to use no regularization.
On Efficient Heuristic Ranking of Hypotheses
Authors: Steve A. Chien, Andre Stechert, Darren Mutz
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper considers  the  problem of learning the  ranking of a  set  of alternatives based  upon incomplete information (e.g.,  a  limited  number  of observations).  We  describe  two  algorithms for  hypoth(cid:173) esis  ranking and their application for  probably approximately cor(cid:173) rect  (PAC)  and  expected  loss  (EL)  learning  criteria.  Empirical  results are provided  to demonstrate the effectiveness of these  rank(cid:173) ing procedures on both synthetic datasets and real-world data from  a  spacecraft design optimization problem.
Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction
Authors: Hilbert J. Kappen, Francisco de Borja Rodríguez Ortiz
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  new  approximate learning  algorithm for  Boltzmann  Machines,  using a systematic expansion of the Gibbs free  energy to  second  order  in the weights.  The  linear response  correction  to the  correlations is  given  by  the  Hessian  of the  Gibbs free  energy.  The  computational complexity of the  algorithm is  cubic  in  the number  of neurons.  We compare the performance of the exact BM learning  algorithm  with  first  order  (Weiss)  mean  field  theory  and  second  order  (TAP)  mean field  theory.  The learning task consists of a fully  connected  Ising spin glass  model on  10  neurons.  We conclude that  1)  the  method  works  well  for  paramagnetic problems  2)  the  TAP  correction gives a significant improvement over the Weiss mean field  theory,  both for  paramagnetic and spin glass problems and 3)  that  the inclusion of diagonal weights improves the Weiss approximation  for  paramagnetic problems , but not for spin  glass  problems.
Incorporating Test Inputs into Learning
Authors: Zehra Cataltepe, Malik Magdon-Ismail
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In many  applications, such as credit default prediction and medical im(cid:173) age recognition, test inputs are available in  addition to the labeled train(cid:173) ing examples.  We  propose a method to  incorporate the test  inputs into  learning.  Our method results in  solutions having smaller test errors than  that of simple training solution, especially for  noisy  problems or small  training sets.
Reinforcement Learning for Continuous Stochastic Control Problems
Authors: Rémi Munos, Paul Bourgine
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper is concerned with the problem of Reinforcement Learn(cid:173) ing (RL) for continuous state space and time stocha.stic control  problems. We state the Harnilton-Jacobi-Bellman equation satis(cid:173) fied by the value function and use a Finite-Difference method for  designing a convergent approximation scheme. Then we propose a  RL algorithm based on this scheme and prove its convergence to  the optimal solution. 
1 
Introduction to RL in the continuous, stochastic case 
The objective of RL is to find -thanks to a reinforcement signal- an optimal strategy  for solving a dynamical control problem. Here we sudy the continuous time, con(cid:173) tinuous state-space stochastic case, which covers a wide variety of control problems  including target, viability, optimization problems (see [FS93], [KP95])}or which a  formalism is the following. The evolution of the current state x(t) E 0 (the state(cid:173) space, with 0 open subset of IRd ), depends on the control u(t) E U (compact subset)  by a stochastic differential equation, called the state dynamics: 
dx = f(x(t), u(t))dt + a(x(t), u(t))dw 
(1)  where f is the local drift and a .dw (with w a brownian motion of dimension rand  (j a d x r-matrix) the stochastic part (which appears for several reasons such as lake  of precision, noisy influence, random fluctuations) of the diffusion process.  For initial state x and control u(t), (1) leads to an infinity of possible traj~tories  x(t). For some trajectory x(t) (see figure I)., let T be its exit time from 0 (with  the convention that if x(t) always stays in 0, then T = 00). Then, we define the  functional J of initial state x and control u(.) as the expectation for all trajectories  of the discounted cumulative reinforcement : 
J(x; u(.)) = Ex,u( .) {loT '/r(x(t), u(t))dt +,,{ R(X(T))}
Intrusion Detection with Neural Networks
Authors: Jake Ryan, Meng-Jang Lin, Risto Miikkulainen
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: With the rapid expansion of computer networks during the past few years,  security  has  become a  crucial  issue  for  modern  computer systems.  A  good way  to  detect illegitimate use is  through monitoring unusual user  activity. Methods of intrusion detection based on hand-coded rule sets or  predicting commands on-line are laborous to build or not very reliable.  This  paper proposes  a new  way  of applying neural  networks  to  detect  intrusions. We believe that a user leaves a 'print' when using the system;  a neural network can  be  used  to learn this  print and  identify  each  user  much like detectives use thumbprints to place people at crime scenes.  If  a user's behavior does not match hislher print, the system administrator  can be alerted of a possible security breech.  A backpropagation neural  network called NNID  (Neural Network Intrusion Detector) was trained  in  the  identification  task  and  tested  experimentally  on  a  system  of 10  users.  The system was 96%  accurate in detecting unusual activity,  with  7%  false  alarm rate.  These results suggest that learning user profiles is  an effective way for detecting intrusions.
Incorporating Contextual Information in White Blood Cell Identification
Authors: Xubo B. Song, Yaser S. Abu-Mostafa, Joseph Sill, Harvey Kasdan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In this paper we propose a technique to incorporate contextual informa(cid:173) tion into object classification.  In the real world there are cases where the  identity of an object is  ambiguous due to the noise in the measurements  based on  which  the  classification  should  be  made.  It is  helpful  to  re(cid:173) duce the ambiguity by  utilizing extra information referred to  as context,  which  in  our case  is  the  identities  of the  accompanying  objects.  This  technique is applied to white blood cell classification.  Comparisons are  made against "no context" approach,  which  demonstrates  the  superior  classification performance achieved by  using context.  In  our particular  application, it significantly reduces false alarm rate and thus  greatly re(cid:173) duces the cost due to expensive clinical tests. 
• Author for correspondence. 
Incorporating  Contextual Information in White Blood Cell Identification
Function Approximation with the Sweeping Hinge Algorithm
Authors: Don R. Hush, Fernando Lozano, Bill G. Horne
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  computationally efficient  algorithm for  function  ap(cid:173) proximation with  piecewise  linear sigmoidal  nodes.  A one hidden  layer network is constructed one node at a time using the method of  fitting  the residual.  The task of fitting  individual  nodes  is  accom(cid:173) plished using a new algorithm that searchs for the best fit by solving  a sequence of Quadratic Programming problems.  This approach of(cid:173) fers  significant advantages over derivative-based search algorithms  (e.g.  backpropagation and its  extensions).  Unique  characteristics  of this  algorithm  include:  finite  step  convergence,  a  simple  stop(cid:173) ping criterion, a deterministic methodology for seeking "good" local  minima, good scaling properties and a robust numerical implemen(cid:173) tation.
The Rectified Gaussian Distribution
Authors: Nicholas D. Socci, Daniel D. Lee, H. Sebastian Seung
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A simple  but powerful  modification of the standard Gaussian dis(cid:173) tribution  is  studied.  The  variables  of  the  rectified  Gaussian  are  constrained  to  be  nonnegative,  enabling the use of nonconvex en(cid:173) ergy  functions.  Two  multimodal  examples,  the  competitive  and  cooperative  distributions,  illustrate  the  representational  power  of  the rectified  Gaussian.  Since the cooperative distribution can rep(cid:173) resent  the translations of a  pattern, it  demonstrates the  potential  of the rectified  Gaussian for  modeling pattern manifolds.
Learning Generative Models with the Up Propagation Algorithm
Authors: Jong-Hoon Oh, H. Sebastian Seung
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Up-propagation is an algorithm for inverting and learning neural network
generative models Sensory input is processed by inverting a model that
generates patterns from hidden variables using topdown connections
The inversion process is iterative utilizing a negative feedback loop that
depends on an error signal propagated by bottomup connections The
error signal is also used to learn the generative model from examples
The algorithm is benchmarked against principal component analysis in
experiments on images of handwritten digits.
Serial Order in Reading Aloud: Connectionist Models and Neighborhood Structure
Authors: Jeanne C. Milostan, Garrison W. Cottrell
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: If globally high dimensional data has locally only low dimensional distribu(cid:173) tions,  it is  advantageous to perform a local dimensionality reduction before  further processing the data.  In this paper we examine several techniques for  local  dimensionality reduction  in the  context of locally weighted linear re(cid:173) gression.  As possible candidates, we derive local versions of factor analysis  regression, principle component regression, principle component regression  on joint distributions, and partial least squares regression. After outlining the  statistical bases of these  methods,  we perform Monte Carlo  simulations to  evaluate  their  robustness  with  respect  to  violations  of their  statistical  as(cid:173) sumptions.  One  surprising  outcome  is  that  locally  weighted  partial  least  squares  regression offers the best average results,  thus outperforming even  factor analysis, the theoretically most appealing of our candidate techniques.
Adaptation in Speech Motor Control
Authors: John F. Houde, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Human  subjects  are  known  to  adapt  their  motor  behavior  to  a  shift  of the  visual  field  brought  about  by  wearing  prism  glasses  over their eyes.  We  have studied the analog of this effect  in speech.  U sing  a  device  that  can  feed  back  transformed  speech  signals  in  real  time,  we  exposed  subjects  to  alterations  of their  own  speech  feedback.  We  found  that speakers learn to adjust their production  of a  vowel  to compensate for  feedback  alterations that  change the  vowel's perceived phonetic identity; moreover, the effect generalizes  across consonant contexts and to different  vowels.
Nonlinear Markov Networks for Continuous Variables
Authors: Reimar Hofmann, Volker Tresp
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We address the problem oflearning structure in nonlinear Markov networks  with continuous variables.  This  can  be  viewed  as  non-Gaussian multidi(cid:173) mensional density estimation exploiting certain conditional independencies  in the variables.  Markov networks are  a graphical way of describing con(cid:173) ditional independencies well suited to model relationships which do not ex(cid:173) hibit a natural causal ordering.  We use neural network structures to model  the quantitative relationships between variables.  The main focus  in this pa(cid:173) per will be on learning the structure for the purpose of gaining insight into  the underlying process.  Using two data sets we show that interesting struc(cid:173) tures can be found using our approach.  Inference will be briefly addressed.
On-line Learning from Finite Training Sets in Nonlinear Networks
Authors: Peter Sollich, David Barber
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Online  learning  is  one  of the  most  common  forms  of neural  net(cid:173) work training.  We present an analysis of online learning from finite  training sets for  non-linear networks  (namely,  soft-committee ma(cid:173) chines),  advancing the  theory to more  realistic learning scenarios.  Dynamical  equations  are  derived  for  an  appropriate  set  of order  parameters;  these  are  exact  in  the  limiting  case  of  either  linear  networks  or  infinite  training  sets.  Preliminary  comparisons  with  simulations suggest that the theory captures some  effects  of finite  training sets, but may not yet account correctly for  the presence of  local minima.
Automated Aircraft Recovery via Reinforcement Learning: Initial Experiments
Authors: Jeffrey F. Monaco, David G. Ward, Andrew G. Barto
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Initial experiments described here were directed toward using reinforce(cid:173) ment learning (RL) to develop an automated recovery system (ARS) for  high-agility aircraft. An ARS is an outer-loop flight-control system de(cid:173) signed to bring an aircraft from a range of out-of-control states to straight(cid:173) and-level flight in minimum time while satisfying physical and phys(cid:173) iological constraints. Here we report on results for a simple version  of the problem involving only single-axis (pitch) simulated recoveries.  Through simulated control experience using a medium-fidelity aircraft  simulation, the RL system approximates an optimal policy for pitch-stick  inputs to produce minimum-time transitions to straight-and-Ievel flight in  unconstrained cases while avoiding ground-strike. The RL system was  also able to adhere to a pilot-station acceleration constraint while execut(cid:173) ing simulated recoveries. 
Automated Aircraft Recovery via Reinforcement Learning
Graph Matching with Hierarchical Discrete Relaxation
Authors: Richard C. Wilson, Edwin R. Hancock
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Our aim in this paper is to develop a Bayesian framework for match(cid:173) ing hierarchical relational models.  The goal is  to make discrete la(cid:173) bel assignments so as to optimise a  global cost function that draws  information concerning the consistency of match from different lev(cid:173) els  of the  hierarchy.  Our  Bayesian  development  naturally  distin(cid:173) guishes  between intra-level and inter-level constraints.  This allows  the  impact  of reassigning  a  match  to  be  assessed  not  only  at  its  own (or peer) level ofrepresentation, but also upon its parents and  children in  the hierarchy.