mtybilly/Instruct2DS huggingface dataset:
Temporal Dynamics of Generalization in Neural Networks
Authors: Santosh S. Venkatesh Changfeng Wang
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper  presents  a  rigorous  characterization  of how  a  general  nonlinear learning machine generalizes  during  the training process  when  it  is  trained  on  a  random  sample  using  a  gradient  descent  algorithm  based  on  reduction  of  training  error.  It is  shown,  in  particular, that best generalization performance occurs,  in general,  before  the global  minimum of the  training error  is  achieved.  The  different  roles  played  by  the  complexity of the  machine  class  and  the  complexity of the  specific  machine in the class  during learning  are  also precisely  demarcated.
Learning Stochastic Perceptrons Under k-Blocking Distributions
Authors: Saeed Hadjifaradji Mario Marchand
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  statistical  method  that  PAC  learns  the  class  of  stochastic  perceptrons  with  arbitrary  monotonic  activation  func(cid:173) tion and weights Wi  E  {-I, 0, + I} when the probability distribution  that  generates  the input examples  is  member of a  family  that we  call k-blocking distributions.  Such distributions represent an impor(cid:173) tant step beyond the case where each input variable is statistically  independent  since the 2k-blocking  family  contains  all  the  Markov  distributions of order k.  By stochastic  percept ron we  mean  a  per(cid:173) ceptron which,  upon presentation of input vector x, outputs 1 with  probability  fCLJi WiXi  - B).  Because the same algorithm works  for  any  monotonic  (nondecreasing  or  nonincreasing)  activation  func(cid:173) tion  f  on  Boolean  domain,  it  handles  the  well  studied  cases  of  sigmolds  and the  "usual"  radial basis functions.
Unsupervised Classification of 3D Objects from 2D Views
Authors: Hiroshi Ando Satoshi Suzuki
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper presents an unsupervised learning scheme for categorizing  3D  objects  from  their  2D  projected images.  The  scheme  exploits  an  auto-associative network's ability to encode each view of a single object  into a representation that indicates its view direction.  We propose two  models that employ different classification mechanisms; the first model  selects an auto-associative network whose recovered view best matches  the input view, and the second model is based on a modular architecture  whose  additional  network classifies  the  views  by  splitting  the  input  space  nonlinearly.  We demonstrate  the  effectiveness  of the  proposed  classification models through simulations using 3D wire-frame objects.
Hyperparameters Evidence and Generalisation for an Unrealisable Rule
Authors: David Saad Glenn Marion
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Using a statistical mechanical formalism we  calculate the evidence,  generalisation  error  and  consistency  measure  for  a  linear  percep(cid:173) tron  trained  and  tested  on  a  set  of examples  generated  by  a  non  linear  teacher.  The teacher  is  said  to be  unrealisable  because  the  student  can  never  model it without error.  Our model allows  us  to  interpolate between the known case of a linear teacher,  and an un(cid:173) realisable, nonlinear teacher.  A comparison of the hyperparameters  which  maximise the evidence  with  those  that optimise the perfor(cid:173) mance measures  reveals  that,  in  the  non-linear  case,  the  evidence  procedure is a misleading guide to optimising performance.  Finally,  we explore the extent to which the evidence procedure is  unreliable  and find  that, despite being sub-optimal, in  some circumstances  it  might be  a  useful  method for  fixing  the hyperparameters.
Adaptive Elastic Input Field for Recognition Improvement
Authors: Minoru Asogawa
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: For machines to perform classification tasks, such as speech and  character recognition, appropriately handling deformed patterns  is a key to achieving high performance. The authors presents a  new type of classification system, an Adaptive Input Field Neu(cid:173) ral Network (AIFNN), which includes a simple pre-trained neural  network and an elastic input field attached to an input layer. By  using an iterative method, AIFNN can determine an optimal affine  translation for an elastic input field to compensate for the original  deformations. The convergence of the AIFNN algorithm is shown.  AIFNN is applied for handwritten numerals recognition. Conse(cid:173) quently, 10.83% of originally misclassified patterns are correctly  categorized and total performance is improved, without modifying  the neural network.
A Model for Chemosensory Reception
Authors: Thomas Ragg, Martin Hammer Rainer Malaka
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A new  model for chemosensory reception is presented.  It models reacti(cid:173) ons between odor molecules and receptor proteins and the activation of  second  messenger  by receptor proteins.  The mathematical  formulation  of the  reaction  kinetics  is  transformed  into an  artificial  neural  network  (ANN). The resulting feed-forward network provides a powerful means  for parameter fitting by applying learning algorithms. The weights of the  network corresponding to chemical parameters can be trained by presen(cid:173) ting experimental data.  We demonstrate the simulation capabilities of the  model with experimental data from honey bee chemosensory neurons.  It  can be shown that our model is sufficient to rebuild the observed data and  that simpler models are not able to do this task.
A Real Time Clustering CMOS Neural Engine
Authors: Bernabé Linares-Barranco, José Luis Huertas Teresa Serrano-Gotarredona
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe  an  analog  VLSI  implementation  of the ARTI  algorithm  (Carpenter, 1987). A prototype chip has been fabricated in a standard low  cost  1.5~m double-metal single-poly CMOS process. It has a die area of  lcm2  and  is  mounted in  a  12O-pins PGA package.  The chip realizes  a  modified version of the original ARTI  architecture.  Such modification  has been shown to preserve all computational properties of the original  algorithm  (Serrano,  1994a),  while  being  more  appropriate  for  VLSI  realizations. The chip implements an ARTI network with 100 F 1 nodes  and 18 F2 nodes. It can therefore cluster 100 binary pixels input patterns  into up to 18 different categories. Modular expansibility of the system is  possible  by  assembling  an  NxM  array  of  chips  without  any  extra  interfacing circuitry, resulting in an F 1 layer with  l00xN nodes, and an  F2  layer  with  18xM nodes.  Pattern  classification  is  performed  in  less  than  1.8~s, which  means  an  equivalent  computing  power of 2.2x109  connections and connection-updates per second. Although internally the  chip is analog in nature, it interfaces to the outside world through digital  signals, thus having a true asynchrounous digital behavior. Experimental  chip  test  results  are  available,  which  have  been  obtained  through  test  equipments for digital chips.
Learning from queries for maximum information gain in imperfectly learnable problems
Authors: David Saad Peter Sollich
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In  supervised  learning,  learning  from  queries  rather  than  from  random  examples  can  improve  generalization  performance  signif(cid:173) icantly.  We  study  the  performance of query  learning for  problems  where  the  student  cannot learn  the  teacher  perfectly,  which  occur  frequently  in  practice.  As  a  prototypical scenario of this  kind,  we  consider  a  linear  perceptron  student  learning a  binary  perceptron  teacher.  Two kinds of queries for  maximum information gain,  i.e.,  minimum entropy,  are  investigated:  Minimum  student  space  en(cid:173) tropy  (MSSE)  queries,  which  are  appropriate  if the  teacher  space  is  unknown,  and  minimum teacher  space entropy  (MTSE)  queries,  which can be used  if the teacher space is  assumed to be known, but  a student of a simpler form  has deliberately  been  chosen.  We  find  that  for  MSSE  queries,  the  structure  of the  student  space  deter(cid:173) mines  the  efficacy  of query  learning,  whereas  MTSE  queries  lead  to  a  higher  generalization  error  than  random examples,  due  to  a  lack of feedback about the progress of the student in the way queries  are  selected.
Factorial Learning by Clustering Features
Authors: Emanuel V. Todorov Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We introduce a novel algorithm for factorial learning, motivated  by segmentation problems in computational vision, in which the  underlying factors correspond to clusters of highly correlated input  features. The algorithm derives from a new kind of competitive  clustering model, in which the cluster generators compete to ex(cid:173) plain each feature of the data set and cooperate to explain each  input example, rather than competing for examples and cooper(cid:173) ating on features, as in traditional clustering algorithms. A natu(cid:173) ral extension of the algorithm recovers hierarchical models of data  generated from multiple unknown categories, each with a differ(cid:173) ent, multiple causal structure. Several simulations demonstrate  the power of this approach.
Generalization in Reinforcement Learning: Safely Approximating the Value Function
Authors: Andrew W. Moore Justin A. Boyan
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  straightforward  approach  to  the  curse  of  dimensionality  in  re(cid:173) inforcement  learning  and  dynamic  programming  is  to  replace  the  lookup table with a generalizing function approximator such as a neu(cid:173) ral net.  Although this has been successful in the domain of backgam(cid:173) mon,  there  is  no  guarantee  of convergence.  In  this  paper,  we  show  that the combination of dynamic programming and function approx(cid:173) imation  is  not  robust,  and  in  even  very  benign  cases,  may produce  an  entirely  wrong  policy.  We  then  introduce  Grow-Support,  a  new  algorithm which is safe from divergence yet can still reap the benefits  of successful  generalization .
A Rapid Graph-based Method for Arbitrary Transformation-Invariant Pattern Classification
Authors: David G. Stork Alessandro Sperduti
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  graph-based  method  for  rapid,  accurate  search  through prototypes for  transformation-invariant pattern classifica(cid:173) tion.  Our method has in theory the same recognition accuracy as  other recent  methods  based  on  ''tangent  distance"  [Simard et al.,  1994],  since it uses the same categorization rule.  Nevertheless ours  is  significantly  faster  during  classification  because  far  fewer  tan(cid:173) gent  distances  need  be  computed.  Crucial  to  the success  of our  system  are  1)  a  novel  graph  architecture in  which  transformation  constraints and geometric  relationships  among  prototypes are en(cid:173) coded  during learning,  and 2)  an improved graph search criterion,  used during classification.  These architectural insights are applica(cid:173) ble to a wide range of problem domains.  Here we demonstrate that  on  a  handwriting  recognition task,  a  basic implementation of our  system  requires  less  than  half the  computation  of the  Euclidean  sorting method.
A solvable connectionist model of immediate recall of ordered lists
Authors: Neil Burgess
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  model  of short-term  memory for  serially  ordered  lists  of verbal  stimuli is proposed as an implementation of the 'articulatory loop'  thought  to  mediate  this  type  of memory  (Baddeley,  1986).  The  model predicts the presence  of a  repeatable time-varying 'context'  signal  coding  the  timing  of items'  presentation  in  addition  to  a  store  of phonological information and a  process  of serial rehearsal.  Items are associated with context nodes and phonemes by Hebbian  connections showing both short and long term plasticity.  Items are  activated  by  phonemic input  during  presentation  and reactivated  by context and phonemic feedback  during output.  Serial selection  of items  occurs  via a  winner-take-all  interaction  amongst  items,  with  the  winner  subsequently  receiving  decaying  inhibition.  An  approximate analysis  of error  probabilities due  to  Gaussian  noise  during  output  is  presented.  The  model  provides  an  explanatory  account of the  probability of error as  a  function of serial  position,  list  length,  word  length,  phonemic similarity,  temporal  grouping,  item and list familiarity,  and is proposed  as the starting point for  a  model of rehearsal and vocabulary acquisition.
Grammar Learning by a Self-Organizing Network
Authors: Michiro Negishi
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper presents the design and simulation results of a  self(cid:173) organizing neural network which induces a grammar from exam(cid:173) ple sentences. Input sentences are generated from a simple phrase  structure grammar including number agreement,  verb  transitiv(cid:173) ity,  and recursive  noun phrase construction rules.  The  network  induces a grammar explicitly in the form of symbol categorization  rules and phrase structure rules. 
1  Purpose and related works 
The purpose of this research is to show that a self-organizing network with a certain  structure can acquire syntactic knowledge from only positive (i.e.  grammatical)  data,  without requiring  any initial  knowledge  or  external  teachers  that correct  errors.  There has been research on supervised neural network models of language acquisi(cid:173) tion tasks [Elman, 1991, Miikkulainen and Dyer, 1988, John and McClelland, 1988].  Unlike these supervised models, the current model self-organizes word and phrasal  categories and phrase construction rules through mere exposure to input sentences,  without any artificially defined  task goals.  There  also have been self-organizing  models of language  acquisition  tasks [Ritter and Kohonen, 1990,  Scholtes, 1991].  Compared to these models, the current model acquires phrase structure rules in  more explicit forms, and it learns wider and more structured contexts, as will be  explained below. 
2  Network Structure and Algorithm 
The design of the current network is motivated by the observation that humans  have  the  ability  to  handle  a  frequently  occurring  sequence of symbols (chunk)  as an unit of information [Grossberg, 1978, Mannes, 1993].  The network consists  of two parts:  classification networks and production networks (Figure 1).  The  classification networks categorize words and phrases, and the production networks 
28
Coarse-to-Fine Image Search Using Neural Networks
Authors: John C. Pearson, Jim Bergen Clay Spence
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The  efficiency  of image  search  can  be  greatly  improved  by  using  a  coarse-to-fine search strategy with a  multi-resolution image representa(cid:173) tion. However,  if the resolution is so low that the objects have few  dis(cid:173) tinguishing  features,  search  becomes  difficult.  We  show  that  the  performance of search at such low resolutions can be improved by using  context information, i.e., objects visible at low-resolution which are not  the objects of interest but are associated with them. The networks can be  given explicit context information as inputs, or they can learn to detect  the context objects, in which case the user does not have to be aware of  their existence. We also use Integrated Feature Pyramids, which repre(cid:173) sent high-frequency  information  at low  resolutions.  The use of multi(cid:173) resolution search techniques allows us to combine information about the  appearance of the objects on many scales in an efficient way.  A natural  fOlm  of exemplar selection also arises from these  techniques.  We illus(cid:173) trate these ideas by training hierarchical systems of neural  networks to  find clusters of buildings in aerial photographs of farmland.
ICEG Morphology Classification using an Analogue VLSI Neural Network
Authors: Marwan A. Jabri, Barry Flower, Stephen Pickard Richard Coggins
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: An  analogue  VLSI  neural  network  has  been  designed  and  tested  to perform cardiac morphology classification tasks.  Analogue tech(cid:173) niques were chosen to meet the strict power and area requirements  of an Implantable Cardioverter Defibrillator (ICD) system.  The ro(cid:173) bustness  of the neural network  architecture  reduces  the impact of  noise,  drift  and offsets  inherent  in  analogue approaches.  The net(cid:173) work  is a  10:6:3 multi-layer percept ron  with on  chip  digital weight  storage,  a  bucket  brigade input  to feed  the  Intracardiac  Electro(cid:173) gram  (ICEG)  to  the  network  and  has  a  winner  take  all  circuit  at  the  output.  The  network  was  trained  in  loop  and  included  a  commercial ICD in the signal processing path.  The system has suc(cid:173) cessfully distinguished arrhythmia for different patients with better  than 90%  true positive and true negative detections for  dangerous  rhythms which cannot be detected  by present  ICDs.  The chip  was  implemented in 1.2um CMOS and consumes less than 200n W max(cid:173) imum average power in an area of 2.2  x 2.2mm2.
Prior Knowledge in Support Vector Kernels
Authors: Patrice Simard, Alex J. Smola, Vladimir Vapnik Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We explore methods for incorporating prior knowledge about a problem  at hand in Support Vector learning machines.  We  show that both invari(cid:173) ances under group transfonnations and prior knowledge about locality in  images can be incorporated by constructing appropriate kernel functions.
A Revolution: Belief Propagation in Graphs with Cycles
Authors: David J. C. MacKay Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Until recently, artificial intelligence researchers have frowned  upon  the  application of probability propagation in  Bayesian  belief net(cid:173) works  that  have cycles.  The probability propagation algorithm is  only exact in networks that are cycle-free.  However, it has recently  been  discovered  that the  two  best  error-correcting decoding  algo(cid:173) rithms  are  actually  performing  probability  propagation  in  belief  networks with cycles. 
1  Communicating over a  noisy channel 
Our increasingly wired world demands efficient  methods for  communicating bits of  information  over  physical  channels  that  introduce  errors.  Examples  of real-world  channels  include  twisted-pair telephone  wires,  shielded  cable-TV  wire,  fiber-optic  cable,  deep-space  radio,  terrestrial  radio,  and  indoor  radio.  Engineers  attempt  to  correct  the  errors  introduced  by  the  noise  in  these  channels  through  the  use  of  channel  coding  which  adds  protection  to the information  source,  so  that  some  channel errors  can  be  corrected.  A  popular  model  of a  physical  channel  is  shown  in  Fig.  1.  A vector of K  information bits u  =  (Ut, ... ,UK), Uk  E {O, I}  is  encoded,  and a  vector of N  codeword bits x  =  (Xl! ... ,XN)  is  transmitted into the channel.  Independent  Gaussian noise  with  variance  (12  is  then  added to each codeword bit, 
.. Brendan Frey  is  currently a  Beckman Fellow  at the Beckman Institute for  Advanced 
Science and Technology, University of Illinois  at Urbana-Champaign. 
480 
B.  J  Frey and D.  J.  C. MacKay 
Gaussian noise  with variance (J2 
--U--'~~I~ _ E _ ncoder __ ~1  x  ~ y 
.I~ _ Decoder _ ~--U~.~ 
Figure 1:  A communication system with a channel that adds Gaussian noise to the  transmitted discrete-time sequence. 
producing  the  real-valued  channel  output  vector  y  =  (Y!, ... ,YN).  The  decoder  must  then use  this  received  vector  to  make  a  guess  U at the  original  information  vector.  The  probability  P" (e)  of  bit  error  is  minimized  by  choosing  the  Uk  that  maximizes  P(ukly)  for  k  = 1, ... , K.  The  rate  K/N of a  code  is  the  number  of  information  bits  communicated  per  codeword  bit.  We  will  consider  rate  ~ 1/2  systems in this paper,  where  N  ==  2K.  The simplest rate 1/2 encoder duplicates each information hit:  X2k-l  =  X2k  =  Uk,  k  =  1, ... , K.  The optimal decoder for this repetition code  simply averages together  pairs of noisy channel outputs and then applies  a threshold:
Dynamic Stochastic Synapses as Computational Units
Authors: Anthony M. Zador Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In  most  neural network models, synapses are treated as static weights  that  change only on  the slow  time scales of learning.  In  fact,  however, synapses  are highly  dynamic,  and show  use-dependent  plasticity over a  wide  range  of time scales.  Moreover,  synaptic transmission is  an  inherently stochastic  process:  a  spike  arriving  at  a  presynaptic  terminal  triggers  release  of a  vesicle  of neurotransmitter from  a  release  site  with  a  probability that can  be  much  less  than one.  Changes in  release probability represent one of the  main mechanisms by which synaptic efficacy is modulated in neural circuits.  We propose and investigate a simple model for dynamic stochastic synapses  that can  easily be  integrated into common models  for  neural computation.  We  show  through  computer  simulations  and  rigorous  theoretical  analysis  that  this  model  for  a  dynamic  stochastic synapse increases  computational  power in a nontrivial way.  Our results may have implications for the process(cid:173) ing of time-varying signals by both biological and artificial neural networks. 
A synapse 8  carries out computations on spike trains, more precisely on  trains of spikes  from  the  presynaptic  neuron.  Each  spike  from  the  presynaptic  neuron  mayor may  not  trigger the release  of a  neurotransmitter-filled  vesicle  at the synapse.  The  probability of a  vesicle  release ranges from  about  0.01  to almost  1.  Furthermore this  release probability is  known  to  be strongly  "history dependent"  [Dobrunz and  Stevens, 1997].  A spike causes an  excitatory or inhibitory potential  (EPSP or IPSP, respectively)  in  the postsynaptic  neuron  only when a  vesicle  is  released. 
A spike  train is  represented  as a sequence 1 of firing  times,  i.e.  as increasing sequences  of numbers tl < t2  < ... from  R+ := {z E R: z ~ O}  . For each  spike train 1 the output of  synapse  8  consists of the  sequence 8W  of those  ti  E 10n which  vesicles  are  "released"  by  8  , i.e.  of those t, E 1 which cause an excitatory or inhibitory postsynaptic potential (EPSP  or IPSP,  respectively).  The map 1 -+  8(1)  may  be  viewed  as  a  stochastic function  that is  computed by  synapse S.  Alternatively one can  characterize  the  output  SW  of a  synapse  8  through its  release  pattern q = qlq2 ... E  {R, F}·  , where  R  stands for  release and  F  for  failure  of release.  For each  t, E 1 one sets q,  =  R  if ti  E 8(1)  , and qi  =  F  if ti  ¢ 8W  . 
Dynamic Stochastic Synapses as Computational Units 
195
Self-similarity Properties of Natural Images
Authors: Germán Mato, Néstor Parga, Jean-Pierre Nadal Antonio Turiel
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Scale  invariance  is  a  fundamental  property  of ensembles  of  nat(cid:173) ural  images  [1].  Their  non  Gaussian  properties  [15,  16]  are  less  well  understood,  but  they  indicate  the  existence  of  a  rich  statis(cid:173) tical  structure.  In  this  work  we  present  a  detailed  study  of the  marginal statistics of a  variable related to the edges in the images.  A numerical analysis shows that it exhibits extended self-similarity  [3,  4,  5].  This  is  a  scaling  property  stronger  than  self-similarity:  all its moments can be expressed as  a power of any given moment.  More  interesting,  all  the  exponents  can  be  predicted  in  terms  of  a  multiplicative  log-Poisson process.  This  is  the very  same model  that  was  used  very  recently  to  predict  the  correct  exponents  of  the structure functions  of turbulent flows  [6].  These  results  allow  us to study the underlying multifractal singularities.  In  particular  we  find  that the most singular structures are one-dimensional:  the  most singular manifold  consists of sharp edges. 
Category:  Visual Processing.
Multiple Threshold Neural Logic
Authors: Jehoshua Bruck Vasken Bohossian
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We introduce a new Boolean computing element related to the Lin(cid:173) ear Threshold element, which is the Boolean version of the neuron.  Instead of the sign function,  it computes an arbitrary  (with poly(cid:173) nornialy many transitions) Boolean function of the weighted sum of  its inputs.  We  call the new computing element  an LT M  element,  which stands for  Linear Threshold with Multiple transitions.  The  paper consists  of the following  main  contributions related to  our study of LTM circuits:  (i)  the creation  of efficient  designs  of  LTM circuits for  the addition of a multiple number of integers and  the product of two integers.  In particular, we show how to compute  the  addition  of m  integers  with  a  single  layer  of  LT M  elements.  (ii)  a proof that the area of the VLSI layout is reduced from O(n2 )  in  LT circuits  to  O(n)  in  LTM circuits,  for  n  inputs  symmetric  Boolean functions,  and  (iii)  the characterization of the computing  power  of LT M  relative to LT circuits.
A General Purpose Image Processing Chip: Orientation Detection
Authors: Donghui Cai Ralph Etienne-Cummings
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The  generalization  ability  of a  neural  network  can  sometimes  be  improved dramatically by regularization.  To  analyze the improve(cid:173) ment  one  needs  more  refined  results  than  the  asymptotic  distri(cid:173) bution  of  the  weight  vector.  Here  we  study  the  simple  case  of  one-dimensional  linear  regression  under  quadratic  regularization,  i.e.,  ridge  regression.  We  study  the  random  design,  misspecified  case, where we derive expansions for  the optimal regularization pa(cid:173) rameter and  the ensuing improvement.  It is  possible  to construct  examples  where it  is  best to use no regularization.
On Efficient Heuristic Ranking of Hypotheses
Authors: Andre Stechert, Darren Mutz Steve A. Chien
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper considers  the  problem of learning the  ranking of a  set  of alternatives based  upon incomplete information (e.g.,  a  limited  number  of observations).  We  describe  two  algorithms for  hypoth(cid:173) esis  ranking and their application for  probably approximately cor(cid:173) rect  (PAC)  and  expected  loss  (EL)  learning  criteria.  Empirical  results are provided  to demonstrate the effectiveness of these  rank(cid:173) ing procedures on both synthetic datasets and real-world data from  a  spacecraft design optimization problem.
Learning Continuous Attractors in Recurrent Networks
Authors: H. Sebastian Seung
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: One approach  to invariant  object  recognition  employs  a  recurrent  neu(cid:173) ral  network  as an associative  memory.  In  the standard depiction of the  network's state space, memories of objects are stored as attractive fixed  points of the dynamics.  I argue for  a  modification of this picture:  if an  object has a continuous family of instantiations, it should be represented  by  a  continuous  attractor.  This  idea is  illustrated with  a  network  that  learns to complete patterns.  To  perform the task of filling  in missing in(cid:173) formation,  the network develops a  continuous attractor that models the  manifold  from  which  the patterns  are  drawn.  From  a  statistical  view(cid:173) point, the pattern completion task allows  a  formulation of unsupervised  learning in  terms of regression  rather than density estimation. 
A classic approach to invariant object recognition is  to  use  a  recurrent neural net(cid:173) work  as  an  associative  memory[l].  In  spite  of the  intuitive  appeal  and  biological  plausibility of this approach, it has largely been abandoned in practical applications.  This paper introduces two new concepts  that could help resurrect it:  object repre(cid:173) sentation by continuous attractors, and learning attractors by  pattern completion.  In most models of associative memory, memories are stored as attractive fixed points  at discrete locations in state space[l].  Discrete attractors may not be appropriate for  patterns with  continuous  variability,  like the images of a  three-dimensional object  from  different viewpoints.  When the instantiations of an object lie on a continuous  pattern manifold, it is more appropriate to represent objects by attractive manifolds  of fixed  points, or continuous attractors.  To  make this idea practical, it is  important to find  methods for  learning attractors  from examples.  A naive method is  to train the network to retain examples in short(cid:173) term memory.  This  method  is  deficient  because  it  does  not  prevent  the  network  from  storing spurious  fixed  points that are  unrelated to the examples.  A  superior  method  is  to  train the network to  restore  examples  that  have  been  corrupted,  so  that it learns to complete patterns by filling  in missing information. 
Learning Continuous Attractors in Recurrent Networks
Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction
Authors: Francisco de Borja Rodríguez Ortiz Hilbert J. Kappen
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  new  approximate learning  algorithm for  Boltzmann  Machines,  using a systematic expansion of the Gibbs free  energy to  second  order  in the weights.  The  linear response  correction  to the  correlations is  given  by  the  Hessian  of the  Gibbs free  energy.  The  computational complexity of the  algorithm is  cubic  in  the number  of neurons.  We compare the performance of the exact BM learning  algorithm  with  first  order  (Weiss)  mean  field  theory  and  second  order  (TAP)  mean field  theory.  The learning task consists of a fully  connected  Ising spin glass  model on  10  neurons.  We conclude that  1)  the  method  works  well  for  paramagnetic problems  2)  the  TAP  correction gives a significant improvement over the Weiss mean field  theory,  both for  paramagnetic and spin glass problems and 3)  that  the inclusion of diagonal weights improves the Weiss approximation  for  paramagnetic problems , but not for spin  glass  problems.
Incorporating Test Inputs into Learning
Authors: Malik Magdon-Ismail Zehra Cataltepe
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In many  applications, such as credit default prediction and medical im(cid:173) age recognition, test inputs are available in  addition to the labeled train(cid:173) ing examples.  We  propose a method to  incorporate the test  inputs into  learning.  Our method results in  solutions having smaller test errors than  that of simple training solution, especially for  noisy  problems or small  training sets.
An Application of Reversible-Jump MCMC to Multivariate Spherical Gaussian Mixtures
Authors: Alan D. Marrs
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Applications  of Gaussian  mixture  models  occur  frequently  in  the  fields  of statistics  and  artificial  neural  networks.  One  of the  key  issues  arising  from  any  mixture  model  application  is  how  to  es(cid:173) timate  the  optimum  number  of mixture  components.  This  paper  extends the Reversible-Jump Markov Chain Monte Carlo (MCMC)  algorithm  to the  case of multivariate spherical  Gaussian  mixtures  using  a  hierarchical  prior  model.  Using  this  method  the  number  of mixture  components  is  no  longer  fixed  but  becomes  a  param(cid:173) eter of the  model  which  we  shall estimate.  The Reversible-Jump  MCMC  algorithm  is  capable  of  moving  between  parameter  sub(cid:173) spaces which  correspond to models  with different  numbers of mix(cid:173) ture components.  As  a result a  sample from  the full joint distribu(cid:173) tion of all  unknown model parameters is generated.  The technique  is  then  demonstrated  on  a  simulated  example  and  a  well  known  vowel dataset.
A 1, 000-Neuron System with One Million 7-bit Physical Interconnections
Authors: Yuzo Hirai
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: An asynchronous PDM (Pulse-Density-Modulating) digital neural  network system has been developed in our laboratory. It consists  of one thousand neurons that are physically interconnected via one  million 7-bit synapses. It can solve one thousand simultaneous  nonlinear first-order differential equations in a fully parallel and  continuous fashion. The performance of this system was measured  by a winner-take-all network with one thousand neurons. Although  the magnitude of the input and network parameters were identi(cid:173) cal for each competing neuron, one of them won in 6 milliseconds.  This processing speed amounts to 360 billion connections per sec(cid:173) ond. A broad range of neural networks including spatiotemporal  filtering, feedforward, and feedback networks can be run by loading  appropriate network parameters from a host system.
Reinforcement Learning for Continuous Stochastic Control Problems
Authors: Paul Bourgine Rémi Munos
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper is concerned with the problem of Reinforcement Learn(cid:173) ing (RL) for continuous state space and time stocha.stic control  problems. We state the Harnilton-Jacobi-Bellman equation satis(cid:173) fied by the value function and use a Finite-Difference method for  designing a convergent approximation scheme. Then we propose a  RL algorithm based on this scheme and prove its convergence to  the optimal solution. 
1 
Introduction to RL in the continuous, stochastic case 
The objective of RL is to find -thanks to a reinforcement signal- an optimal strategy  for solving a dynamical control problem. Here we sudy the continuous time, con(cid:173) tinuous state-space stochastic case, which covers a wide variety of control problems  including target, viability, optimization problems (see [FS93], [KP95])}or which a  formalism is the following. The evolution of the current state x(t) E 0 (the state(cid:173) space, with 0 open subset of IRd ), depends on the control u(t) E U (compact subset)  by a stochastic differential equation, called the state dynamics: 
dx = f(x(t), u(t))dt + a(x(t), u(t))dw 
(1)  where f is the local drift and a .dw (with w a brownian motion of dimension rand  (j a d x r-matrix) the stochastic part (which appears for several reasons such as lake  of precision, noisy influence, random fluctuations) of the diffusion process.  For initial state x and control u(t), (1) leads to an infinity of possible traj~tories  x(t). For some trajectory x(t) (see figure I)., let T be its exit time from 0 (with  the convention that if x(t) always stays in 0, then T = 00). Then, we define the  functional J of initial state x and control u(.) as the expectation for all trajectories  of the discounted cumulative reinforcement : 
J(x; u(.)) = Ex,u( .) {loT '/r(x(t), u(t))dt +,,{ R(X(T))}
Intrusion Detection with Neural Networks
Authors: Meng-Jang Lin, Risto Miikkulainen Jake Ryan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: With the rapid expansion of computer networks during the past few years,  security  has  become a  crucial  issue  for  modern  computer systems.  A  good way  to  detect illegitimate use is  through monitoring unusual user  activity. Methods of intrusion detection based on hand-coded rule sets or  predicting commands on-line are laborous to build or not very reliable.  This  paper proposes  a new  way  of applying neural  networks  to  detect  intrusions. We believe that a user leaves a 'print' when using the system;  a neural network can  be  used  to learn this  print and  identify  each  user  much like detectives use thumbprints to place people at crime scenes.  If  a user's behavior does not match hislher print, the system administrator  can be alerted of a possible security breech.  A backpropagation neural  network called NNID  (Neural Network Intrusion Detector) was trained  in  the  identification  task  and  tested  experimentally  on  a  system  of 10  users.  The system was 96%  accurate in detecting unusual activity,  with  7%  false  alarm rate.  These results suggest that learning user profiles is  an effective way for detecting intrusions.
Incorporating Contextual Information in White Blood Cell Identification
Authors: Yaser S. Abu-Mostafa, Joseph Sill, Harvey Kasdan Xubo B. Song
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In this paper we propose a technique to incorporate contextual informa(cid:173) tion into object classification.  In the real world there are cases where the  identity of an object is  ambiguous due to the noise in the measurements  based on  which  the  classification  should  be  made.  It is  helpful  to  re(cid:173) duce the ambiguity by  utilizing extra information referred to  as context,  which  in  our case  is  the  identities  of the  accompanying  objects.  This  technique is applied to white blood cell classification.  Comparisons are  made against "no context" approach,  which  demonstrates  the  superior  classification performance achieved by  using context.  In  our particular  application, it significantly reduces false alarm rate and thus  greatly re(cid:173) duces the cost due to expensive clinical tests. 
• Author for correspondence. 
Incorporating  Contextual Information in White Blood Cell Identification
Function Approximation with the Sweeping Hinge Algorithm
Authors: Fernando Lozano, Bill G. Horne Don R. Hush
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  computationally efficient  algorithm for  function  ap(cid:173) proximation with  piecewise  linear sigmoidal  nodes.  A one hidden  layer network is constructed one node at a time using the method of  fitting  the residual.  The task of fitting  individual  nodes  is  accom(cid:173) plished using a new algorithm that searchs for the best fit by solving  a sequence of Quadratic Programming problems.  This approach of(cid:173) fers  significant advantages over derivative-based search algorithms  (e.g.  backpropagation and its  extensions).  Unique  characteristics  of this  algorithm  include:  finite  step  convergence,  a  simple  stop(cid:173) ping criterion, a deterministic methodology for seeking "good" local  minima, good scaling properties and a robust numerical implemen(cid:173) tation.
Correlates of Attention in a Model of Dynamic Visual Recognition
Authors: Rajesh P. N. Rao
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Given a set of objects in the visual field, how does the the visual system learn  to attend to a particular object of interest while ignoring the rest?  How are  occlusions and background clutter so effortlessly discounted for when rec(cid:173) ognizing a familiar object? In this paper, we attempt to answer these ques(cid:173) tions in the context of a Kalman filter-based model of visual recognition that  has previously proved useful in explaining certain neurophysiological phe(cid:173) nomena such as endstopping and related extra-classical receptive field ef(cid:173) fects in the visual cortex. By using results from the field of robust statistics,  we describe an extension of the Kalman filter model that can handle multiple  objects in the visual field.  The resulting robust Kalman filter model demon(cid:173) strates how certain forms of attention can be viewed as an emergent prop(cid:173) erty of the interaction between top-down expectations and bottom-up sig(cid:173) nals.  The model also suggests functional interpretations of certain attention(cid:173) related effects that have been observed in visual cortical neurons.  Exper(cid:173) imental results are provided to  help demonstrate the ability  of the  model  to perform robust segmentation and recognition of objects and image se(cid:173) quences in the presence of varying degrees of occlusions and clutter.
Mapping a Manifold of Perceptual Observations
Authors: Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Nonlinear dimensionality reduction is formulated here as the problem of trying to  find a Euclidean feature-space embedding of a set of observations that preserves  as closely as possible their intrinsic metric structure - the distances between points  on  the  observation manifold as  measured along geodesic paths.  Our isometric  feature mapping procedure, or isomap, is able to reliably recover low-dimensional  nonlinear structure in  realistic  perceptual data  sets,  such as  a manifold  of face  images,  where  conventional global  mapping  methods  find  only  local  minima.  The  recovered  map  provides  a canonical  set  of globally  meaningful  features,  which allows perceptual transformations such as interpolation, extrapolation, and  analogy - highly nonlinear transformations in the original observation space - to  be computed with simple linear operations in feature space.
The Rectified Gaussian Distribution
Authors: Daniel D. Lee, H. Sebastian Seung Nicholas D. Socci
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A simple  but powerful  modification of the standard Gaussian dis(cid:173) tribution  is  studied.  The  variables  of  the  rectified  Gaussian  are  constrained  to  be  nonnegative,  enabling the use of nonconvex en(cid:173) ergy  functions.  Two  multimodal  examples,  the  competitive  and  cooperative  distributions,  illustrate  the  representational  power  of  the rectified  Gaussian.  Since the cooperative distribution can rep(cid:173) resent  the translations of a  pattern, it  demonstrates the  potential  of the rectified  Gaussian for  modeling pattern manifolds.
RCC Cannot Compute Certain FSA, Even with Arbitrary Transfer Functions
Authors: Mark Ring
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Existing proofs demonstrating the computational limitations of Re(cid:173) current  Cascade Correlation and similar networks  (Fahlman, 1991;  Bachrach,  1988;  Mozer,  1988)  explicitly limit their  results  to  units  having sigmoidal or hard-threshold  transfer functions  (Giles et  aI.,  1995;  and  Kremer,  1996).  The  proof given  here  shows  that  for  any  finite,  discrete  transfer  function  used  by  the  units of an  RCC  network,  there  are  finite-state  automata  (FSA)  that  the  network  cannot model, no matter how  many units are  used.  The proof also  applies  to  continuous  transfer  functions  with  a  finite  number  of  fixed-points,  such  as  sigmoid and  radial-basis functions.
Learning Generative Models with the Up Propagation Algorithm
Authors: H. Sebastian Seung Jong-Hoon Oh
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Up-propagation is an algorithm for inverting and learning neural network
generative models Sensory input is processed by inverting a model that
generates patterns from hidden variables using topdown connections
The inversion process is iterative utilizing a negative feedback loop that
depends on an error signal propagated by bottomup connections The
error signal is also used to learn the generative model from examples
The algorithm is benchmarked against principal component analysis in
experiments on images of handwritten digits.
Serial Order in Reading Aloud: Connectionist Models and Neighborhood Structure
Authors: Garrison W. Cottrell Jeanne C. Milostan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: If globally high dimensional data has locally only low dimensional distribu(cid:173) tions,  it is  advantageous to perform a local dimensionality reduction before  further processing the data.  In this paper we examine several techniques for  local  dimensionality reduction  in the  context of locally weighted linear re(cid:173) gression.  As possible candidates, we derive local versions of factor analysis  regression, principle component regression, principle component regression  on joint distributions, and partial least squares regression. After outlining the  statistical bases of these  methods,  we perform Monte Carlo  simulations to  evaluate  their  robustness  with  respect  to  violations  of their  statistical  as(cid:173) sumptions.  One  surprising  outcome  is  that  locally  weighted  partial  least  squares  regression offers the best average results,  thus outperforming even  factor analysis, the theoretically most appealing of our candidate techniques.
Adaptation in Speech Motor Control
Authors: Michael I. Jordan John F. Houde
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Human  subjects  are  known  to  adapt  their  motor  behavior  to  a  shift  of the  visual  field  brought  about  by  wearing  prism  glasses  over their eyes.  We  have studied the analog of this effect  in speech.  U sing  a  device  that  can  feed  back  transformed  speech  signals  in  real  time,  we  exposed  subjects  to  alterations  of their  own  speech  feedback.  We  found  that speakers learn to adjust their production  of a  vowel  to compensate for  feedback  alterations that  change the  vowel's perceived phonetic identity; moreover, the effect generalizes  across consonant contexts and to different  vowels.