mtybilly/Instruct2DS huggingface dataset:
Adaptive On-line Learning in Changing Environments
Authors: Noboru Murata, Klaus-Robert Müller, Andreas Ziehe, Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: An adaptive on-line algorithm extending the learning of learning  idea is proposed and theoretically motivated. Relying only on gra(cid:173) dient flow information it can be applied to learning continuous  functions or distributions, even when no explicit loss function is gi(cid:173) ven and the Hessian is not available. Its efficiency is demonstrated  for a non-stationary blind separation task of acoustic signals.
A Micropower Analog VLSI HMM State Decoder for Wordspotting
Authors: John Lazzaro, John Wawrzynek, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  describe  the  implementation of a  hidden  Markov  model  state  decoding  system,  a component for  a  wordspotting speech  recogni(cid:173) tion system.  The key  specification for  this state decoder  design  is  microwatt power dissipation; this requirement led to a continuous(cid:173) time, analog circuit implementation.  We characterize the operation  of a  10-word (81  state)  state decoder  test chip.
Dual Kalman Filtering Methods for Nonlinear Prediction, Smoothing and Estimation
Authors: Eric A. Wan, Alex T. Nelson
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Prediction,  estimation,  and smoothing  are  fundamental  to  signal  processing.  To perform  these  interrelated  tasks  given  noisy  data,  we  form  a  time  series  model  of  the  process  that  generates  the  data.  Taking noise in the system explicitly into account, maximum(cid:173) likelihood and Kalman frameworks are discussed  which involve the  dual process  of estimating both the model parameters and the un(cid:173) derlying state of the system.  We  review  several  established  meth(cid:173) ods in the linear case,  and propose severa! extensions utilizing dual  Kalman filters  (DKF)  and forward-backward  (FB)  filters  that  are  applicable  to  neural  networks.  Methods  are  compared on  several  simulations of noisy  time series.  We  also  include  an  example  of  nonlinear noise reduction in speech.
Source Separation and Density Estimation by Faithful Equivariant SOM
Authors: Juan K. Lin, Jack D. Cowan, David G. Grier
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  couple  the  tasks  of source  separation  and  density  estimation  by  extracting  the  local  geometrical  structure  of distributions  ob(cid:173) tained  from  mixtures  of statistically  independent  sources.  Our  modifications of the self-organizing map  (SOM)  algorithm  results  in  purely digital learning rules which perform non-parametric his(cid:173) togram density estimation.  The non-parametric nature of the sep(cid:173) aration  allows  for  source  separation  of non-linear  mixtures.  An  anisotropic coupling  is  introduced  into  our  SOM  with  the  role  of  aligning the network locally with the independent component con(cid:173) tours.  This  approach  provides  an  exact  verification  condition  for  source separation with no prior on  the source distributions.
A Mean Field Algorithm for Bayes Learning in Large Feed-forward Neural Networks
Authors: Manfred Opper, Ole Winther
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We present an algorithm which is expected to realise Bayes optimal  predictions in large feed-forward networks.  It is based on mean field  methods developed  within  statistical  mechanics  of disordered  sys(cid:173) tems.  We give a derivation for  the single layer perceptron and show  that  the  algorithm  also  provides  a  leave-one-out  cross-validation  test of the  predictions.  Simulations show excellent  agreement with  theoretical  results  of statistical mechanics.
Multi-Task Learning for Stock Selection
Authors: Joumana Ghosn, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Artificial  Neural  Networks  can  be  used  to  predict  future  returns  of stocks  in order to  take financial  decisions .  Should  one  build  a  separate network for  each stock or share the same network for  all  the  stocks?  In  this  paper  we  also  explore  other  alternatives,  in  which  some  layers  are  shared  and  others  are  not  shared.  When  the  prediction  of future  returns  for  different  stocks  are viewed  as  different  tasks,  sharing  some  parameters  across  stocks  is  a  form  of multi-task  learning.  In  a  series  of experiments  with  Canadian  stocks,  we obtain yearly returns that are more than 14% above  various benchmarks. 
1
Cholinergic Modulation Preserves Spike Timing Under Physiologically Realistic Fluctuating Input
Authors: Akaysha C. Tang, Andreas M. Bartels, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Neuromodulation  can  change  not  only  the  mean  firing  rate  of a  neuron,  but  also  its  pattern  of firing .  Therefore,  a  reliable  neu(cid:173) ral  coding  scheme,  whether  a  rate  coding  or  a  spike  time  based  coding,  must  be  robust  in  a  dynamic  neuromodulatory  environ(cid:173) ment.  The common observation that cholinergic modulation leads  to  a  reduction  in  spike  frequency  adaptation  implies  a  modifica(cid:173) tion  of spike  timing,  which  would  make  a  neural  code  based  on  precise spike timing difficult to maintain.  In this paper, the effects  of cholinergic modulation were  studied to test  the hypothesis that  precise  spike timing can serve  as a  reliable neural code.  Using  the  whole  cell  patch-clamp technique  in rat neocortical  slice  prepara(cid:173) tion and compartmental modeling techniques,  we show that cholin(cid:173) ergic  modulation, surprisingly,  preserved  spike  timing in response  to a fluctuating  inputs that resembles  in  vivo conditions.  This re(cid:173) sult suggests that in vivo spike timing may be much more resistant  to changes in neuromodulator concentrations than previous physi(cid:173) ological studies have  implied. 
112 
A. C.  Tang, A.  M.  Bartels and T.  J. Sejnowski
Analytical Mean Squared Error Curves in Temporal Difference Learning
Authors: Satinder P. Singh, Peter Dayan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  calculated  analytical  expressions  for  how  the  bias  and  variance of the estimators provided by various temporal difference  value estimation algorithms change with offline updates over trials  in absorbing Markov chains using lookup table representations.  We  illustrate  classes  of learning curve  behavior in various  chains,  and  show the manner in which TD is sensitive to the choice of its step(cid:173) size  and eligibility trace parameters.
Dynamics of Training
Authors: Siegfried Bös, Manfred Opper
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A new method to calculate the full training process of a neural net(cid:173) work is  introduced.  No sophisticated methods like the replica trick  are  used.  The results  are directly  related  to  the actual number of  training  steps.  Some  results  are  presented  here,  like  the  maximal  learning rate, an exact description of early stopping, and the neces(cid:173) sary number of training steps.  Further problems can be addressed  with  this approach.
A Constructive RBF Network for Writer Adaptation
Authors: John C. Platt, Nada Matic
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This  paper  discusses  a  fairly  general  adaptation  algorithm  which  augments a standard neural network  to increase  its recognition  ac(cid:173) curacy  for  a  specific  user.  The  basis for  the  algorithm is  that  the  output of a neural network is characteristic of the input, even when  the  output  is  incorrect.  We  exploit  this  characteristic  output  by  using  an  Output  Adaptation  Module  (OAM)  which  maps this out(cid:173) put  into  the  correct  user-dependent  confidence  vector.  The  OAM  is  a  simplified  Resource  Allocating  Network  which  constructs  ra(cid:173) dial  basis  functions  on-line.  We  applied  the  OAM  to  construct  a  writer-adaptive  character  recognition  system  for  on-line  hand(cid:173) printed  characters.  The  OAM  decreases  the  word  error  rate  on  a  test  set  by  an  average  of 45%,  while  creating  only  3  to  25  basis  functions for  each  writer  in  the  test  set.
MLP Can Provably Generalize Much Better than VC-bounds Indicate
Authors: Adam Kowalczyk, Herman L. Ferrá
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Results  of a  study of the  worst  case  learning  curves  for  a  partic(cid:173) ular class of probability  distribution on input  space  to MLP  with  hard  threshold  hidden  units are  presented.  It is  shown in  partic(cid:173) ular,  that  in  the  thermodynamic  limit  for  scaling  by  the  number  of connections to the first  hidden layer, although the true learning  curve behaves as  ~ a-I for  a  ~ 1, its VC-dimension  based bound  is trivial  (= 1)  and  its VC-entropy bound is trivial for  a  ::;  6.2.  It  is also shown that bounds following the true learning curve can be  derived  from  a  formalism  based on the density of error patterns.
3D Object Recognition: A Model of View-Tuned Neurons
Authors: Emanuela Bricolo, Tomaso Poggio, Nikos K. Logothetis
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  1990  Poggio and  Edelman proposed  a  view-based  model of ob(cid:173) ject recognition that accounts for several psychophysical properties  of certain  recognition  tasks.  The model predicted  the existence  of  view-tuned  and view-invariant units,  that were  later found  by  Lo(cid:173) gothetis  et  al.  (Logothetis  et  al.,  1995)  in  IT  cortex  of monkeys  trained  with views  of specific  paperclip  objects.  The model,  how(cid:173) ever,  does  not specify  the inputs to the view-tuned  units and their  internal  organization.  In  this  paper  we  propose  a  model of these  view-tuned  units  that  is  consistent  with  physiological  data from  single cell  responses.
Learning Bayesian Belief Networks with Neural Network Estimators
Authors: Stefano Monti, Gregory F. Cooper
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  this  paper  we  propose  a  method  for  learning  Bayesian  belief  networks  from  data.  The  method  uses  artificial  neural  networks  as  probability estimators, thus  avoiding the need for  making prior  assumptions on the nature of the probability distributions govern(cid:173) ing  the relationships  among the  participating variables.  This new  method has  the potential for  being  applied to domains containing  both discrete  and continuous variables  arbitrarily distributed.  We  compare  the  learning  performance  of this  new  method  with  the  performance  of the  method  proposed  by  Cooper  and  Herskovits  in  [7].  The experimental results  show  that,  although  the  learning  scheme based on the use  of ANN  estimators is  slower, the learning  accuracy  of the two methods is  comparable.  Category:  Algorithms and Architectures.
Approximate Solutions to Optimal Stopping Problems
Authors: John N. Tsitsiklis, Benjamin Van Roy
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  propose and analyze an algorithm that approximates solutions  to the problem of optimal stopping in a discounted irreducible ape(cid:173) riodic  Markov  chain.  The  scheme  involves  the  use  of linear  com(cid:173) binations  of  fixed  basis  functions  to  approximate  a  Q-function.  The weights  of the  linear  combination  are incrementally  updated  through  an iterative process similar  to  Q-Iearning,  involving sim(cid:173) ulation of the underlying  Markov chain.  Due to space limitations,  we  only provide an overview of a  proof of convergence  (with prob(cid:173) ability 1)  and bounds on the approximation error.  This is  the first  theoretical result  that establishes the soundness  of a  Q-Iearning(cid:173) like  algorithm  when  combined  with  arbitrary  linear  function  ap(cid:173) proximators  to  solve  a  sequential  decision  problem.  Though  this  paper focuses  on  the case of finite  state spaces, the results extend  naturally to continuous and unbounded state spaces, which are ad(cid:173) dressed in a forthcoming full-length  paper.
An Orientation Selective Neural Network for Pattern Identification in Particle Detectors
Authors: Halina Abramowicz, David Horn, Ury Naftaly, Carmit Sahar-Pikielny
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  present  an  algorithm for  identifying linear  patterns  on  a  two(cid:173) dimensional lattice based on the concept of an orientation selective  cell,  a  concept  borrowed  from  neurobiology  of vision.  Construct(cid:173) ing  a  multi-layered  neural  network  with  fixed  architecture  which  implements orientation selectivity,  we  define  output elements cor(cid:173) responding  to different  orientations,  which  allow  us  to make  a  se(cid:173) lection  decision.  The  algorithm takes into account  the granularity  of the lattice as well  as the presence of noise and inefficiencies.  The  method  is  applied  to  a  sample  of data  collected  with  the  ZEUS  detector  at  HERA  in  order  to  identify  cosmic  muons  that  leave  a  linear  pattern  of signals  in  the  segmented  calorimeter.  A  two  dimensional  representation  of the  relevant  part  of the  detector  is  used.  The  algorithm performs  very  well.  Given  its  architecture,  this system  becomes a  good candidate for fast  pattern recognition  in  parallel processing  devices.
Early Brain Damage
Authors: Volker Tresp, Ralph Neuneier, Hans-Georg Zimmermann
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Optimal Brain Damage  (OBD)  is  a  method for  reducing  the num(cid:173) ber of weights in a  neural network.  OBD estimates the increase  in  cost  function  if weights  are  pruned  and  is  a  valid  approximation  if the learning algorithm has converged  into a  local minimum.  On  the  other hand it is  often desirable  to terminate the learning pro(cid:173) cess  before  a  local  minimum is  reached  (early  stopping).  In  this  paper  we  show  that  OBD  estimates  the  increase  in cost  function  incorrectly  if the network is  not in a local minimum.  We also show  how  OBD  can  be  extended  such  that  it  can  be  used  in  connec(cid:173) tion  with  early  stopping.  We  call  this  new  approach  Early  Brain  Damage,  EBD.  EBD also allows to revive  already pruned weights.  We  demonstrate  the  improvements achieved  by  EBD  using  three  publicly available data sets.
Text-Based Information Retrieval Using Exponentiated Gradient Descent
Authors: Ron Papka, James P. Callan, Andrew G. Barto
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The following  investigates  the  use  of single-neuron  learning  algo(cid:173) rithms  to improve  the  performance of text-retrieval  systems  that  accept  natural-language  queries.  A  retrieval  process  is  explained  that transforms the natural-language query into the query syntax  of a real retrieval system:  the initial query is expanded using statis(cid:173) tical and learning techniques and is then used for document ranking  and binary classification.  The results  of experiments suggest that  Kivinen and Warmuth's Exponentiated Gradient Descent learning  algorithm works significantly better than previous approaches.
An Analog Implementation of the Constant Average Statistics Constraint For Sensor Calibration
Authors: John G. Harris, Yu-Ming Chiang
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  use  the  constant  statistics  constraint  to calibrate an  array  of  sensors that contains gain and offset variations.  This algorithm has  been mapped to analog hardware and designed and fabricated with  a 2um CMOS technology.  Measured results from the chip show that  the system achieves invariance to gain and offset  variations of the  input signal.
Unsupervised Learning by Convex and Conic Coding
Authors: Daniel D. Lee, H. Sebastian Seung
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Unsupervised  learning algorithms  based  on  convex  and  conic  en(cid:173) coders are proposed.  The encoders find  the closest convex or conic  combination of basis vectors to the input.  The learning algorithms  produce basis vectors that minimize the reconstruction error of the  encoders.  The convex  algorithm develops  locally  linear models  of  the  input,  while  the  conic  algorithm  discovers  features.  Both  al(cid:173) gorithms are used to model handwritten digits and compared with  vector quantization and principal component analysis.  The neural  network implementations involve feedback connections that project  a  reconstruction back to the input layer.
Reconstructing Stimulus Velocity from Neuronal Responses in Area MT
Authors: Wyeth Bair, James R. Cavanaugh, J. Anthony Movshon
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  employed  a  white-noise velocity  signal to study the dynamics  of the response of single neurons in the cortical area MT to visual  motion.  Responses were quantified using reverse correlation, opti(cid:173) mal linear reconstruction filters,  and reconstruction signal-to-noise  ratio  (SNR).  The  SNR and  lower  bound estimates  of information  rate were  lower  than we  expected.  Ninety percent of the informa(cid:173) tion was transmitted below  18 Hz, and the highest lower bound on  bit rate was  12  bits/so  A  simulated opponent motion energy sub(cid:173) unit with Poisson spike statistics was  able to out-perform the MT  neurons.  The temporal integration window, measured from  the re(cid:173) verse  correlation half-width,  ranged  from  30-90 ms.  The window  was  narrower  when  a  stimulus  moved  faster,  but  did  not  change  when temporal frequency  was  held constant.
Viewpoint Invariant Face Recognition using Independent Component Analysis and Attractor Networks
Authors: Marian Stewart Bartlett, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  explored  two  approaches  to  recogmzmg  faces  across  changes in pose.  First, we developed a representation of face images  based on independent  component analysis  (ICA)  and compared it  to  a  principal  component  analysis  (PCA)  representation for  face  recognition.  The  ICA  basis  vectors  for  this  data set  were  more  spatially local than the PCA basis vectors and the ICA representa(cid:173) tion had greater invariance to changes in pose.  Second, we present  a  model  for  the  development  of  viewpoint  invariant  responses  to  faces  from  visual experience  in a  biological system.  The temporal  continuity  of natural  visual  experience  was  incorporated  into  an  attractor network model  by  Hebbian learning following  a  lowpass  temporal  filter  on  unit  activities.  When  combined  with  the  tem(cid:173) poral filter,  a  basic  Hebbian  update rule  became  a  generalization  of Griniasty  et  al.  (1993),  which  associates  temporally  proximal  input patterns into basins of attraction.  The system acquired rep(cid:173) resentations of faces  that were largely independent of pose. 
1 
Independent  component representations of faces 
Important  advances  in face  recognition  have  employed  forms  of principal  compo(cid:173) nent analysis, which  considers only second-order moments of the input  (Cottrell &  Metcalfe,  1991;  Turk  &  Pentland  1991).  Independent  component  analysis  (ICA)  is  a  generalization of principal component  analysis  (PCA),  which  decorrelates the  higher-order moments of the input  (Comon,  1994).  In a task such  as face  recogni(cid:173) tion,  much of the important information is contained in the high-order statistics of  the images.  A representational basis in which the high-order statistics are decorre(cid:173) lated may  be more powerful for face recognition than one in which  only the second  order statistics are decorrelated, as in PCA representations.  We compared an ICA(cid:173) based  representation  to  a  PCA-based  representation  for  recognizing  faces  across  changes in pose. 
818 
M. S.  Bartlett and T.  J.  Sejnowski
On the Effect of Analog Noise in Discrete-Time Analog Computations
Authors: Wolfgang Maass, Pekka Orponen
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  introduce a  model  for  noise-robust  analog  computations  with  discrete  time  that is  flexible  enough  to cover  the  most  important  concrete  cases,  such  as  computations  in  noisy  analog  neural  nets  and networks of noisy spiking neurons.  We  show that the presence  of arbitrarily small  amounts of analog noise  reduces the  power  of  analog  computational  models  to  that  of  finite  automata,  and  we  also  prove  a  new  type  of  upper  bound  for  the  VC-dimension  of  computational models with  analog noise.
Combinations of Weak Classifiers
Authors: Chuanyi Ji, Sheng Ma
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: To obtain classification systems with both good generalization per(cid:173) formance  and  efficiency  in  space  and  time,  we  propose  a  learning  method based on combinations of weak classifiers,  where weak clas(cid:173) sifiers are linear classifiers (perceptrons) which can do a little better  than making random guesses.  A randomized algorithm is  proposed  to find  the weak classifiers.  They· are then combined through a  ma(cid:173) jority vote.  As  demonstrated through systematic experiments,  the  method developed is able to obtain combinations of weak classifiers  with  good  generalization  performance  and  a  fast  training  time on  a  variety of test  problems and real  applications.
Extraction of Temporal Features in the Electrosensory System of Weakly Electric Fish
Authors: Fabrizio Gabbiani, Walter Metzner, Ralf Wessel, Christof Koch
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The encoding of random time-varying stimuli in single spike trains  of electrosensory neurons in the weakly electric fish Eigenmannia  was investigated using methods of statistical signal processing. At  the first stage of the electrosensory system, spike trains were found  to encode faithfully the detailed time-course of random stimuli,  while at the second stage neurons responded specifically to features  in the temporal waveform of the stimulus. Therefore stimulus infor(cid:173) mation is processed at the second stage of the electrosensory system  by extracting temporal features from the faithfully preserved image  of the environment sampled at the first stage.
A Model of Recurrent Interactions in Primary Visual Cortex
Authors: Emanuel Todorov, Athanassios Siapas, David Somers
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  general  feature  of the  cerebral  cortex  is  its  massive  intercon(cid:173) nectivity  - it  has  been  estimated  anatomically  [19]  that  cortical  neurons  receive  upwards  of 5,000 synapses,  the  majority of which  originate  from  other  nearby  cortical  neurons.  Numerous  experi(cid:173) ments in primary visual cortex (VI)  have revealed strongly nonlin(cid:173) ear interactions between stimulus elements which activate classical  and  non-classical  receptive  field  regions.  Recurrent  cortical  con(cid:173) nections  likely  contribute  substantially  to  these  effects.  However,  most  theories  of visual  processing  have  either  assumed  a  feedfor(cid:173) ward  processing  scheme  [7],  or have used  recurrent  interactions  to  account  for  isolated  effects  only  [1,  16,  18].  Since  nonlinear  sys(cid:173) tems  cannot  in  general  be  taken  apart  and  analyzed  in  pieces,  it  is  not  clear  what  one  learns  by  building  a  recurrent  model  that  only  accounts  for  one,  or  very  few  phenomena.  Here  we  develop  a  relatively simple model of recurrent  interactions  in  VI,  that  re(cid:173) flects  major anatomical and  physiological features  of intracortical  connectivity,  and  simultaneously accounts for  a  wide range of phe(cid:173) nomena observed  physiologically.  All  phenomena we  address  are  strongly nonlinear,  and cannot be  explained by  linear feedforward  models.
On a Modification to the Mean Field EM Algorithm in Factorial Learning
Authors: A. P. Dunmur, D. M. Titterington
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  modification  is  described  to  the  use  of mean  field  approxima(cid:173) tions in the E step of EM algorithms for  analysing data from latent  structure models, as described by Ghahramani (1995), among oth(cid:173) ers.  The modification involves second-order Taylor approximations  to expectations computed in the E step.  The potential benefits of  the method are illustrated using very simple latent profile models.
VLSI Implementation of Cortical Visual Motion Detection Using an Analog Neural Computer
Authors: Ralph Etienne-Cummings, Jan Van der Spiegel, Naomi Takahashi, Alyssa Apsel, Paul Mueller
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Two dimensional image motion detection neural networks have been  implemented using a general purpose analog neural computer.  The  neural circuits perform spatiotemporal feature extraction based on the  cortical motion detection model of Adelson and Bergen. The neural  computer provides the neurons, synapses and synaptic time-constants  required to realize the model in VLSI hardware. Results show that  visual motion estimation can be implemented with simple sum-and(cid:173) threshold neural hardware with temporal computational capabilities.  The neural circuits compute general 20 visual motion in real-time.
Local Bandit Approximation for Optimal Learning Problems
Authors: Michael O. Duff, Andrew G. Barto
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  general,  procedures  for  determining  Bayes-optimal  adaptive  controls  for  Markov  decision  processes  (MDP's)  require  a  pro(cid:173) hibitive  amount  of  computation-the  optimal  learning  problem  is  intractable.  This  paper  proposes  an  approximate  approach  in  which bandit processes are used to model, in a certain "local" sense,  a given MDP. Bandit processes constitute an important subclass of  MDP's,  and  have  optimal learning  strategies  (defined  in  terms  of  Gittins indices)  that can be  computed relatively  efficiently.  Thus,  one  scheme  for  achieving  approximately-optimal learning  for  gen(cid:173) eral MDP's proceeds by taking actions suggested by strategies that  are  optimal with respect  to local  bandit  models.
Learning Appearance Based Models: Mixtures of Second Moment Experts
Authors: Christoph Bregler, Jitendra Malik
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes a new technique for object recognition based on learning  appearance  models.  The  image is  decomposed  into  local  regions  which  are  described  by  a  new  texture  representation  called  "Generalized  Second  Mo(cid:173) ments"  that are  derived  from the  output of multiscale,  multiorientation filter  banks.  Class-characteristic local texture features and their global composition  is learned by a hierarchical mixture of experts architecture (Jordan &  Jacobs).  The  technique  is  applied  to  a  vehicle  database  consisting  of 5  general  car  categories  (Sedan,  Van  with back-doors, Van  without back-doors, old Sedan,  and Volkswagen Bug).  This  is  a difficult problem with considerable in-class  variation.  The new  technique has  a 6.5% misclassification rate, compared to  eigen-images which give  17.4% misclassification rate,  and nearest neighbors  which give  15 .7% misclassification rate.
Why did TD-Gammon Work?
Authors: Jordan B. Pollack, Alan D. Blair
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Although TD-Gammon is one of the major successes in machine learn(cid:173) ing,  it has  not led to  similar impressive breakthroughs  in  temporal dif(cid:173) ference  learning  for  other applications  or even  other games.  We  were  able  to  replicate  some  of the  success  of TD-Gammon,  developing  a  competitive evaluation function on a 4000 parameter feed-forward neu(cid:173) ral network, without using back-propagation, reinforcement or temporal  difference learning methods. Instead we apply simple hill-climbing in  a  relative fitness  environment.  These results and further  analysis  suggest  that the surprising success of Tesauro's program had more to do with the  co-evolutionary  structure of the  learning task and  the  dynamics  of the  backgammon game itself.
Time Series Prediction using Mixtures of Experts
Authors: Assaf J. Zeevi, Ron Meir, Robert J. Adler
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  consider  the  problem  of  prediction  of stationary  time  series,  using the architecture known as  mixtures of experts  (MEM).  Here  we  suggest  a  mixture  which  blends  several autoregressive models.  This study focuses  on  some  theoretical foundations  of the  predic(cid:173) tion  problem  in  this  context.  More  precisely,  it  is  demonstrated  that this model is  a  universal approximator,  with  respect to learn(cid:173) ing the  unknown  prediction function .  This statement is  strength(cid:173) ened  as  upper  bounds  on the  mean  squared error are  established.  Based on these results it is  possible to compare the MEM to other  families of models  (e.g.,  neural networks and state dependent mod(cid:173) els).  It is  shown  that  a  degenerate  version  of the  MEM  is  in  fact  equivalent to a  neural  network, and  the  number  of experts  in  the  architecture  plays  a  similar role  to the  number  of hidden  units  in  the latter model.
MIMIC: Finding Optima by Estimating Probability Densities
Authors: Jeremy S. De Bonet, Charles Lee Isbell Jr., Paul A. Viola
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  many optimization problems,  the structure of solutions reflects  complex relationships between  the different  input parameters.  For  example, experience may tell us  that certain parameters are closely  related  and  should  not  be  explored  independently.  Similarly, ex(cid:173) perience  may establish  that  a  subset  of parameters must  take  on  particular  values.  Any  search  of the  cost  landscape  should  take  advantage of these relationships.  We  present  MIMIC,  a framework  in which  we  analyze the global structure of the optimization land(cid:173) scape.  A  novel  and  efficient  algorithm for  the  estimation of this  structure is  derived.  We  use  knowledge of this structure to guide a  randomized search  through the solution space  and,  in  turn,  to re(cid:173) fine our estimate ofthe structure.  Our technique obtains significant  speed  gains over  other randomized optimization procedures.
Neural Network Models of Chemotaxis in the Nematode Caenorhabditis Elegans
Authors: Thomas C. Ferrée, Ben A. Marcotte, Shawn R. Lockery
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  train recurrent networks  to control chemotaxis in a  computer  model of the nematode  C.  elegans.  The model  presented  is  based  closely on the body mechanics, behavioral analyses, neuroanatomy  and neurophysiology of  C.  elegans,  each imposing constraints rel(cid:173) evant  for  information  processing.  Simulated  worms  moving  au(cid:173) tonomously in simulated chemical environments display  a  variety  of chemotaxis strategies similar to those of biological worms.
GTM: A Principled Alternative to the Self-Organizing Map
Authors: Christopher M. Bishop, Markus Svensén, Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Self-Organizing Map  (SOM)  algorithm  has  been  extensively  studied and has  been  applied with considerable success to a  wide  variety of problems.  However, the algorithm is derived from heuris(cid:173) tic  ideas  and  this  leads  to a  number of significant  limitations.  In  this  paper,  we  consider  the  problem  of  modelling  the  probabil(cid:173) ity  density  of  data  in  a  space  of several  dimensions  in  terms  of  a  smaller number of latent,  or hidden,  variables.  We  introduce  a  novel form  of latent variable model,  which  we  call the GTM algo(cid:173) rithm (for  Generative  Topographic  Mapping),  which allows general  non-linear  transformations  from  latent  space  to  data  space,  and  which  is  trained  using  the  EM  (expectation-maximization)  algo(cid:173) rithm.  Our approach overcomes the limitations of the SOM, while  introducing no significant disadvantages.  We demonstrate the per(cid:173) formance of the GTM algorithm on simulated data from flow  diag(cid:173) nostics for  a  multi-phase oil pipeline.
Support Vector Method for Function Approximation, Regression Estimation and Signal Processing
Authors: Vladimir Vapnik, Steven E. Golowich, Alex J. Smola
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Support  Vector  (SV)  method  was  recently  proposed  for  es(cid:173) timating  regressions,  constructing  multidimensional  splines,  and  solving linear operator equations  [Vapnik,  1995].  In  this  presenta(cid:173) tion we report results of applying the SV method to these problems.
Smoothing Regularizers for Projective Basis Function Networks
Authors: John E. Moody, Thorsteinn S. Rögnvaldsson
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Smoothing regularizers for radial basis functions have been studied extensively,  but no general smoothing regularizers for projective basis junctions (PBFs), such  as  the  widely-used  sigmoidal  PBFs,  have  heretofore  been  proposed.  We  de(cid:173) rive  new  classes of algebraically-simple  mH'-order smoothing regularizers  for  networks of the form  f(W, x)  =  L7=1 Ujg [x T  Vj + Vjol  + uo,  with general  projective basis functions g[.].  These regularizers are: 
Ra(W,m)  =  LU;lIvjIl2m-1  GlobalForm
Bangs, Clicks, Snaps, Thuds and Whacks: An Architecture for Acoustic Transient Processing
Authors: Fernando J. Pineda, Gert Cauwenberghs, R. Timothy Edwards
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We propose  a  neuromorphic  architecture  for  real-time  processing  of 
acoustic transients in analog VLSI.  We show how judicious normalization 
of a time-frequency signal allows an elegant and robust implementation 
of a correlation algorithm. The algorithm uses binary multiplexing instead 
of analog-analog  multiplication.  This  removes  the  need  for  analog 
storage and analog-multiplication.  Simulations  show that the resulting 
algorithm has the same out-of-sample classification performance (-93% 
correct) as a baseline template-matching algorithm.
Salient Contour Extraction by Temporal Binding in a Cortically-based Network
Authors: Shih-Cheng Yen, Leif H. Finkel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: It has been suggested that long-range intrinsic  connections  in  striate cortex  may  play  a  role  in  contour  extraction  (Gilbert  et  aI.,  1996).  A  number  of  recent  physiological  and  psychophysical  studies  have  examined  the  possible  role  of  long range connections in the modulation  of contrast detection  thresholds  (Polat  and Sagi,  1993,1994; Kapadia et aI.,  1995; Kovacs and Julesz,  1994) and various  pre-attentive  detection  tasks  (Kovacs  and  Julesz,  1993;  Field  et aI.,  1993).  We  have  developed  a  network  architecture based  on  the  anatomical  connectivity  of  striate cortex,  as  well  as  the  temporal  dynamics  of neuronal  processing,  that  is  able to reproduce the observed experimental results. The  network  has  been  tested  on  real  images  and  has  applications  in  terms  of identifying  salient  contours  in  automatic image processing systems.
Learning Decision Theoretic Utilities through Reinforcement Learning
Authors: Magnus Stensmo, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Probability models can be used to predict outcomes and compensate for  missing data, but even a perfect model cannot be used to make decisions  unless the utility of the outcomes, or preferences between them, are also  provided.  This arises in many real-world problems, such as medical di(cid:173) agnosis,  where the cost of the test as  well as  the expected improvement  in the outcome must be considered. Relatively little work has been done  on learning the utilities of outcomes for optimal decision making. In this  paper, we show how temporal-difference reinforcement learning (TO(A»  can be used to determine decision theoretic utilities within the context of  a mixture model and apply this new approach to a problem in medical di(cid:173) agnosis.  TO( A) learning of utilities reduces the number of tests that have  to be done to achieve the same level of performance compared with the  probability model alone, which results in significant cost savings and in(cid:173) creased efficiency.
Spatial Decorrelation in Orientation Tuned Cortical Cells
Authors: Alexander Dimitrov, Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  this  paper  we  propose  a  model  for  the  lateral  connectivity  of  orientation-selective cells in the visual cortex based on information(cid:173) theoretic  considerations.  We study the properties of the input sig(cid:173) nal  to  the  visual  cortex  and  find  new  statistical  structures  which  have not been processed in the retino-geniculate pathway.  Applying  the idea that the system optimizes the  representation  of incoming  signals,  we  derive  the  lateral connectivity that will  achieve  this for  a  set  of local orientation-selective patches,  as  well  as  the  complete  spatial structure of a layer of such patches.  We compare the results  with various  physiological measurements.
Are Hopfield Networks Faster than Conventional Computers?
Authors: Ian Parberry, Hung-Li Tseng
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: It is shown that conventional computers can be exponentiallx faster  than planar Hopfield  networks:  although there  are planar Hopfield  networks that take exponential time to converge,  a stable state of an  arbitrary  planar Hopfield  network  can be found  by  a  conventional  computer  in  polynomial  time.  The  theory  of 'P.cS-completeness  gives  strong evidence  that such  a separation is  unlikely for  nonpla(cid:173) nar  Hopfield  networks,  and it is  demonstrated  that  this is  also  the  case  for  several  restricted  classes  of nonplanar  Hopfield  networks,  including  those  who  interconnection  graphs are  the class  of bipar(cid:173) tite graphs,  graphs of degree  3, the dual of the knight's graph,  the  8-neighbor mesh,  the hypercube ,  the butterfly,  the cube-connected  cycles,  and the shuffle-exchange  graph.
Promoting Poor Features to Supervisors: Some Inputs Work Better as Outputs
Authors: Rich Caruana, Virginia R. de Sa
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In supervised  learning there is  usually  a  clear  distinction  between  inputs  and  outputs - inputs are  what  you  will  measure,  outputs  are  what  you  will  predict  from  those  measurements.  This  paper  shows  that the distinction between  inputs  and  outputs is  not this  simple.  Some  features  are  more  useful  as  extra  outputs  than  as  inputs.  By using a feature  as  an output we  get  more than just the  case  values but can. learn a  mapping from  the other inputs to that  feature.  For many features  this mapping may be more useful  than  the  feature  value  itself.  We  present  two  regression  problems  and  one  classification problem  where  performance improves  if features  that  could  have  been  used  as  inputs  are  used  as  extra  outputs  instead.  This result is  surprising since  a feature used  as  an output  is  not  used  during testing.
Hidden Markov Decision Trees
Authors: Michael I. Jordan, Zoubin Ghahramani, Lawrence K. Saul
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  study  a  time  series  model  that  can  be  viewed  as  a  decision  tree  with Markov  temporal structure.  The model is  intractable for  exact  calculations,  thus we  utilize  variational approximations.  We  consider  three different  distributions for  the approximation: one in  which the Markov calculations are performed exactly and the layers  of the  decision  tree  are  decoupled,  one  in  which  the  decision  tree  calculations are performed exactly and the time steps of the Markov  chain are decoupled,  and one in which  a  Viterbi-like assumption is  made to  pick out  a  single  most  likely  state sequence.  We  present  simulation results for  artificial data and the Bach chorales.
Representing Face Images for Emotion Classification
Authors: Curtis Padgett, Garrison W. Cottrell
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  compare  the  generalization  performance of three  distinct  rep(cid:173) resentation schemes for facial emotions using a  single classification  strategy  (neural  network).  The face  images presented  to the  clas(cid:173) sifiers  are  represented  as:  full  face  projections  of the  dataset  onto  their  eigenvectors  (eigenfaces);  a  similar projection  constrained  to  eye  and  mouth  areas  (eigenfeatures);  and  finally  a  projection  of  the eye and mouth areas onto the eigenvectors obtained from 32x32  random image patches from the dataset.  The latter system achieves  86%  generalization on novel face  images  (individuals the networks  were  not trained  on)  drawn from  a  database in  which  human sub(cid:173) jects  consistently  identify a single emotion for  the face .
Separating Style and Content
Authors: Joshua B. Tenenbaum, William T. Freeman
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  seek to analyze and manipulate two factors,  which we  call style  and content,  underlying a set of observations.  We fit  training data  with bilinear models which explicitly represent the two-factor struc(cid:173) ture.  These models can adapt easily during testing to new  styles or  content,  allowing us  to solve three general tasks:  extrapolation of a  new style to unobserved content;  classification of content observed  in  a  new  style;  and  translation of new  content  observed  in  a  new  style.  For classification, we embed bilinear models in a probabilistic  framework,  Separable  Mixture  Models  (SMMsj,  which  generalizes  earlier  work  on  factorial  mixture  models  [7,  3].  Significant  per(cid:173) formance  improvement  on  a  benchmark  speech  dataset  shows  the  benefits  of our approach.
Contour Organisation with the EM Algorithm
Authors: José A. F. Leite, Edwin R. Hancock
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes how the early visual process of contour organ(cid:173) isation  can  be  realised  using  the  EM  algorithm.  The  underlying  computational representation is  based on fine  spline coverings.  Ac(cid:173) cording  to our  EM  approach  the  adjustment  of spline  parameters  draws  on  an  iterative  weighted  least-squares  fitting  process.  The  expectation  step  of our  EM  procedure  computes  the  likelihood  of  the data using a mixture model defined over the set of spline cover(cid:173) ings.  These splines  are limited in  their  spatial extent  using  Gaus(cid:173) sian windowing functions.  The maximisation of the likelihood leads  to a set of linear equations in the spline parameters which solve the  weighted least squares problem.  We  evaluate the technique  on  the  localisation of road structures in  aerial infra-red images.
Combining Neural Network Regression Estimates with Regularized Linear Weights
Authors: Christopher J. Merz, Michael J. Pazzani
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: When  combining a  set  of learned  models  to form  an  improved es(cid:173) timator, the  issue  of redundancy  or  multicollinearity in the set  of  models  must  be  addressed.  A  progression  of existing  approaches  and  their  limitations with  respect  to the  redundancy  is  discussed.  A  new  approach,  PCR ,  based  on  principal  components  regres(cid:173) sion  is  proposed to address  these  limitations.  An evaluation of the  new  approach  on  a  collection  of domains  reveals  that:  1)  PCR  was the most robust combination method as the redundancy of the  learned models increased,  2)  redundancy could be handled without  eliminating any of the learned models, and 3) the principal compo(cid:173) nents of the learned models provided a  continuum of "regularized"  weights from  which  PCR * could choose.