mtybilly/Instruct2DS huggingface dataset:
Viewpoint Invariant Face Recognition using Independent Component Analysis and Attractor Networks
Authors: Terrence J. Sejnowski Marian Stewart Bartlett
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  explored  two  approaches  to  recogmzmg  faces  across  changes in pose.  First, we developed a representation of face images  based on independent  component analysis  (ICA)  and compared it  to  a  principal  component  analysis  (PCA)  representation for  face  recognition.  The  ICA  basis  vectors  for  this  data set  were  more  spatially local than the PCA basis vectors and the ICA representa(cid:173) tion had greater invariance to changes in pose.  Second, we present  a  model  for  the  development  of  viewpoint  invariant  responses  to  faces  from  visual experience  in a  biological system.  The temporal  continuity  of natural  visual  experience  was  incorporated  into  an  attractor network model  by  Hebbian learning following  a  lowpass  temporal  filter  on  unit  activities.  When  combined  with  the  tem(cid:173) poral filter,  a  basic  Hebbian  update rule  became  a  generalization  of Griniasty  et  al.  (1993),  which  associates  temporally  proximal  input patterns into basins of attraction.  The system acquired rep(cid:173) resentations of faces  that were largely independent of pose. 
1 
Independent  component representations of faces 
Important  advances  in face  recognition  have  employed  forms  of principal  compo(cid:173) nent analysis, which  considers only second-order moments of the input  (Cottrell &  Metcalfe,  1991;  Turk  &  Pentland  1991).  Independent  component  analysis  (ICA)  is  a  generalization of principal component  analysis  (PCA),  which  decorrelates the  higher-order moments of the input  (Comon,  1994).  In a task such  as face  recogni(cid:173) tion,  much of the important information is contained in the high-order statistics of  the images.  A representational basis in which the high-order statistics are decorre(cid:173) lated may  be more powerful for face recognition than one in which  only the second  order statistics are decorrelated, as in PCA representations.  We compared an ICA(cid:173) based  representation  to  a  PCA-based  representation  for  recognizing  faces  across  changes in pose. 
818 
M. S.  Bartlett and T.  J.  Sejnowski
On the Effect of Analog Noise in Discrete-Time Analog Computations
Authors: Pekka Orponen Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  introduce a  model  for  noise-robust  analog  computations  with  discrete  time  that is  flexible  enough  to cover  the  most  important  concrete  cases,  such  as  computations  in  noisy  analog  neural  nets  and networks of noisy spiking neurons.  We  show that the presence  of arbitrarily small  amounts of analog noise  reduces the  power  of  analog  computational  models  to  that  of  finite  automata,  and  we  also  prove  a  new  type  of  upper  bound  for  the  VC-dimension  of  computational models with  analog noise.
Combinations of Weak Classifiers
Authors: Sheng Ma Chuanyi Ji
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: To obtain classification systems with both good generalization per(cid:173) formance  and  efficiency  in  space  and  time,  we  propose  a  learning  method based on combinations of weak classifiers,  where weak clas(cid:173) sifiers are linear classifiers (perceptrons) which can do a little better  than making random guesses.  A randomized algorithm is  proposed  to find  the weak classifiers.  They· are then combined through a  ma(cid:173) jority vote.  As  demonstrated through systematic experiments,  the  method developed is able to obtain combinations of weak classifiers  with  good  generalization  performance  and  a  fast  training  time on  a  variety of test  problems and real  applications.
Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient
Authors: Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  parameter  space  of neural  networks  has  a  Riemannian  met(cid:173) ric  structure.  The  natural  Riemannian  gradient  should  be  used  instead of the  conventional gradient, since  the former  denotes  the  true steepest descent direction of a loss function in the Riemannian  space.  The behavior of the  stochastic gradient learning  algorithm  is  much more effective if the natural gradient is  used.  The present  paper studies the information-geometrical structure of perceptrons  and  other  networks,  and  prove  that  the  on-line  learning  method  based  on  the  natural  gradient  is  asymptotically  as  efficient  as  the  optimal  batch  algorithm.  Adaptive  modification  of the  learning  constant is proposed and analyzed in terms of the Riemannian mea(cid:173) sure  and  is  shown  to  be  efficient.  The  natural  gradient  is  finally  applied to blind separation of mixtured independent signal sources. 
1
Extraction of Temporal Features in the Electrosensory System of Weakly Electric Fish
Authors: Walter Metzner, Ralf Wessel, Christof Koch Fabrizio Gabbiani
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The encoding of random time-varying stimuli in single spike trains  of electrosensory neurons in the weakly electric fish Eigenmannia  was investigated using methods of statistical signal processing. At  the first stage of the electrosensory system, spike trains were found  to encode faithfully the detailed time-course of random stimuli,  while at the second stage neurons responded specifically to features  in the temporal waveform of the stimulus. Therefore stimulus infor(cid:173) mation is processed at the second stage of the electrosensory system  by extracting temporal features from the faithfully preserved image  of the environment sampled at the first stage.
A Model of Recurrent Interactions in Primary Visual Cortex
Authors: Athanassios Siapas, David Somers Emanuel Todorov
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  general  feature  of the  cerebral  cortex  is  its  massive  intercon(cid:173) nectivity  - it  has  been  estimated  anatomically  [19]  that  cortical  neurons  receive  upwards  of 5,000 synapses,  the  majority of which  originate  from  other  nearby  cortical  neurons.  Numerous  experi(cid:173) ments in primary visual cortex (VI)  have revealed strongly nonlin(cid:173) ear interactions between stimulus elements which activate classical  and  non-classical  receptive  field  regions.  Recurrent  cortical  con(cid:173) nections  likely  contribute  substantially  to  these  effects.  However,  most  theories  of visual  processing  have  either  assumed  a  feedfor(cid:173) ward  processing  scheme  [7],  or have used  recurrent  interactions  to  account  for  isolated  effects  only  [1,  16,  18].  Since  nonlinear  sys(cid:173) tems  cannot  in  general  be  taken  apart  and  analyzed  in  pieces,  it  is  not  clear  what  one  learns  by  building  a  recurrent  model  that  only  accounts  for  one,  or  very  few  phenomena.  Here  we  develop  a  relatively simple model of recurrent  interactions  in  VI,  that  re(cid:173) flects  major anatomical and  physiological features  of intracortical  connectivity,  and  simultaneously accounts for  a  wide range of phe(cid:173) nomena observed  physiologically.  All  phenomena we  address  are  strongly nonlinear,  and cannot be  explained by  linear feedforward  models.
On a Modification to the Mean Field EM Algorithm in Factorial Learning
Authors: D. M. Titterington A. P. Dunmur
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  modification  is  described  to  the  use  of mean  field  approxima(cid:173) tions in the E step of EM algorithms for  analysing data from latent  structure models, as described by Ghahramani (1995), among oth(cid:173) ers.  The modification involves second-order Taylor approximations  to expectations computed in the E step.  The potential benefits of  the method are illustrated using very simple latent profile models.
VLSI Implementation of Cortical Visual Motion Detection Using an Analog Neural Computer
Authors: Jan Van der Spiegel, Naomi Takahashi, Alyssa Apsel, Paul Mueller Ralph Etienne-Cummings
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Two dimensional image motion detection neural networks have been  implemented using a general purpose analog neural computer.  The  neural circuits perform spatiotemporal feature extraction based on the  cortical motion detection model of Adelson and Bergen. The neural  computer provides the neurons, synapses and synaptic time-constants  required to realize the model in VLSI hardware. Results show that  visual motion estimation can be implemented with simple sum-and(cid:173) threshold neural hardware with temporal computational capabilities.  The neural circuits compute general 20 visual motion in real-time.
Local Bandit Approximation for Optimal Learning Problems
Authors: Andrew G. Barto Michael O. Duff
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  general,  procedures  for  determining  Bayes-optimal  adaptive  controls  for  Markov  decision  processes  (MDP's)  require  a  pro(cid:173) hibitive  amount  of  computation-the  optimal  learning  problem  is  intractable.  This  paper  proposes  an  approximate  approach  in  which bandit processes are used to model, in a certain "local" sense,  a given MDP. Bandit processes constitute an important subclass of  MDP's,  and  have  optimal learning  strategies  (defined  in  terms  of  Gittins indices)  that can be  computed relatively  efficiently.  Thus,  one  scheme  for  achieving  approximately-optimal learning  for  gen(cid:173) eral MDP's proceeds by taking actions suggested by strategies that  are  optimal with respect  to local  bandit  models.
Learning Appearance Based Models: Mixtures of Second Moment Experts
Authors: Jitendra Malik Christoph Bregler
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes a new technique for object recognition based on learning  appearance  models.  The  image is  decomposed  into  local  regions  which  are  described  by  a  new  texture  representation  called  "Generalized  Second  Mo(cid:173) ments"  that are  derived  from the  output of multiscale,  multiorientation filter  banks.  Class-characteristic local texture features and their global composition  is learned by a hierarchical mixture of experts architecture (Jordan &  Jacobs).  The  technique  is  applied  to  a  vehicle  database  consisting  of 5  general  car  categories  (Sedan,  Van  with back-doors, Van  without back-doors, old Sedan,  and Volkswagen Bug).  This  is  a difficult problem with considerable in-class  variation.  The new  technique has  a 6.5% misclassification rate, compared to  eigen-images which give  17.4% misclassification rate,  and nearest neighbors  which give  15 .7% misclassification rate.
Interpreting Images by Propagating Bayesian Beliefs
Authors: Yair Weiss
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A central theme of computational vision research  has been  the  re(cid:173) alization that reliable estimation of local scene  properties requires  propagating  measurements  across  the  image.  Many  authors  have  therefore  suggested  solving  vision  problems using  architectures  of  locally  connected  units updating their  activity in  parallel.  Unfor(cid:173) tunately, the convergence of traditional relaxation methods on such  architectures  has  proven  to  be  excruciatingly  slow  and  in  general  they  do  not  guarantee that the stable  point will  be  a  global mini(cid:173) mum.  In this paper we  show  that an  architecture  in  which  Bayesian  Be(cid:173) liefs  about  image  properties  are  propagated  between  neighboring  units  yields  convergence  times  which  are  several  orders  of magni(cid:173) tude faster  than traditional methods and  avoids  local  minima.  In  particular our architecture is  non-iterative in  the sense of Marr [5]:  at every  time step,  the  local estimates  at  a  given  location are  op(cid:173) timal given the information which  has already been  propagated to  that  location.  We  illustrate  the  algorithm's  performance  on  real  images and  compare it to several existing methods.
Why did TD-Gammon Work?
Authors: Alan D. Blair Jordan B. Pollack
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Although TD-Gammon is one of the major successes in machine learn(cid:173) ing,  it has  not led to  similar impressive breakthroughs  in  temporal dif(cid:173) ference  learning  for  other applications  or even  other games.  We  were  able  to  replicate  some  of the  success  of TD-Gammon,  developing  a  competitive evaluation function on a 4000 parameter feed-forward neu(cid:173) ral network, without using back-propagation, reinforcement or temporal  difference learning methods. Instead we apply simple hill-climbing in  a  relative fitness  environment.  These results and further  analysis  suggest  that the surprising success of Tesauro's program had more to do with the  co-evolutionary  structure of the  learning task and  the  dynamics  of the  backgammon game itself.
Time Series Prediction using Mixtures of Experts
Authors: Ron Meir, Robert J. Adler Assaf J. Zeevi
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  consider  the  problem  of  prediction  of stationary  time  series,  using the architecture known as  mixtures of experts  (MEM).  Here  we  suggest  a  mixture  which  blends  several autoregressive models.  This study focuses  on  some  theoretical foundations  of the  predic(cid:173) tion  problem  in  this  context.  More  precisely,  it  is  demonstrated  that this model is  a  universal approximator,  with  respect to learn(cid:173) ing the  unknown  prediction function .  This statement is  strength(cid:173) ened  as  upper  bounds  on the  mean  squared error are  established.  Based on these results it is  possible to compare the MEM to other  families of models  (e.g.,  neural networks and state dependent mod(cid:173) els).  It is  shown  that  a  degenerate  version  of the  MEM  is  in  fact  equivalent to a  neural  network, and  the  number  of experts  in  the  architecture  plays  a  similar role  to the  number  of hidden  units  in  the latter model.
MIMIC: Finding Optima by Estimating Probability Densities
Authors: Charles Lee Isbell Jr., Paul A. Viola Jeremy S. De Bonet
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  many optimization problems,  the structure of solutions reflects  complex relationships between  the different  input parameters.  For  example, experience may tell us  that certain parameters are closely  related  and  should  not  be  explored  independently.  Similarly, ex(cid:173) perience  may establish  that  a  subset  of parameters must  take  on  particular  values.  Any  search  of the  cost  landscape  should  take  advantage of these relationships.  We  present  MIMIC,  a framework  in which  we  analyze the global structure of the optimization land(cid:173) scape.  A  novel  and  efficient  algorithm for  the  estimation of this  structure is  derived.  We  use  knowledge of this structure to guide a  randomized search  through the solution space  and,  in  turn,  to re(cid:173) fine our estimate ofthe structure.  Our technique obtains significant  speed  gains over  other randomized optimization procedures.
Neural Network Models of Chemotaxis in the Nematode Caenorhabditis Elegans
Authors: Ben A. Marcotte, Shawn R. Lockery Thomas C. Ferrée
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  train recurrent networks  to control chemotaxis in a  computer  model of the nematode  C.  elegans.  The model  presented  is  based  closely on the body mechanics, behavioral analyses, neuroanatomy  and neurophysiology of  C.  elegans,  each imposing constraints rel(cid:173) evant  for  information  processing.  Simulated  worms  moving  au(cid:173) tonomously in simulated chemical environments display  a  variety  of chemotaxis strategies similar to those of biological worms.
GTM: A Principled Alternative to the Self-Organizing Map
Authors: Markus Svensén, Christopher K. I. Williams Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Self-Organizing Map  (SOM)  algorithm  has  been  extensively  studied and has  been  applied with considerable success to a  wide  variety of problems.  However, the algorithm is derived from heuris(cid:173) tic  ideas  and  this  leads  to a  number of significant  limitations.  In  this  paper,  we  consider  the  problem  of  modelling  the  probabil(cid:173) ity  density  of  data  in  a  space  of several  dimensions  in  terms  of  a  smaller number of latent,  or hidden,  variables.  We  introduce  a  novel form  of latent variable model,  which  we  call the GTM algo(cid:173) rithm (for  Generative  Topographic  Mapping),  which allows general  non-linear  transformations  from  latent  space  to  data  space,  and  which  is  trained  using  the  EM  (expectation-maximization)  algo(cid:173) rithm.  Our approach overcomes the limitations of the SOM, while  introducing no significant disadvantages.  We demonstrate the per(cid:173) formance of the GTM algorithm on simulated data from flow  diag(cid:173) nostics for  a  multi-phase oil pipeline.
Support Vector Method for Function Approximation, Regression Estimation and Signal Processing
Authors: Steven E. Golowich, Alex J. Smola Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Support  Vector  (SV)  method  was  recently  proposed  for  es(cid:173) timating  regressions,  constructing  multidimensional  splines,  and  solving linear operator equations  [Vapnik,  1995].  In  this  presenta(cid:173) tion we report results of applying the SV method to these problems.
Smoothing Regularizers for Projective Basis Function Networks
Authors: Thorsteinn S. Rögnvaldsson John E. Moody
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Smoothing regularizers for radial basis functions have been studied extensively,  but no general smoothing regularizers for projective basis junctions (PBFs), such  as  the  widely-used  sigmoidal  PBFs,  have  heretofore  been  proposed.  We  de(cid:173) rive  new  classes of algebraically-simple  mH'-order smoothing regularizers  for  networks of the form  f(W, x)  =  L7=1 Ujg [x T  Vj + Vjol  + uo,  with general  projective basis functions g[.].  These regularizers are: 
Ra(W,m)  =  LU;lIvjIl2m-1  GlobalForm
Bangs, Clicks, Snaps, Thuds and Whacks: An Architecture for Acoustic Transient Processing
Authors: Gert Cauwenberghs, R. Timothy Edwards Fernando J. Pineda
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We propose  a  neuromorphic  architecture  for  real-time  processing  of 
acoustic transients in analog VLSI.  We show how judicious normalization 
of a time-frequency signal allows an elegant and robust implementation 
of a correlation algorithm. The algorithm uses binary multiplexing instead 
of analog-analog  multiplication.  This  removes  the  need  for  analog 
storage and analog-multiplication.  Simulations  show that the resulting 
algorithm has the same out-of-sample classification performance (-93% 
correct) as a baseline template-matching algorithm.
Salient Contour Extraction by Temporal Binding in a Cortically-based Network
Authors: Leif H. Finkel Shih-Cheng Yen
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: It has been suggested that long-range intrinsic  connections  in  striate cortex  may  play  a  role  in  contour  extraction  (Gilbert  et  aI.,  1996).  A  number  of  recent  physiological  and  psychophysical  studies  have  examined  the  possible  role  of  long range connections in the modulation  of contrast detection  thresholds  (Polat  and Sagi,  1993,1994; Kapadia et aI.,  1995; Kovacs and Julesz,  1994) and various  pre-attentive  detection  tasks  (Kovacs  and  Julesz,  1993;  Field  et aI.,  1993).  We  have  developed  a  network  architecture based  on  the  anatomical  connectivity  of  striate cortex,  as  well  as  the  temporal  dynamics  of neuronal  processing,  that  is  able to reproduce the observed experimental results. The  network  has  been  tested  on  real  images  and  has  applications  in  terms  of identifying  salient  contours  in  automatic image processing systems.
Learning Decision Theoretic Utilities through Reinforcement Learning
Authors: Terrence J. Sejnowski Magnus Stensmo
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Probability models can be used to predict outcomes and compensate for  missing data, but even a perfect model cannot be used to make decisions  unless the utility of the outcomes, or preferences between them, are also  provided.  This arises in many real-world problems, such as medical di(cid:173) agnosis,  where the cost of the test as  well as  the expected improvement  in the outcome must be considered. Relatively little work has been done  on learning the utilities of outcomes for optimal decision making. In this  paper, we show how temporal-difference reinforcement learning (TO(A»  can be used to determine decision theoretic utilities within the context of  a mixture model and apply this new approach to a problem in medical di(cid:173) agnosis.  TO( A) learning of utilities reduces the number of tests that have  to be done to achieve the same level of performance compared with the  probability model alone, which results in significant cost savings and in(cid:173) creased efficiency.
Spatial Decorrelation in Orientation Tuned Cortical Cells
Authors: Jack D. Cowan Alexander Dimitrov
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  this  paper  we  propose  a  model  for  the  lateral  connectivity  of  orientation-selective cells in the visual cortex based on information(cid:173) theoretic  considerations.  We study the properties of the input sig(cid:173) nal  to  the  visual  cortex  and  find  new  statistical  structures  which  have not been processed in the retino-geniculate pathway.  Applying  the idea that the system optimizes the  representation  of incoming  signals,  we  derive  the  lateral connectivity that will  achieve  this for  a  set  of local orientation-selective patches,  as  well  as  the  complete  spatial structure of a layer of such patches.  We compare the results  with various  physiological measurements.
Sequential Tracking in Pricing Financial Options using Model Based and Neural Network Approaches
Authors: Mahesan Niranjan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper shows how the prices of option contracts traded in finan(cid:173) cial markets can be tracked sequentially by means of the Extended  Kalman Filter algorithm. I consider call and put option pairs with  identical strike price and time of maturity as a two output nonlin(cid:173) ear system. The Black-Scholes approach popular in Finance liter(cid:173) ature and the Radial Basis Functions neural network are used in  modelling the nonlinear system generating these observations. I  show how both these systems may be identified recursively using  the EKF algorithm. I present results of simulations on some FTSE  100 Index options data and discuss the implications of viewing the  pricing problem in this sequential manner.
Are Hopfield Networks Faster than Conventional Computers?
Authors: Hung-Li Tseng Ian Parberry
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: It is shown that conventional computers can be exponentiallx faster  than planar Hopfield  networks:  although there  are planar Hopfield  networks that take exponential time to converge,  a stable state of an  arbitrary  planar Hopfield  network  can be found  by  a  conventional  computer  in  polynomial  time.  The  theory  of 'P.cS-completeness  gives  strong evidence  that such  a separation is  unlikely for  nonpla(cid:173) nar  Hopfield  networks,  and it is  demonstrated  that  this is  also  the  case  for  several  restricted  classes  of nonplanar  Hopfield  networks,  including  those  who  interconnection  graphs are  the class  of bipar(cid:173) tite graphs,  graphs of degree  3, the dual of the knight's graph,  the  8-neighbor mesh,  the hypercube ,  the butterfly,  the cube-connected  cycles,  and the shuffle-exchange  graph.
Learning from Demonstration
Authors: Stefan Schaal
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: By  now  it is  widely  accepted  that learning  a  task  from  scratch,  i.e.,  without  any prior knowledge,  is a daunting  undertaking. Humans,  however,  rarely at(cid:173) tempt  to  learn  from  scratch.  They  extract  initial  biases  as  well  as  strategies  how  to  approach a learning problem from  instructions and/or demonstrations  of other  humans.  For  learning  control,  this  paper  investigates  how  learning  from  demonstration  can  be  applied  in  the  context of reinforcement  learning.  We  consider priming  the  Q-function,  the  value  function,  the  policy,  and  the  model of the task dynamics as possible areas where demonstrations can speed  up  learning.  In  general  nonlinear learning  problems,  only  model-based  rein(cid:173) forcement learning shows significant speed-up after a demonstration,  while in  the  special  case  of linear  quadratic  regulator  (LQR)  problems,  all  methods  profit  from  the  demonstration.  In  an  implementation  of pole  balancing  on  a  complex  anthropomorphic  robot  arm,  we  demonstrate  that,  when  facing  the  complexities  of real  signal  processing,  model-based  reinforcement  learning  offers  the  most robustness for LQR problems. Using  the suggested methods,  the  robot  learns  pole  balancing  in  just a  single trial  after  a  30  second  long  demonstration of the human instructor.
Clustering Sequences with Hidden Markov Models
Authors: Padhraic Smyth
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper discusses  a probabilistic model-based approach to clus(cid:173) tering sequences,  using hidden Markov models (HMMs) .  The prob(cid:173) lem  can  be  framed  as  a  generalization  of  the  standard  mixture  model approach to clustering in feature space.  Two primary issues  are  addressed.  First,  a  novel  parameter initialization procedure  is  proposed,  and  second,  the  more  difficult  problem  of determining  the  number of clusters  K,  from  the data,  is  investigated.  Experi(cid:173) mental results  indicate that the proposed  techniques  are useful  for  revealing hidden  cluster structure in  data sets of sequences.
Promoting Poor Features to Supervisors: Some Inputs Work Better as Outputs
Authors: Virginia R. de Sa Rich Caruana
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In supervised  learning there is  usually  a  clear  distinction  between  inputs  and  outputs - inputs are  what  you  will  measure,  outputs  are  what  you  will  predict  from  those  measurements.  This  paper  shows  that the distinction between  inputs  and  outputs is  not this  simple.  Some  features  are  more  useful  as  extra  outputs  than  as  inputs.  By using a feature  as  an output we  get  more than just the  case  values but can. learn a  mapping from  the other inputs to that  feature.  For many features  this mapping may be more useful  than  the  feature  value  itself.  We  present  two  regression  problems  and  one  classification problem  where  performance improves  if features  that  could  have  been  used  as  inputs  are  used  as  extra  outputs  instead.  This result is  surprising since  a feature used  as  an output  is  not  used  during testing.
Hidden Markov Decision Trees
Authors: Zoubin Ghahramani, Lawrence K. Saul Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  study  a  time  series  model  that  can  be  viewed  as  a  decision  tree  with Markov  temporal structure.  The model is  intractable for  exact  calculations,  thus we  utilize  variational approximations.  We  consider  three different  distributions for  the approximation: one in  which the Markov calculations are performed exactly and the layers  of the  decision  tree  are  decoupled,  one  in  which  the  decision  tree  calculations are performed exactly and the time steps of the Markov  chain are decoupled,  and one in which  a  Viterbi-like assumption is  made to  pick out  a  single  most  likely  state sequence.  We  present  simulation results for  artificial data and the Bach chorales.
Representing Face Images for Emotion Classification
Authors: Garrison W. Cottrell Curtis Padgett
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  compare  the  generalization  performance of three  distinct  rep(cid:173) resentation schemes for facial emotions using a  single classification  strategy  (neural  network).  The face  images presented  to the  clas(cid:173) sifiers  are  represented  as:  full  face  projections  of the  dataset  onto  their  eigenvectors  (eigenfaces);  a  similar projection  constrained  to  eye  and  mouth  areas  (eigenfeatures);  and  finally  a  projection  of  the eye and mouth areas onto the eigenvectors obtained from 32x32  random image patches from the dataset.  The latter system achieves  86%  generalization on novel face  images  (individuals the networks  were  not trained  on)  drawn from  a  database in  which  human sub(cid:173) jects  consistently  identify a single emotion for  the face .
Separating Style and Content
Authors: William T. Freeman Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  seek to analyze and manipulate two factors,  which we  call style  and content,  underlying a set of observations.  We fit  training data  with bilinear models which explicitly represent the two-factor struc(cid:173) ture.  These models can adapt easily during testing to new  styles or  content,  allowing us  to solve three general tasks:  extrapolation of a  new style to unobserved content;  classification of content observed  in  a  new  style;  and  translation of new  content  observed  in  a  new  style.  For classification, we embed bilinear models in a probabilistic  framework,  Separable  Mixture  Models  (SMMsj,  which  generalizes  earlier  work  on  factorial  mixture  models  [7,  3].  Significant  per(cid:173) formance  improvement  on  a  benchmark  speech  dataset  shows  the  benefits  of our approach.
Contour Organisation with the EM Algorithm
Authors: Edwin R. Hancock José A. F. Leite
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes how the early visual process of contour organ(cid:173) isation  can  be  realised  using  the  EM  algorithm.  The  underlying  computational representation is  based on fine  spline coverings.  Ac(cid:173) cording  to our  EM  approach  the  adjustment  of spline  parameters  draws  on  an  iterative  weighted  least-squares  fitting  process.  The  expectation  step  of our  EM  procedure  computes  the  likelihood  of  the data using a mixture model defined over the set of spline cover(cid:173) ings.  These splines  are limited in  their  spatial extent  using  Gaus(cid:173) sian windowing functions.  The maximisation of the likelihood leads  to a set of linear equations in the spline parameters which solve the  weighted least squares problem.  We  evaluate the technique  on  the  localisation of road structures in  aerial infra-red images.
Combining Neural Network Regression Estimates with Regularized Linear Weights
Authors: Michael J. Pazzani Christopher J. Merz
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: When  combining a  set  of learned  models  to form  an  improved es(cid:173) timator, the  issue  of redundancy  or  multicollinearity in the set  of  models  must  be  addressed.  A  progression  of existing  approaches  and  their  limitations with  respect  to the  redundancy  is  discussed.  A  new  approach,  PCR ,  based  on  principal  components  regres(cid:173) sion  is  proposed to address  these  limitations.  An evaluation of the  new  approach  on  a  collection  of domains  reveals  that:  1)  PCR  was the most robust combination method as the redundancy of the  learned models increased,  2)  redundancy could be handled without  eliminating any of the learned models, and 3) the principal compo(cid:173) nents of the learned models provided a  continuum of "regularized"  weights from  which  PCR * could choose.
Bayesian Model Comparison by Monte Carlo Chaining
Authors: Christopher M. Bishop David Barber
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The techniques of Bayesian inference  have been  applied with great  success to many problems in neural computing including evaluation  of regression  functions,  determination of error bars on predictions,  and  the  treatment of hyper-parameters.  However,  the  problem of  model comparison is a much more challenging one for which current  techniques  have significant limitations.  In this paper we  show  how  an  extended  form  of Markov  chain  Monte  Carlo,  called  chaining,  is  able  to provide effective  estimates of the relative probabilities of  different  models.  We  present  results  from  the  robot  arm problem  and  compare  them  with  the  corresponding  results  obtained  using  the standard  Gaussian approximation framework. 
1  Bayesian Model  Comparison 
In a Bayesian treatment of statistical inference, our state of knowledge of the values  of the parameters w  in a model M  is described in terms of a probability distribution  function.  Initially this  is  chosen  to be  some prior distribution p(wIM), which  can  be  combined with a  likelihood function  p( Dlw, M) using Bayes'  theorem  to give  a  posterior  distribution p(wID, M) in  the form
Consistent Classification, Firm and Soft
Authors: Yoram Baram
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A classifier is called  consistent with respect to a given set of class(cid:173) labeled points if it correctly  classifies the set.  We  consider  classi(cid:173) fiers  defined  by unions of local separators and propose  algorithms  for consistent classifier reduction.  The expected complexities of the  proposed algorithms are derived along with the expected classifier  sizes.  In  particular, the proposed approach yields  a  consistent re(cid:173) duction  of the  nearest  neighbor  classifier,  which  performs  "firm"  classification,  assigning  each  new  object  to  a  class,  regardless  of  the  data  structure.  The  proposed  reduction  method  suggests  a  notion  of "soft"  classification,  allowing for  indecision  with respect  to  objects  which  are  insufficiently  or  ambiguously  supported  by  the data.  The performances of the proposed classifiers in  predict(cid:173) ing  stock behavior  are  compared  to  that  achieved  by  the nearest  neighbor method.
Bayesian Unsupervised Learning of Higher Order Structure
Authors: Terrence J. Sejnowski Michael S. Lewicki
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Multilayer architectures such  as  those used in Bayesian belief net(cid:173) works  and  Helmholtz  machines  provide  a  powerful framework  for  representing  and learning higher  order statistical relations  among  inputs.  Because  exact  probability  calculations  with  these  mod(cid:173) els  are often intractable, there is  much interest in finding  approxi(cid:173) mate algorithms.  We present an algorithm that efficiently discovers  higher  order structure using EM  and Gibbs  sampling.  The model  can be interpreted as a stochastic recurrent network in which ambi(cid:173) guity in lower-level states is resolved through feedback from higher  levels.  We demonstrate the performance of the algorithm on bench(cid:173) mark problems.
An Architectural Mechanism for Direction-tuned Cortical Simple Cells: The Role of Mutual Inhibition
Authors: Fabio Solari, Giacomo M. Bisio Silvio P. Sabatini
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  linear  architectural  model  of cortical  simple cells  is  presented.  The  model  evidences  how  mutual  inhibition,  occurring  through  synaptic  coupling  functions  asymmetrically  distributed  in  space,  can be a possible basis for  a wide variety of spatio-temporal simple  cell  response properties, including direction selectivity and velocity  tuning.  While  spatial  asymmetries  are  included  explicitly  in  the  structure of the inhibitory interconnections, temporal asymmetries  originate  from  the  specific  mutual  inhibition  scheme  considered.  Extensive simulations supporting the model are reported.
Complex-Cell Responses Derived from Center-Surround Inputs: The Surprising Power of Intradendritic Computation
Authors: Daniel L. Ruderman, Kevin A. Archie Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Biophysical  modeling  studies  have  previously  shown  that  cortical  pyramidal  cells  driven  by  strong  NMDA-type  synaptic  currents  and/or containing dendritic voltage-dependent Ca++ or Na+  chan(cid:173) nels, respond more strongly when synapses are activated in several  spatially  clustered  groups  of  optimal  size-in  comparison  to  the  same  number  of synapses  activated  diffusely  about  the  dendritic  arbor  [8]- The  nonlinear  intradendritic  interactions  giving  rise  to  this  "cluster sensitivity"  property are akin to a layer of virtual non(cid:173) linear "hidden units" in the dendrites, with implications for the cel(cid:173) lular basis of learning and memory  [7,  6],  and for  certain classes of  nonlinear sensory processing [8]- In the present study, we show that  a  single  neuron,  with  access  only  to excitatory inputs  from  unori(cid:173) ented ON- and OFF-center cells in the LGN, exhibits the principal  nonlinear response properties of a  "complex"  cell in primary visual  cortex,  namely orientation tuning  coupled with  translation invari(cid:173) ance  and  contrast  insensitivity_  We  conjecture  that  this  type  of  intradendritic processing could explain how complex cell responses  can persist in the absence of oriented simple cell input  [13]-
84 
B.  W.  Mel,  D.  L.  Ruderman and K.  A. Archie
A Constructive Learning Algorithm for Discriminant Tangent Models
Authors: Alessandro Sperduti, Antonina Starita Diego Sona
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: (HSS)  developed  an  algo(cid:173)
To  reduce  the  computational complexity of classification systems  using  tangent  distance,  Hastie  et  al.  rithm  to  devise  rich  models  for  representing  large  subsets  of the  data  which  computes  automatically  the  "best"  associated  tan(cid:173) gent subspace.  Schwenk  &  Milgram proposed  a  discriminant mod(cid:173) ular classification system (Diabolo)  based on several autoassociative  multilayer perceptrons  which  use  tangent  distance  as  error  recon(cid:173) struction  measure.  We  propose  a  gradient  based  constructive  learning  algorithm for  building  a  tangent  subspace  model  with  discriminant  capabilities  which  combines  several  of the  the  advantages  of both  HSS  and  Diabolo:  devised  tangent  models  hold  discriminant  capabilities,  space  requirements  are  improved  with  respect  to  HSS  since  our  algorithm is discriminant and thus it needs fewer prototype models,  dimension of the tangent subspace is  determined automatically by  the  constructive algorithm, and our  algorithm is  able to learn new  transformations.
Effective Training of a Neural Network Character Classifier for Word Recognition
Authors: Richard F. Lyon, Brandyn J. Webb Larry S. Yaeger
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  combined  an  artificial  neural  network  (ANN)  character  classifier with context-driven search over character segmentation, word  segmentation,  and  word  recognition  hypotheses  to  provide  robust  recognition  of hand-printed  English  text  in  new  models  of  Apple  Computer's Newton MessagePad.  We present some innovations in  the  training and use of ANNs al;  character classifiers for  word recognition,  including normalized output error, frequency balancing, error emphasis,  negative training, and stroke warping.  A recurring theme of reducing a  priori biases emerges and is discussed.
Second-order Learning Algorithm with Squared Penalty Term
Authors: Ryohei Nakano Kazumi Saito
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This  paper compares three penalty terms  with respect to the effi(cid:173) ciency of supervised learning, by using first- and second-order learn(cid:173) ing algorithms.  Our experiments showed that for  a reasonably ade(cid:173) quate penalty factor,  the combination of the squared penalty term  and  the second-order  learning  algorithm  drastically improves  the  convergence  performance more than 20  times  over  the other  com(cid:173) binations, at the same time bringing about a better generalization  performance.
Multi-effect Decompositions for Financial Data Modeling
Authors: John E. Moody Lizhong Wu
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: High  frequency  foreign  exchange  data  can  be  decomposed  into  three  components:  the  inventory effect component,  the  surprise infonnation  (news) component and the regular infonnation component. The presence  of the inventory effect and news can make analysis of trends due to  the  diffusion of infonnation (regular information component) difficult.  We propose a neural-net-based, independent component analysis to sep(cid:173) arate high frequency foreign exchange data into these three components.  Our empirical results show that our proposed multi-effect decomposition  can reveal the intrinsic price behavior.
Microscopic Equations in Rough Energy Landscape for Neural Networks
Authors: K. Y. Michael Wong
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  consider  the  microscopic  equations  for  learning  problems  in  neural  networks.  The  aligning  fields  of an  example  are  obtained  from  the  cavity  fields,  which  are  the  fields  if  that  example  were  absent  in  the  learning  process.  In  a  rough  energy  landscape,  we  assume  that  the  density  of the  local  minima obey  an  exponential  distribution, yielding macroscopic properties agreeing with the first  step replica symmetry breaking solution.  Iterating the microscopic  equations  provide  a  learning  algorithm,  which  results  in  a  higher  stability than conventional algorithms.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Authors: Javier R. Movellan, Terrence J. Sejnowski Michael S. Gray
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Humans  use  visual as  well  as  auditory speech signals to recognize  spoken words.  A variety of systems have been investigated for per(cid:173) forming  this  task.  The main  purpose of this  research  was  to sys(cid:173) tematically compare the performance of a range of dynamic visual  features  on  a  speechreading  task.  We  have  found  that  normal(cid:173) ization  of images  to eliminate  variation  due  to  translation,  scale,  and planar rotation  yielded  substantial  improvements  in  general(cid:173) ization performance regardless of the visual representation used.  In  addition,  the dynamic  information  in  the difference  between  suc(cid:173) cessive  frames  yielded  better performance than optical-flow  based  approaches, and compression by local low-pass filtering worked sur(cid:173) prisingly better than global principal components analysis  (PCA).  These results are examined and possible explanations are explored.
The Effect of Correlated Input Data on the Dynamics of Learning
Authors: Ole Winther Søren Halkjær
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The convergence properties of the gradient descent  algorithm in the  case  of the  linear  perceptron  may  be  obtained  from  the  response  function.  We  derive  a general expression for  the  response  function  and  apply it to the  case  of data with simple input  correlations.  It  is  found  that  correlations severely  may slow  down  learning.  This  explains the success of PCA as a method for reducing training time.  Motivated by this finding  we  furthermore propose to transform the  input  data by  removing the mean across  input variables as  well  as  examples to decrease correlations.  Numerical findings for  a medical  classification  problem  are  in  fine  agreement  with  the  theoretical  results.
Exploiting Model Uncertainty Estimates for Safe Dynamic Control Learning
Authors: Jeff G. Schneider
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Model  learning combined  with  dynamic  programming has  been  shown  to  be effective  for  learning control  of continuous state dynamic systems.  The  simplest method assumes the learned model is correct  and applies dynamic  programming to it, but many approximators provide uncertainty estimates  on  the  fit.  How  can  they  be  exploited?  This  paper  addresses  the  case  where the system must be prevented from having catastrophic failures dur(cid:173) ing learning.  We  propose  a  new  algorithm adapted  from  the  dual  control  literature  and  use  Bayesian  locally  weighted  regression  models  with  dy(cid:173) namic programming.  A common reinforcement learning assumption is that  aggressive exploration should be encouraged.  This paper addresses the con(cid:173) verse  case  in which  the system  has  to  reign  in exploration.  The algorithm  is illustrated on a 4 dimensional simulated control  problem.