mtybilly/Instruct2DS huggingface dataset:
Shared Context Probabilistic Transducers
Authors: Samy Bengio, Jean-Franc Isabelle, Yoram Singer Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Recently,  a model for  supervised learning of probabilistic transduc(cid:173) ers  represented  by suffix trees  was  introduced.  However,  this  algo(cid:173) rithm tends  to build very  large  trees,  requiring very large amounts  of computer memory.  In this paper, we  propose  anew, more com(cid:173) pact, transducer model in which one shares the parameters of distri(cid:173) butions  associated  to  contexts  yielding similar conditional output  distributions .  We  illustrate the  advantages  of the  proposed  algo(cid:173) rithm  with  comparative  experiments  on  inducing  a  noun  phrase  recogmzer.
Learning to Schedule Straight-Line Code
Authors: Paul E. Utgoff, John Cavazos, Doina Precup, Darko Stefanovic, Carla E. Brodley, David Scheeff J. Eliot B. Moss
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Program execution speed on modem computers is sensitive,  by  a factor  of  two or more, to the order in which instructions are presented to the proces(cid:173) sor.  To realize potential execution efficiency,  an optimizing compiler must  employ  a heuristic  algorithm  for  instruction  scheduling.  Such  algorithms  are painstakingly hand-crafted, which is expensive and time-consuming. We  show how to cast the instruction scheduling problem as  a learning task, ob(cid:173) taining the  heuristic  scheduling  algorithm  automatically.  Our focus  is  the  narrower problem of scheduling straight-line code (also called basic blocks  of instructions).  Our empirical results  show that just a few features  are ad(cid:173) equate for quite good performance at this  task for a real modem processor,  and that  any  of several  supervised  learning  methods  perform nearly  opti(cid:173) mally with respect to the features used.
Approximating Posterior Distributions in Belief Networks Using Mixtures
Authors: Neil D. Lawrence, Tommi Jaakkola, Michael I. Jordan Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Exact inference in densely connected Bayesian networks is computation(cid:173) ally intractable, and so there is considerable interest in developing effec(cid:173) tive approximation schemes. One approach which has been adopted is to  bound the log likelihood using a mean-field approximating distribution.  While this leads to a tractable algorithm, the mean field distribution is as(cid:173) sumed to be factorial and hence unimodal.  In this paper we demonstrate  the feasibility of using a richer class of approximating distributions based  on mixtures of mean field distributions.  We derive an efficient algorithm  for updating the mixture parameters and apply it to the problem of learn(cid:173) ing  in  sigmoid  belief networks.  Our results  demonstrate  a  systematic  improvement over  simple  mean  field  theory  as  the  number of mixture  components is increased.
The Canonical Distortion Measure in Feature Space and 1-NN Classification
Authors: Peter L. Bartlett Jonathan Baxter
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  prove  that  the  Canonical  Distortion Measure  (CDM)  [2,  3]  is  the  optimal distance measure to use for  I  nearest-neighbour (l-NN) classifi(cid:173) cation, and show that it reduces to squared Euclidean distance in feature  space  for function classes that can be expressed  as  linear combinations  of a  fixed  set  of features.  PAC-like  bounds  are  given  on the  sample(cid:173) complexity  required to  learn the CDM.  An experiment  is presented  in  which  a  neural network CDM was  learnt for a Japanese  OCR environ(cid:173) ment and then used to do  I-NN classification.
A Non-Parametric Multi-Scale Statistical Model for Natural Images
Authors: Paul A. Viola Jeremy S. De Bonet
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The  observed  distribution  of natural  images  is  far  from  uniform.  On  the  contrary,  real  images  have  complex  and  important  struc(cid:173) ture  that  can  be  exploited  for  image  processing,  recognition  and  analysis.  There have been many proposed approaches to the prin(cid:173) cipled  statistical modeling of images, but each has been  limited  in  either  the complexity  of the  models  or  the  complexity  of the  im(cid:173) ages.  We present a non-parametric multi-scale statistical model for  images  that can be  used for  recognition,  image  de-noising,  and  in  a  "generative mode"  to synthesize high  quality textures.
A Solution for Missing Data in Recurrent Neural Networks with an Application to Blood Glucose Prediction
Authors: Thomas Briegel Volker Tresp
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  consider neural  network models for stochastic nonlinear dynamical  systems  where measurements  of the  variable of interest are only avail(cid:173) able at irregular intervals i.e.  most realizations are missing.  Difficulties  arise  since the solutions for  prediction and maximum  likelihood learn(cid:173) ing with missing data lead  to  complex integrals, which even  for simple  cases  cannot  be  solved  analytically.  In  this  paper  we  propose  a  spe(cid:173) cific  combination of a nonlinear recurrent  neural  predictive model  and  a  linear error model  which  leads  to  tractable prediction and  maximum  likelihood adaptation rules.  In particular,  the  recurrent  neural  network  can be trained using the real-time recurrent learning rule and  the linear  error model  can be trained by an EM  adaptation rule,  implemented us(cid:173) ing forward-backward Kalman filter equations. The model is applied to  predict the glucose/insulin metabolism of a diabetic patient where blood  glucose measurements  are  only available a few  times a day  at  irregular  intervals.  The new model shows considerable improvement with respect  to both recurrent neural networks trained with teacher forcing or in a free  running mode and various linear models.
Regularisation in Sequential Learning Algorithms
Authors: Mahesan Niranjan, Andrew H. Gee João F. G. de Freitas
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In this paper,  we  discuss regularisation in online/sequential learn(cid:173) ing  algorithms.  In environments  where  data  arrives  sequentially,  techniques  such  as  cross-validation  to  achieve  regularisation  or  model  selection  are  not  possible.  Further,  bootstrapping  to  de(cid:173) termine  a  confidence  level  is  not  practical.  To  surmount  these  problems, a minimum variance estimation approach that makes use  of the extended Kalman algorithm for  training multi-layer percep(cid:173) trons is employed.  The novel contribution of this paper is  to show  the  theoretical links  between  extended  Kalman filtering,  Sutton's  variable  learning  rate  algorithms  and  Mackay's  Bayesian  estima(cid:173) tion  framework.  In  doing  so,  we  propose  algorithms to overcome  the  need  for  heuristic  choices  of  the  initial  conditions  and  noise  covariance matrices in  the Kalman approach.
An Improved Policy Iteration Algorithm for Partially Observable MDPs
Authors: Eric A. Hansen
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A new  policy iteration algorithm for  partially observable Markov  decision processes is  presented that is simpler and more efficient than  an earlier policy  iteration algorithm of Sondik  (1971,1978).  The key  simplification is  representation of a  policy as a finite-state  controller.  This representation makes policy evaluation straightforward.  The pa(cid:173) per's contribution is  to show that the dynamic-programming update  used  in the policy improvement step can be interpreted as the trans(cid:173) formation of a finite-state controller into an improved finite-state con(cid:173) troller.  The  new  algorithm  consistently  outperforms  value  iteration  as an approach to solving infinite-horizon  problems.
The Asymptotic Convergence-Rate of Q-learning
Authors: Csaba Szepesvári
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In  this  paper  we  show  that  for  discounted  MDPs  with  discount  factor,  >  1/2  the  asymptotic  rate  of convergence  of  Q-Iearning  if  R(1 - ,) <  1/2 and  O( Jlog log tit)  otherwise  is  O(1/tR (1-1')  provided that the state-action pairs are sampled from a fixed prob(cid:173) ability  distribution.  Here  R  =  Pmin/Pmax  is  the ratio of the  min(cid:173) imum  and maximum state-action occupation frequencies.  The re(cid:173) sults extend to convergent on-line learning provided that Pmin  > 0,  where  Pmin  and  Pmax  now  become  the  minimum  and  maximum  state-action  occupation frequencies  corresponding  to  the  station(cid:173) ary distribution.
Learning Human-like Knowledge by Singular Value Decomposition: A Progress Report
Authors: Darrell Laham, Peter W. Foltz Thomas K. Landauer
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Singular value decomposition  (SVD)  can  be  viewed  as  a  method  for  unsupervised training of a network that associates two  classes  of events  reciprocally by linear connections  through  a single  hidden  layer.  SVD  was used to learn and represent relations among  very large numbers  of  words  (20k-60k)  and  very  large numbers  of natural  text  passages  (lk- 70k)  in  which  they  occurred.  The  result  was  100-350  dimensional  "semantic spaces" in which any trained or newly aibl word or passage  could be represented as  a vector,  and  similarities  were measured by  the  cosine  of  the  contained  angle  between  vectors.  Good  accmacy  in  simulating  human judgments and behaviors has  been  demonstrated  by  performance  on  multiple-choice  vocabulary  and  domain  knowledge  tests, emulation of expert essay evaluations,  and  in  several other ways.  Examples are also given of how the kind of knowledge extracted by  this  method can be applied.
Toward a Single-Cell Account for Binocular Disparity Tuning: An Energy Model May Be Hiding in Your Dendrites
Authors: Daniel L. Ruderman, Kevin A. Archie Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Converging  evidence  has  shown  that  human  object  recognition  depends  on  familiarity  with  the  images  of  an  object.  Further,  the  greater  the  similarity  between  objects,  the  stronger  is  the  dependence  on  object  appearance,  and  the  more  important  two(cid:173) dimensional (2D) image information becomes.  These findings,  how(cid:173) ever, do not rule out the use of 3D structural information in recog(cid:173) nition,  and  the  degree  to  which  3D  information  is  used  in  visual  memory is an important issue.  Liu, Knill, & Kersten (1995) showed  that  any model  that  is  restricted  to  rotations  in  the  image  plane  of independent  2D  templates  could not  account for  human perfor(cid:173) mance in discriminating novel object views.  We now present results  from models of generalized radial basis functions  (GRBF), 2D near(cid:173) est  neighbor  matching that  allows  2D  affine  transformations,  and  a Bayesian statistical estimator that integrates over all possible 2D  affine  transformations.  The  performance  of the  human  observers  relative  to  each  of  the  models  is  better for  the  novel  views  than  for  the familiar  template views,  suggesting that humans generalize  better to novel  views  from  template views.  The  Bayesian estima(cid:173) tor yields the optimal performance with  2D  affine  transformations  and  independent  2D  templates.  Therefore,  models  of  2D  affine  matching  operations  with  independent  2D  templates  are unlikely  to account for  human recognition performance.
Recurrent Neural Networks Can Learn to Implement Symbol-Sensitive Counting
Authors: Janet Wiles Paul Rodriguez
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Recently researchers have derived formal complexity analysis of analog  computation in the setting of discrete-time dynamical  systems.  As  an  empirical constrast, training recurrent neural networks (RNNs) produces  self -organized systems that are realizations of analog mechanisms.  Pre(cid:173) vious work showed that a RNN can learn to process a simple context-free  language (CFL) by counting. Herein, we extend that work to show that a  RNN can learn a harder CFL, a simple palindrome, by organizing its re(cid:173) sources into a symbol-sensitive counting solution, and we provide a dy(cid:173) namical systems analysis which demonstrates how the network: can  not  only count, but also copy and store counting infonnation.
Modeling Acoustic Correlations by Factor Analysis
Authors: Mazin G. Rahim Lawrence K. Saul
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Hidden  Markov  models (HMMs)  for  automatic speech  recognition  rely  on  high  dimensional feature  vectors  to  summarize the  short(cid:173) time properties  of speech.  Correlations between  features  can  arise  when the speech signal is non-stationary or corrupted by  noise.  We  investigate  how  to  model  these  correlations  using  factor  analysis,  a  statistical method for  dimensionality reduction .  Factor  analysis  uses  a  small number of parameters to model  the  covariance struc(cid:173) ture  of  high  dimensional  data.  These  parameters  are  estimated  by  an  Expectation-Maximization (EM)  algorithm that can  be  em(cid:173) bedded  in  the  training  procedures  for  HMMs.  We  evaluate  the  combined  use  of  mixture  densities  and  factor  analysis  in  HMMs  that  recognize  alphanumeric strings.  Holding the  total  number of  parameters fixed,  we  find  that  these  methods, properly  combined,  yield  better models than either  method on its own.
MELONET I: Neural Nets for Inventing Baroque-Style Chorale Variations
Authors: Dominik Hörnel
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: MELONET I is a multi-scale neural network system producing  baroque-style melodic variations. Given a melody, the system in(cid:173) vents a four-part chorale harmonization and a variation of any  chorale voice, after being trained on music pieces of composers like  J. S. Bach and J . Pachelbel. Unlike earlier approaches to the learn(cid:173) ing of melodic structure, the system is able to learn and reproduce  high-order structure like harmonic, motif and phrase structure in  melodic sequences. This is achieved by using mutually interacting  feedforward networks operating at different time scales, in combi(cid:173) nation with Kohonen networks to classify and recognize musical  structure. The results are chorale partitas in the style of J. Pachel(cid:173) bel. Their quality has been judged by experts to be comparable to  improvisations invented by an experienced human organist.
Hippocampal Model of Rat Spatial Abilities Using Temporal Difference Learning
Authors: Richard G. M. Morris, Peter Dayan David J. Foster
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We provide a model of the standard watermaze task, and of a more  challenging task involving novel platform locations, in which rats  exhibit one-trial learning after a few days of training.  The model  uses hippocampal place cells to support reinforcement learning,  and also,  in an integrated manner,  to build  and use  allocentric  coordinates.
Hierarchical Non-linear Factor Analysis and Topographic Maps
Authors: Geoffrey E. Hinton Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  first  describe  a  hierarchical,  generative  model  that  can  be  viewed  as  a  non-linear  generalisation  of factor  analysis  and  can  be  implemented  in  a  neural  network.  The  model  performs  per(cid:173) ceptual  inference  in  a  probabilistically consistent  manner by  using  top-down,  bottom-up and  lateral  connections.  These  connections  can  be  learned  using  simple  rules  that  require  only  locally  avail(cid:173) able  information.  We  then  show  how  to  incorporate  lateral  con(cid:173) nections  into the  generative  model.  The  model extracts  a  sparse,  distributed,  hierarchical  representation  of  depth  from  simplified  random-dot  stereograms  and  the  localised  disparity  detectors  in  the  first  hidden  layer  form  a  topographic  map.  When  presented  with image patches from  natural scenes,  the  model develops  topo(cid:173) graphically organised local feature detectors.
Learning Path Distributions Using Nonequilibrium Diffusion Networks
Authors: Javier R. Movellan, Ruth J. Williams Paul Mineiro
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  propose diffusion  networks, a  type of recurrent neural network  with probabilistic dynamics, as models for  learning natural signals  that are continuous in  time and space.  We  give  a  formula for  the  gradient  of the  log-likelihood  of  a  path  with  respect  to  the  drift  parameters for  a  diffusion  network.  This  gradient  can  be used  to  optimize diffusion networks in the nonequilibrium regime for a wide  variety of problems paralleling techniques which have succeeded in  engineering  fields  such  as  system  identification,  state estimation  and  signal  filtering.  An  aspect  of this  work  which  is  of  particu(cid:173) lar interest to  computational neuroscience and hardware design  is  that with a suitable choice of activation function,  e.g.,  quasi-linear  sigmoidal, the gradient formula is  local in  space and time.
Unsupervised On-line Learning of Decision Trees for Hierarchical Data Analysis
Authors: Joachim M. Buhmann Marcus Held
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: An adaptive on-line algorithm is proposed to estimate hierarchical  data structures for non-stationary data sources. The approach  is based on the principle of minimum cross entropy to derive a  decision tree for data clustering and it employs a metalearning idea  (learning to learn) to adapt to changes in data characteristics. Its  efficiency is demonstrated by grouping non-stationary artifical data  and by hierarchical segmentation of LANDSAT images.
Recovering Perspective Pose with a Dual Step EM Algorithm
Authors: Edwin R. Hancock Andrew D. J. Cross
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper describes  a  new approach to extracting 3D  perspective  structure  from  2D  point-sets.  The  novel  feature  is  to  unify  the  tasks of estimating transformation geometry and identifying point(cid:173) correspondence matches.  Unification  is  realised  by  constructing a  mixture model over the bi-partite graph representing the correspon(cid:173) dence match and by effecting optimisation using the EM algorithm.  According to our EM framework the probabilities of structural cor(cid:173) respondence gate contributions to the expected likelihood function  used to estimate maximum likelihood perspective pose parameters.  This provides a  means of rejecting structural outliers.
Learning to Order Things
Authors: Robert E. Schapire, Yoram Singer William W. Cohen
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: There are many applications in which it is  desirable to order rather than classify  instances.  Here we consider the problem of learning how to order, given feedback  in the form of preference judgments, i.e., statements to the effect that one instance  should be ranked ahead of another.  We outline a two-stage approach in which one  first learns by conventional means a preference Junction, of the form PREF( u, v),  which  indicates  whether  it is advisable  to  rank  u  before  v.  New  instances  are  then  ordered  so  as  to  maximize  agreements  with  the  learned  preference  func(cid:173) tion.  We  show  that  the  problem  of finding  the  ordering  that  agrees  best  with  a preference function  is  NP-complete,  even under very  restrictive assumptions.  Nevertheless, we describe a simple greedy algorithm that is guaranteed to find  a  good approximation.  We then discuss an on-line learning algorithm, based on the  "Hedge" algorithm,  for  finding  a good linear combination of ranking "experts."  We use the ordering algorithm combined with the on-line learning algorithm to  find a combination of "search experts," each of which is a domain-specific query  expansion strategy for a WWW search engine,  and present experimental  results  that demonstrate the merits of our approach.
Analog VLSI Model of Intersegmental Coordination with Nearest-Neighbor Coupling
Authors: Jeremy H. Holleman, Stephen P. DeWeerth Girish N. Patel
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We have a developed an analog VLSI system that models the coordina(cid:173) tion of neurobiological segmental oscillators. We have implemented and  tested a system that consists of a chain of eleven pattern generating cir(cid:173) cuits that are synaptically coupled to their nearest neighbors. Each pat(cid:173) tern  generating  circuit  is  implemented  with  two  silicon  Morris-Lecar  neurons that are connected in a reciprocally inhibitory network. We dis(cid:173) cuss the mechanisms of oscillations in the two-cell network and explore  system  behavior based  on  isotropic  and  anisotropic  coupling, and  fre(cid:173) quency gradients along the chain of oscillators.
Globally Optimal On-line Learning Rules
Authors: David Saad Magnus Rattray
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present a method for  determining the globally optimal on-line  learning rule for  a  soft  committee machine under a  statistical me(cid:173) chanics  framework.  This  work  complements  previous  results  on  locally  optimal  rules,  where  only  the  rate  of change  in  general(cid:173) ization error was  considered.  We  maximize the total reduction in  generalization error over the whole learning process and show how  the resulting rule can significantly outperform the locally optimal  rule.
Ensemble Learning for Multi-Layer Networks
Authors: Christopher M. Bishop David Barber
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Bayesian  treatments  of learning  in  neural  networks  are  typically  based  either  on  local  Gaussian  approximations  to  a  mode  of  the  posterior  weight  distribution,  or  on  Markov  chain  Monte  Carlo  simulations.  A  third  approach,  called  ensemble  learning,  was  in(cid:173) troduced by Hinton and van  Camp  (1993).  It aims to approximate  the  posterior  distribution  by  minimizing  the  Kullback-Leibler  di(cid:173) vergence  between the true posterior and a parametric approximat(cid:173) ing distribution.  However,  the  derivation  of a  deterministic  algo(cid:173) rithm  relied  on  the  use  of a  Gaussian  approximating distribution  with  a  diagonal  covariance  matrix  and  so  was  unable  to capture  the  posterior  correlations  between  parameters.  In this  paper,  we  show  how the ensemble learning approach can  be extended to full(cid:173) covariance Gaussian distributions while remaining computationally  tractable.  We also extend the framework to deal with hyperparam(cid:173) eters,  leading  to  a  simple  re-estimation  procedure.  Initial  results  from  a standard benchmark problem are encouraging.
Estimating Dependency Structure as a Hidden Variable
Authors: Michael I. Jordan Marina Meila
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This  paper introduces a probability model,  the mixture of trees  that can  account for sparse, dynamically changing dependence relationships.  We  present a  family  of efficient  algorithms  that use EM and  the Minimum  Spanning Tree algorithm to find the ML and MAP mixture of trees for a  variety of priors, including the Dirichlet and the MDL priors.
Ensemble and Modular Approaches for Face Detection: A Comparison
Authors: Olivier Bernier Raphaël Feraud
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A  new  learning  model  based  on  autoassociative  neural  networks  is  developped  and  applied  to  face  detection.  To  extend  the  de(cid:173) tection  ability  in  orientation  and  to  decrease  the  number  of false  alarms,  different  combinations  of networks  are  tested:  ensemble,  conditional  ensemble  and  conditional  mixture  of  networks.  The  use  of a  conditional mixture of networks  allows  to  obtain state of  the  art results on different  benchmark face  databases. 
1  A  constrained generative model 
Our  purpose  is  to  classify  an  extracted  window  x  from  an  image  as  a  face  (x  E  V)  or  non-face  (x  EN).  The  set  of all  possible  windows  is  E  =  V uN,  with  V n N  = 0.  Since collecting a representative set of non-face examples is impossible,  face  detection  by a statistical model is  a difficult  task.  An autoassociative network,  using five  layers of neurons, is able to perform a non-linear dimensionnality reduction  [Kramer,  1991].  However,  its use  as  an  estimator, to classify  an  extracted  window  as  face  or  non-face,  raises  two  problems: 

V',  the obtained sub-manifold can contain  non-face examples (V  C  V'),  2.  owing  to  local  minima,  the  obtained  solution  can  be  close  to  the  linear 

solution:  the  principal components  analysis. 
Our approach is  to use counter-examples in  order to find  a sub-manifold as  close  as  possible  to  V  and  to  constrain  the  algorithm to  converge  to  a  non-linear  solution  [Feraud,  R.  et  al.,  1997].  Each non-face example is  constrained to be  reconstructed  as  its  projection on V.  The projection  P  of a  point  x  of the  input space  E on V,  is  defined  by: 
·email:  feraud@lannion.cnet.fr  t email:  bernier@lannion.cnet.fr 
Ensemble and Modular Approaches for Face Detection: A Comparison 
473 
•  if x  E V,  then P{ x)  =  x,  •  if x rJ.  V:  P{x)  = argminYEv{d(x, V)),  where  d is  the  Euclidian distance. 
During  the  learning  process,  the  projection  P  of x  on  V  is  approximated  by:  P(x)  ,....,  ~ 2:7=1 Vi,  where  VI, V2,  •.. , Vn ,  are  the  n  nearest  neighbours,  in  the training set  of faces,  of v,  the nearest  face  example of x. 
The goal of the learning process  is  to approximate the distance V  of an input space  element  x  to the set  of faces  V: 
•  V{x, V)  =  Ilx  - P(x)11  ,....,  it (x  - £)2,  where  M  is  the size  of input image x 
and £  the  image reconstructed  by  the neural  network, 
•  let  x  E £,  then  x  E V if and only if V{x, V)  S  T,  with  T  E IR,  where  T  is  a 
threshold  used  to adjust the sensitivity of the model.
Stacked Density Estimation
Authors: David Wolpert Padhraic Smyth
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In  this  paper,  the  technique  of stacking,  previously  only  used  for  supervised  learning,  is  applied  to  unsupervised  learning.  Specifi(cid:173) cally, it is used for non-parametric multivariate density estimation,  to combine finite  mixture model and kernel density estimators.  Ex(cid:173) perimental results on both simulated data and real  world data sets  clearly  demonstrate  that  stacked  density  estimation  outperforms  other  strategies  such  as  choosing  the  single  best  model  based  on  cross-validation, combining with uniform weights, and even the sin(cid:173) gle  best  model  chosen  by  "cheating"  by  looking  at  the  data used  for  independent  testing.
An Analog VLSI Neural Network for Phase-based Machine Vision
Authors: Kwok Fai Hui Bertram Emil Shi
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We describe the design, fabrication  and test results of an  analog CMOS  VLSI neural  network prototype chip intended for phase-based machine  vision  algorithms.  The  chip  implements  an  image  filtering  operation  similar  to  Gabor-filtering.  Because  a  Gabor  filter's  output  is  complex  valued, it can be used  to define a phase at every pixel in  an image. This  phase can be used in robust algorithms for disparity estimation and  bin(cid:173) ocular  stereo  vergence  control  in  stereo  vision  and  for  image  motion  analysis. The chip reported here takes an input image and generates two  outputs at every pixel corresponding to  the real  and  imaginary parts of  the output.
Combining Classifiers Using Correspondence Analysis
Authors: Christopher J. Merz
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Several effective  methods for  improving  the  performance  of a  sin(cid:173) gle  learning  algorithm have  been  developed  recently.  The  general  approach is  to create  a  set of learned models  by  repeatedly apply(cid:173) ing  the  algorithm  to  different  versions  of the  training  data,  and  then  combine  the  learned  models'  predictions  according  to  a  pre(cid:173) scribed voting scheme.  Little work has been done in combining the  predictions  of a  collection  of models  generated  by  many  learning  algorithms having different representation and/or search strategies.  This paper describes  a  method which  uses  the strategies of stack(cid:173) ing and correspondence analysis to model the relationship between  the  learning examples and  the way  in  which  they  are  classified  by  a collection  of learned models.  A nearest  neighbor method is  then  applied  within  the  resulting  representation  to  classify  previously  unseen examples.  The new  algorithm consistently performs as  well  or  better than other combining techniques on a suite of data sets.
Synchronized Auditory and Cognitive 40 Hz Attentional Streams, and the Impact of Rhythmic Expectation on Auditory Scene Analysis
Authors: Bill Baird
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We have developed a neural network architecture that implements a the(cid:173) ory  of attention,  learning,  and  trans-cortical communication  based  on  adaptive synchronization of 5-15 Hz and 30-80 Hz oscillations between  cortical areas.  Here we present a specific higher order cortical model of  attentional networks, rhythmic expectancy, and the interaction of hi~her­ order and primar¥, cortical levels of processing. It accounts for the' mis(cid:173) match negativity' of the auditory ERP and the results of psychological  experiments of Jones showing that auditory stream  segregation depends  on the rhythmic structure of inputs. The timing mechanisms of the model  allow  us to explain how relative timing information such as the relative  order of events between  streams is  lost when streams are  formed.  The  model suggests how the theories of auditory perception and attention of  Jones and Bregman may be reconciled.
From Regularization Operators to Support Vector Kernels
Authors: Bernhard Schölkopf Alex J. Smola
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We derive the correspondence between regularization operators used in  Regularization Networks and Hilbert Schmidt Kernels appearing in Sup(cid:173) port Vector Machines. More specifica1ly, we prove that the Green's Func(cid:173) tions associated with regularization operators are suitable Support Vector  Kernels  with equivalent regularization properties.  As  a by-product we  show that a large number of Radial Basis Functions namely  condition(cid:173) ally positive definite functions may be used as Support Vector kernels.
Two Approaches to Optimal Annealing
Authors: Bernhard Schottky, David Saad Todd K. Leen
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We employ both master equation and order parameter approaches  to analyze the asymptotic dynamics of on-line learning with dif(cid:173) ferent learning rate annealing schedules. We examine the relations  between the results obtained by the two approaches and obtain new  results on the optimal decay coefficients and their dependence on  the number of hidden nodes in a two layer architecture.
Bidirectional Retrieval from Associative Memory
Authors: Günther Palm Friedrich T. Sommer
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Similarity based fault  tolerant retrieval in neural associative mem(cid:173) ories  (N AM)  has  not  lead  to  wiedespread  applications.  A  draw(cid:173) back  of  the  efficient  Willshaw  model  for  sparse  patterns  [Ste61,  WBLH69],  is  that  the high  asymptotic information  capacity is  of  little  practical  use  because  of  high  cross  talk  noise  arising  in  the  retrieval for finite  sizes.  Here a new bidirectional iterative retrieval  method for  the Willshaw model is presented, called crosswise bidi(cid:173) rectional (CB)  retrieval, providing enhanced performance.  We  dis(cid:173) cuss its asymptotic capacity limit,  analyze the first  step, and com(cid:173) pare it in experiments with the Willshaw model.  Applying the very  efficient  CB  memory model either in information retrieval systems  or  as  a  functional  model  for  reciprocal  cortico-cortical  pathways  requires  more  than  robustness against  random  noise  in  the input:  Our experiments show also the segmentation ability of CB-retrieval  with  addresses  containing  the  superposition  of  pattens,  provided  even at high memory load.
Using Helmholtz Machines to Analyze Multi-channel Neuronal Recordings
Authors: R. Christopher DeCharms, Michael Merzenich Virginia R. de Sa
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: One of the current challenges to understanding neural information  processing in biological systems is to decipher the "code" carried  by large populations of neurons acting in parallel. We present an  algorithm for automated discovery of stochastic firing patterns in  large ensembles of neurons. The algorithm, from the "Helmholtz  Machine" family, attempts to predict the observed spike patterns in  the data. The model consists of an observable layer which is directly  activated by the input spike patterns, and hidden units that are ac(cid:173) tivated through ascending connections from the input layer. The  hidden unit activity can be propagated down to the observable layer  to create a prediction of the data pattern that produced it. Hidden  units are added incrementally and their weights are adjusted to im(cid:173) prove the fit between the predictions and data, that is, to increase a  bound on the probability of the data given the model. This greedy  strategy is not globally optimal but is computationally tractable for  large populations of neurons. We show benchmark data on artifi(cid:173) cially constructed spike trains and promising early results on neuro(cid:173) physiological data collected from our chronic multi-electrode cortical  implant.
Silicon Retina with Adaptive Filtering Properties
Authors: Shih-Chii Liu
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This  paper  describes  a  small,  compact  circuit  that  captures  the  temporal and adaptation properties both of the photoreceptor and  of the laminar layers  of the fly.  This  circuit  uses  only six  transis(cid:173) tors and two capacitors.  It is operated in the subthreshold domain.  The circuit maintains a  high transient gain by using adaptation to  the  background intensity  as  a  form  of gain  control.  The  adapta(cid:173) tion time  constant of the circuit  can be  controlled via an external  bias.  Its temporal filtering  properties change with the background  intensity or  signal-to-noise conditions.  The frequency  response  of  the  circuit  shows  that in  the  frequency  range  of 1 to  100  Hz,  the  circuit response goes from  highpass filtering under high light levels  to lowpass filtering  under low  light levels  (Le.,  when  the signal-to(cid:173) noise  ratio is  low).  A  chip  with  20x20 pixels  has  been fabricated  in  1.2J.Lm  ORBIT CMOS  nwell technology.
Statistical Models of Conditioning
Authors: Theresa Long Peter Dayan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Conditioning experiments probe the ways that animals make pre(cid:173) dictions  about rewards and punishments  and  use  those  predic(cid:173) tions to control their behavior.  One standard model of condition(cid:173) ing paradigms which involve many conditioned stimuli suggests  that individual predictions should be added together.  Various key  results show that this model fails in some circumstances, and mo(cid:173) tivate an alternative model, in which there is attentional selection  between different available stimuli.  The new model is a  form  of  mixture of experts, has a close relationship with some other exist(cid:173) ing psychological suggestions, and is statistically well-founded.
The Efficiency and the Robustness of Natural Gradient Descent Learning Rule
Authors: Shun-ichi Amari Howard Hua Yang
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The inverse of the Fisher information matrix is used in the natu(cid:173) ral gradient descent algorithm to train single-layer and multi-layer  perceptrons. We have discovered a new scheme to represent the  Fisher information matrix of a stochastic multi-layer perceptron.  Based on this scheme, we have designed an algorithm to compute  the natural gradient. When the input dimension n is much larger  than the number of hidden neurons, the complexity of this algo(cid:173) rithm is of order O(n). It is confirmed by simulations that the  natural gradient descent learning rule is not only efficient but also  robust.
Compositional Plan Vectors
Authors: Daniel Geng, Pieter Abbeel, Trevor Darrell, Sergey Levine Coline Devin
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: Autonomous agents situated in real-world environments must be able to master large repertoires of skills.
While a single short skill can be learned quickly, it would be impractical to learn every task independently. Instead, the agent should share knowledge across behaviors such that each task can be learned efficiently, and such that the resulting model can generalize to new tasks, especially ones that are compositions or subsets of tasks seen previously.
A policy conditioned on a goal or demonstration has the potential to share knowledge between tasks if it sees enough diversity of inputs. However, these methods may not generalize to a more complex task at test time. We introduce compositional plan vectors (CPVs) to enable a policy to perform compositions of tasks without additional supervision. CPVs represent trajectories as the sum of the subtasks within them. We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
 Analogously to embeddings such as word2vec in NLP, CPVs can also support simple arithmetic operations -- for example, we can add the CPVs for two different tasks to command an agent to compose both tasks, without any additional training.
Learning to Propagate for Graph Meta-Learning
Authors: Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang LU LIU
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: Meta-learning extracts the common knowledge from learning different tasks and uses it for unseen tasks. It can signiﬁcantly improve tasks that suffer from insufﬁcient training data, e.g., few-shot learning. In most meta-learning methods, tasks are implicitly related by sharing parameters or optimizer. In this paper, we show that a meta-learner that explicitly relates tasks on a graph describing the relations of their output dimensions (e.g., classes) can signiﬁcantly improve few-shot learning. The graph’s structure is usually free or cheap to obtain but has rarely been explored in previous works. We develop a novel meta-learner of this type for prototype based classiﬁcation, in which a prototype is generated for each class, such that the nearest neighbor search among the prototypes produces an accurate classiﬁcation. The meta-learner, called “Gated Propagation Network (GPN)”, learns to propagate messages between prototypes of different classes on the graph, so that learning the prototype of each class beneﬁts from the data of other related classes. In GPN, an attention mechanism aggregates messages from neighboring classes of each class, with a gate choosing between the aggregated message and the message from the class itself. We train GPN on a sequence of tasks from many-shot to few-shot generated by subgraph sampling. During training, it is able to reuse and update previously achieved prototypes from the memory in a life-long learning cycle. In experiments, under different training-test discrepancy and test task generation settings, GPN outperforms recent meta-learning methods on two benchmark datasets. Code of GPN is publicly available at: https://github.com/liulu112601/Gated-Propagation-Net.
XNAS: Neural Architecture Search with Expert Advice
Authors: Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, Lihi Zelnik Niv Nayman
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. 
Unlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones.
It achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. 
Experiments show that our algorithm achieves a strong performance over several image classification datasets.
Specifically, it obtains an error rate of 1.6% for CIFAR-10, 23.9% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.
Multi-resolution Multi-task Gaussian Processes
Authors: Theodoros Damoulas, Kangrui Wang, Mark Girolami Oliver Hamelijnck
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: We consider evidence integration from potentially dependent observation processes under varying spatio-temporal sampling resolutions and noise levels. We offer a multi-resolution multi-task (MRGP) framework that allows for both inter-task and intra-task multi-resolution and multi-fidelity. We develop shallow Gaussian Process (GP) mixtures that approximate the difficult to estimate joint likelihood with a composite one and deep GP constructions that naturally handle biases. In doing so, we generalize existing approaches and offer information-theoretic corrections and efficient variational approximations. We demonstrate the competitiveness of MRGPs on synthetic settings and on the challenging problem of hyper-local estimation of air pollution levels across London from multiple sensing modalities operating at disparate spatio-temporal resolutions.
Deep Equilibrium Models
Authors: J. Zico Kolter, Vladlen Koltun Shaojie Bai
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is equivalent to running an infinite depth (weight-tied) feedforward network, but has the notable advantage that we can analytically backpropagate through the equilibrium point using implicit differentiation. Using this approach, training and prediction in these networks require only constant memory, regardless of the effective “depth” of the network. We demonstrate how DEQs can be applied to two state-of-the-art deep sequence models: self-attention transformers and trellis networks. On large-scale language modeling tasks, such as the WikiText-103 benchmark, we show that DEQs 1) often improve performance over these state-of-the-art models (for similar parameter counts); 2) have similar computational requirements to existing models; and 3) vastly reduce memory consumption (often the bottleneck for training large sequence models), demonstrating an up-to 88% memory reduction in our experiments. The code is available at https://github.com/locuslab/deq.
Cross Attention Network for Few-shot Classification
Authors: Hong Chang, Bingpeng MA, Shiguang Shan, Xilin Chen Ruibing Hou
Published in: Advances in Neural Information Processing Systems 32  (NeurIPS 2019) (Main Conference Track), 2019

Abstract: Few-shot classification aims to recognize unlabeled samples from unseen classes given only few labeled samples. The unseen classes and low-data problem make few-shot classification very challenging. Many existing approaches extracted features from labeled and unlabeled samples independently, as a result, the features are not discriminative enough. In this work, we propose a novel Cross Attention Network to address the challenging problems in few-shot classification. Firstly, Cross Attention Module is introduced to deal with the problem of unseen classes.  The module generates cross attention maps for each pair of class feature and query sample feature so as to highlight the target object regions, making the extracted feature more discriminative. Secondly, a transductive inference algorithm is proposed to alleviate the low-data problem, which iteratively utilizes the unlabeled query set to augment the support set, thereby making the class features more representative. Extensive experiments on two benchmarks show our method is a simple, effective and computationally efficient framework and outperforms the state-of-the-arts.