mtybilly/Instruct2DS huggingface dataset:
Learning Sparse Codes with a Mixture-of-Gaussians Prior
Authors: K. Jarrod Millman Bruno A. Olshausen
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  describe  a  method  for  learning  an  overcomplete  set  of  basis  functions  for  the  purpose of modeling  sparse structure in images.  The  sparsity  of the  basis  function  coefficients  is  modeled  with  a  mixture-of-Gaussians  distribution.  One  Gaussian  captures  non(cid:173) active  coefficients  with  a  small-variance  distribution  centered  at  zero,  while one or more other Gaussians capture active coefficients  with a large-variance distribution.  We show that when the prior is  in such  a form,  there exist efficient  methods for  learning the basis  functions  as  well  as  the parameters of the prior.  The performance  of the  algorithm  is  demonstrated  on  a  number  of test  cases  and  also  on  natural  images.  The  basis  functions  learned  on  natural  images  are similar to those  obtained with other methods,  but the  sparse form of the coefficient distribution is much better described.  Also, since the parameters of the prior are adapted to the data, no  assumption about sparse  structure in the  images  need  be  made  a  priori,  rather it is  learned from  the data.
Semiparametric Approach to Multichannel Blind Deconvolution of Nonminimum Phase Systems
Authors: Shun-ichi Amari, Andrzej Cichocki Liqing Zhang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this  paper we discuss  the  semi parametric statistical  model  for  blind  deconvolution.  First we  introduce a Lie Group to  the  manifold of non(cid:173) causal  FIR  filters.  Then  blind deconvolution problem is  formulated  in  the  framework of a  semiparametric  model,  and  a  family  of estimating  functions  is  derived for  blind  deconvolution.  A  natural gradient  learn(cid:173) ing algorithm is developed for training noncausal filters.  Stability of the  natural gradient algorithm is also analyzed in this framework.
Application of Blind Separation of Sources to Optical Recording of Brain Activity
Authors: Martin Stetter, Ingo Schießl, John E. W. Mayhew, Jennifer S. Lund, Niall McLoughlin, Klaus Obermayer Holger Schoner
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In the analysis of data recorded by optical imaging from intrinsic signals  (measurement of changes of light reflectance from cortical tissue) the re(cid:173) moval  of noise and  artifacts  such  as  blood  vessel  patterns  is  a  serious  problem. Often bandpass filtering is used, but the underlying assumption  that a spatial frequency  exists,  which separates  the mapping component  from  other components  (especially  the  global  signal),  is  questionable.  Here we propose alternative ways of processing optical imaging data, us(cid:173) ing blind source separation techniques based on the spatial decorre1ation  of the data.  We  first  perform  benchmarks  on  artificial  data in  order  to  select  the  way  of processing, which is  most robust with respect  to  sen(cid:173) sor noise.  We then apply it to recordings of optical imaging experiments  from  macaque primary visual cortex. We show that our BSS technique is  able  to  extract ocular dominance and orientation preference maps  from  single  condition  stacks,  for  data,  where  standard  post-processing pro(cid:173) cedures  fail.  Artifacts,  especially  blood  vessel  patterns,  can  often  be  completely removed  from  the maps.  In  summary,  our method for blind  source separation using extended spatial decorrelation is a superior tech(cid:173) nique for the analysis of optical recording data.
Spectral Cues in Human Sound Localization
Authors: Anna Corderoy, Simon Carlile, André van Schaik Craig T. Jin
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  differential  contribution  of the  monaural  and  interaural  spectral  cues to human sound localization was examined using a combined psy(cid:173) chophysical  and  analytical  approach.  The  cues  to  a  sound's  location  were  correlated  on an individual basis  with  the  human  localization re(cid:173) sponses to a variety of spectrally manipulated sounds.  The spectral cues  derive from the acoustical filtering of an individual's auditory periphery  which  is  characterized by the measured head-related transfer functions  (HRTFs).  Auditory localization performance was determined in  virtual  auditory space (VAS).  Psychoacoustical experiments were conducted in  which the amplitude spectra of the sound stimulus was  varied indepen(cid:173) dentlyat each ear while preserving the normal timing cues, an impossibil(cid:173) ity in the free-field environment. Virtual auditory noise stimuli were gen(cid:173) erated over earphones for a specified target direction such that there was  a "false" flat  spectrum at the left eardrum.  Using the subject's HRTFs,  the sound spectrum at the right eardrum was then adjusted so that either  the  true right  monaural  spectral  cue  or the  true  interaural  spectral  cue  was preserved.  All  subjects showed systematic mislocalizations in both  the true right and true interaural spectral conditions which was absent in  their control localization performance. The analysis of the different cues  along with the subjects' localization responses suggests there are signif(cid:173) icant differences in the use of the monaural and interaural spectral cues  and that the auditory system's reliance on the spectral  cues varies with  the sound condition.
Robust Full Bayesian Methods for Neural Networks
Authors: João F. G. de Freitas, Arnaud Doucet Christophe Andrieu
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this paper, we propose a full Bayesian model for neural networks.  This model treats the model dimension (number of neurons), model  parameters, regularisation parameters and noise parameters as ran(cid:173) dom  variables  that  need  to be estimated.  We  then  propose  a  re(cid:173) versible jump Markov chain Monte Carlo (MCMC)  method to per(cid:173) form  the necessary computations.  We  find  that the results are not  only  better than the previously  reported ones,  but  also  appear to  be  robust  with  respect  to  the  prior  specification.  Moreover,  we  present a  geometric convergence theorem for  the algorithm.
A Neurodynamical Approach to Visual Attention
Authors: Josef Zihl Gustavo Deco
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The psychophysical evidence for "selective attention" originates mainly  from  visual search experiments. In this work, we formulate a hierarchi(cid:173) cal  system of interconnected modules consisting in  populations of neu(cid:173) rons  for  modeling  the  underlying  mechanisms  involved  in  selective  visual  attention.  We  demonstrate  that  our  neural  system  for  visual  search  works  across the  visual  field  in  parallel  but due  to the  different  intrinsic dynamics can show the two experimentally observed modes of  visual  attention,  namely:  the  serial  and  the  parallel  search  mode.  In  other words, neither explicit model of a focus of attention nor saliencies  maps are  used.  The  focus of attention  appears as an  emergent property  of the dynamic behavior of the system. The neural population dynamics  are  handled in  the  framework  of the  mean-field  approximation.  Conse(cid:173) quently, the whole process can be expressed as a system of coupled dif(cid:173) ferential equations.
Dynamics of Supervised Learning with Restricted Training Sets and Noisy Teachers
Authors: C. W. H. Mace Anthony C. C. Coolen
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We generalize a recent formalism to describe the dynamics of supervised  learning in layered neural networks, in the regime where data recycling  is  inevitable, to the case of noisy teachers.  Our theory generates reliable  predictions for the  evolution in  time of training- and generalization er(cid:173) rors, and extends the class of mathematically solvable learning processes  in large neural networks to those situations where overfitting can occur.
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Authors: Javier R. Movellan John R. Hershey
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Psychophysical and physiological evidence shows that sound local(cid:173) ization of acoustic signals is strongly influenced by  their synchrony  with visual signals.  This effect,  known as ventriloquism, is at work  when  sound  coming  from  the  side  of a  TV  set  feels  as  if it  were  coming  from  the  mouth  of the  actors.  The  ventriloquism  effect  suggests that there is  important information about sound location  encoded in  the synchrony between the audio and video signals.  In  spite  of  this  evidence,  audiovisual  synchrony  is  rarely  used  as  a  source of information  in  computer  vision  tasks.  In  this  paper  we  explore the use  of audio  visual  synchrony to locate sound sources.  We developed a system that searches for regions of the visual land(cid:173) scape that correlate highly with the acoustic signals and tags them  as  likely  to contain an acoustic  source.  We  discuss  our experience  implementing the system,  present results on a  speaker localization  task and discuss potential applications of the approach.
Predictive Sequence Learning in Recurrent Neocortical Circuits
Authors: Terrence J. Sejnowski Rajesh P. N. Rao
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Neocortical circuits are dominated by massive excitatory feedback:  more  than eighty percent of the synapses made by excitatory cortical neurons  are onto other excitatory cortical neurons.  Why is there such massive re(cid:173) current excitation in the neocortex and what is its role in cortical compu(cid:173) tation? Recent neurophysiological experiments have shown that the plas(cid:173) ticity of recurrent neocortical synapses is governed by a temporally asym(cid:173) metric Hebbian learning rule.  We  describe how such a rule may  allow  the cortex to modify recurrent synapses for prediction of input sequences.  The goal is to predict the next cortical input from the recent past based on  previous experience of similar input sequences. We show that a temporal  difference learning rule for prediction used in conjunction with dendritic  back-propagating action potentials reproduces the temporally asymmet(cid:173) ric Hebbian plasticity observed physiologically. Biophysical simulations  demonstrate that a network of cortical neurons can learn to predict mov(cid:173) ing stimuli and develop direction selective responses as a consequence of  learning. The space-time response properties of model neurons are shown  to be similar to those of direction selective cells in alert monkey VI.
Differentiating Functions of the Jacobian with Respect to the Weights
Authors: Barak A. Pearlmutter Gary William Flake
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: For many problems, the correct behavior of a model depends not only on  its input-output mapping but also on properties of its Jacobian matrix, the  matrix of partial derivatives of the model's outputs with respect to its in(cid:173) puts.  We introduce the J-prop algorithm, an efficient general method for  computing the exact partial derivatives of a variety of simple functions of  the Jacobian of a model with respect to its free parameters. The algorithm  applies to any parametrized feedforward model,  including nonlinear re(cid:173) gression, multilayer perceptrons, and radial basis function networks.
Effects of Spatial and Temporal Contiguity on the Acquisition of Spatial Information
Authors: Paul W. Munro Thea B. Ghiselli-Crippa
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Spatial  information comes in two forms:  direct spatial information (for  example, retinal position) and indirect temporal contiguity information,  since objects encountered sequentially are in general spatially close. The  acquisition  of spatial  information  by  a  neural  network  is  investigated  here. Given a spatial layout of several objects, networks are trained on a  prediction task.  Networks using temporal sequences with no direct spa(cid:173) tial information are found to  develop internal representations that show  distances correlated with distances in the external layout.  The influence  of spatial information is analyzed by providing direct spatial information  to the system during training that is  either consistent with the layout or  inconsistent with  it.  This  approach  allows  examination of the relative  contributions of spatial and temporal contiguity.
Agglomerative Information Bottleneck
Authors: Naftali Tishby Noam Slonim
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce a novel distributional clustering algorithm that max(cid:173) imizes  the  mutual  information  per  cluster  between  data and  giv(cid:173) en  categories.  This  algorithm  can  be considered  as  a  bottom  up  hard  version  of  the  recently  introduced  "Information  Bottleneck  Method".  The algorithm is  compared with the top-down soft  ver(cid:173) sion  of the  information  bottleneck  method  and  a  relationship  be(cid:173) tween the hard and soft results is established.  We  demonstrate the  algorithm on the 20 Newsgroups data set.  For a subset of two news(cid:173) groups  we  achieve compression  by  3  orders  of magnitudes loosing  only  10%  of the original mutual information.
Spike-based Learning Rules and Stabilization of Persistent Neural Activity
Authors: H. Sebastian Seung Xiaohui Xie
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  analyze  the  conditions  under  which  synaptic  learning  rules  based  on  action  potential timing can  be approximated by  learning rules  based  on  firing  rates.  In  particular, we  consider a form  of plasticity in  which  synapses depress when a presynaptic spike is followed by a postsynaptic  spike, and potentiate with the opposite temporal ordering.  Such differen(cid:173) tial anti-Hebbian plasticity can be approximated under certain conditions  by a learning rule that depends on the time derivative of the postsynaptic  firing rate.  Such a learning rule acts to stabilize persistent neural activity  patterns in recurrent neural networks.
Nonlinear Discriminant Analysis Using Kernel Functions
Authors: Volker Steinhage Volker Roth
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Fishers linear discriminant analysis  (LDA)  is  a  classical multivari(cid:173) ate technique both for  dimension reduction and classification.  The  data vectors are transformed into a low dimensional subspace such  that  the  class  centroids  are  spread  out  as  much  as  possible.  In  this subspace LDA  works  as  a  simple  prototype classifier  with lin(cid:173) ear decision  boundaries.  However,  in many applications the linear  boundaries  do  not  adequately separate the  classes.  We  present  a  nonlinear generalization of discriminant analysis that uses the ker(cid:173) nel trick of representing dot products by kernel functions.  The pre(cid:173) sented algorithm allows a simple formulation of the EM-algorithm  in terms of kernel functions which leads to a unique concept for  un(cid:173) supervised  mixture analysis,  supervised discriminant  analysis  and  semi-supervised discriminant analysis with partially unlabelled ob(cid:173) servations in feature spaces.
On Input Selection with Reversible Jump Markov Chain Monte Carlo Sampling
Authors: Peter Sykacek
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this paper we will treat input selection for a radial basis function  (RBF) like classifier within a Bayesian framework.  We approximate  the a-posteriori distribution over both model coefficients  and input  subsets  by samples drawn with Gibbs updates and reversible jump  moves.  Using some  public  datasets,  we  compare the  classification  accuracy  of the  method with  a  conventional  ARD  scheme.  These  datasets  are  also used  to  infer the  a-posteriori  probabilities of dif(cid:173) ferent  input subsets.
Uniqueness of the SVM Solution
Authors: David J. Crisp Christopher J. C. Burges
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  give  necessary  and  sufficient  conditions  for  uniqueness  of the  support vector solution for the problems of pattern recognition and  regression estimation, for a general class of cost functions.  We show  that if the solution is not unique, all support vectors are necessarily  at  bound,  and  we  give  some  simple examples of non-unique solu(cid:173) tions.  We  note that uniqueness of the primal (dual)  solution does  not necessarily imply uniqueness of the dual (primal) solution.  We  show  how to compute the threshold b when the solution is  unique,  but when  all support vectors are at bound, in which case the usual  method for  determining b does  not work.
The Parallel Problems Server: an Interactive Tool for Large Scale Machine Learning
Authors: Parry Husbands Charles Lee Isbell Jr.
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Imagine that you wish to classify data consisting of tens of thousands of ex(cid:173) amples residing in a twenty thousand dimensional space.  How can one ap(cid:173) ply standard machine learning algorithms? We describe the Parallel Prob(cid:173) lems Server (PPServer) and MATLAB*P.  In tandem they  allow users  of  networked computers to work transparently on large data sets from within  Matlab.  This  work is motivated by the desire to  bring the many  benefits  of scientific computing algorithms and computational power to  machine  learning researchers.  We  demonstrate the usefulness  of the system on  a number of tasks.  For  example,  we perform independent components analysis on very large text  corpora consisting  of tens  of thousands of documents,  making  minimal  changes  to  the original Bell  and  Sejnowski Matlab source  (Bell  and  Se(cid:173) jnowski,  1995).  Applying ML techniques to data previously beyond their  reach leads to interesting analyses of both data and algorithms.
Search for Information Bearing Components in Speech
Authors: Hynek Hermansky Howard Hua Yang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this  paper,  we  use  mutual information to characterize  the  dis(cid:173) tributions of phonetic  and speaker/channel  information in  a  time(cid:173) frequency  space.  The  mutual information  (MI)  between  the  pho(cid:173) netic label and one feature,  and the joint mutual information (JMI)  between the phonetic label and two or three features are estimated .  The Miller's bias formulas for  entropy and mutual information es(cid:173) timates  are  extended  to include  higher  order  terms.  The  MI  and  the  JMI for  speaker/channel  recognition  are  also  estimated.  The  results  are complementary to those for phonetic classification.  Our  results  show  how  the  phonetic  information is  locally  spread  and  how  the  speaker/channel  information  is  globally  spread  in  time  and frequency.
An Oscillatory Correlation Frame work for Computational Auditory Scene Analysis
Authors: DeLiang L. Wang Guy J. Brown
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  neural  model  is  described  which  uses  oscillatory  correlation  to  segregate speech from  interfering sound sources. The core of the model  is  a two-layer neural  oscillator network.  A sound  stream  is  represented  by  a  synchronized  population  of oscillators,  and  different  streams  are  represented  by  desynchronized  oscillator  populations.  The  model  has  been evaluated using a corpus of speech mixed with interfering sounds,  and produces an  improvement in signal-to-noise ratio for every  mixture.
Manifold Stochastic Dynamics for Bayesian Learning
Authors: Yoram Baram Mark Zlochin
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We propose a new Markov Chain Monte Carlo algorithm which is a gen(cid:173) eralization of the stochastic dynamics method.  The algorithm performs  exploration of the state space using its intrinsic geometric structure, facil(cid:173) itating efficient sampling of complex distributions. Applied to Bayesian  learning in neural networks, our algorithm was found to perform at least  as well as the best state-of-the-art method while consuming considerably  less time.
Bifurcation Analysis of a Silicon Neuron
Authors: Gennady S. Cymbalyuk, Ronald L. Calabrese, Stephen P. DeWeerth Girish N. Patel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  have developed a VLSI  silicon  neuron  and  a corresponding  mathe(cid:173) matical  model  that  is  a two  state-variable system. We  describe the  cir(cid:173) cuit implementation and compare the  behaviors  observed in  the  silicon  neuron and the mathematical model. We also perform bifurcation analy(cid:173) sis of the mathematical model by  varying the externally applied current  and show that the behaviors exhibited by the silicon neuron under corre(cid:173) sponding  conditions  are  in  good  agreement  to  those  predicted  by  the  bifurcation analysis.
Speech Modelling Using Subspace and EM Techniques
Authors: João F. G. de Freitas, Tony Robinson, Mahesan Niranjan Gavin Smith
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The speech waveform can be modelled as  a piecewise-stationary linear  stochastic state space system, and its parameters can be estimated using  an  expectation-maximisation (EM)  algorithm.  One problem is  the  ini(cid:173) tialisation of the EM algorithm. Standard initialisation schemes can lead  to poor formant trajectories.  But these  trajectories however are impor(cid:173) tant for  vowel  intelligibility.  The aim of this paper is to investigate the  suitability of subspace identification methods to initialise EM.  The  paper  compares  the  subspace  state  space  system  identification  (4SID) method with the EM algorithm.  The 4SID and EM methods are  similar in that they both estimate a state sequence (but using Kalman fil(cid:173) ters and Kalman smoothers respectively),  and then estimate parameters  (but using least-squares and maximum likelihood respectively). The sim(cid:173) ilarity of 4SID and EM motivates the use of 4SID to initialise EM. Also,  4SID is non-iterative and requires no initialisation, whereas EM is itera(cid:173) tive and requires initialisation.  However 4SID is sub-optimal compared  to EM in a probabilistic sense. During experiments on real speech, 4SID  methods compare favourably with conventional initialisation techniques.  They produce smoother formant trajectories, have greater frequency res(cid:173) olution, and produce higher likelihoods. 
1  Work done while in Cambridge Engineering Dept., UK. 
Speech Modelling Using Subspace and EM Techniques 
797
Optimal Kernel Shapes for Local Linear Regression
Authors: Trevor Hastie Dirk Ormoneit
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Local linear regression performs very well in many low-dimensional  forecasting problems.  In high-dimensional spaces, its performance  typically  decays  due  to the  well-known  "curse-of-dimensionality".  A possible way to approach this problem is  by varying the  "shape"  of the weighting kernel.  In this work we suggest a new, data-driven  method to estimating the optimal  kernel  shape.  Experiments  us(cid:173) ing an artificially generated data set and data from  the UC  Irvine  repository show the benefits of kernel shaping.
Robust Neural Network Regression for Offline and Online Learning
Authors: Volker Tresp Thomas Briegel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  replace  the  commonly  used  Gaussian  noise  model  in  nonlinear  regression  by  a  more  flexible  noise  model  based  on  the  Student-t(cid:173) distribution.  The degrees of freedom of the t-distribution can be  chosen  such that as  special cases either the Gaussian distribution or the Cauchy  distribution are  realized.  The  latter is  commonly used in  robust regres(cid:173) sion.  Since the t-distribution can be  interpreted as being an infinite mix(cid:173) ture of Gaussians,  parameters and  hyperparameters such  as  the  degrees  of freedom of the t-distribution can be learned from the data based on an  EM-learning algorithm.  We  show that modeling using the  t-distribution  leads  to  improved  predictors  on  real  world  data  sets.  In  particular,  if  outliers are present,  the  t-distribution  is  superior to  the  Gaussian  noise  model.  In  effect, by  adapting  the  degrees of freedom,  the  system  can  "learn" to distinguish  between outliers  and  non-outliers.  Especially for  online learning tasks, one is  interested in  avoiding inappropriate weight  changes  due  to  measurement  outliers  to  maintain  stable  online  learn(cid:173) ing  capability.  We  show experimentally that using the  t-distribution  as  a noise model leads to stable online learning algorithms and outperforms  state-of-the art online learning methods like  the  extended Kalman filter  algorithm.
Neural Network Based Model Predictive Control
Authors: James D. Keeler, Greg Martin, Gene Boe, Doug Johnson, Mark Gerules Stephen Piche
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Model  Predictive  Control  (MPC),  a  control  algorithm which  uses  an  optimizer  to solve  for  the optimal  control  moves  over  a  future  time horizon based upon a model of the process, has become a stan(cid:173) dard control technique  in  the process industries over  the  past  two  decades.  In  most  industrial  applications,  a  linear  dynamic  model  developed using empirical data is  used even though the process it(cid:173) self is often nonlinear.  Linear models have been used because of the  difficulty  in  developing  a  generic  nonlinear  model  from  empirical  data and  the  computational expense  often involved  in  using  non(cid:173) linear models.  In  this  paper,  we  present  a  generic neural  network  based technique for developing nonlinear dynamic models from em(cid:173) pirical  data and show that these  models  can be efficiently  used  in  a  model predictive control framework.  This nonlinear MPC based  approach has been successfully implemented in a  number of indus(cid:173) trial  applications  in  the  refining,  petrochemical,  paper  and  food  industries.  Performance of the controller on  a  nonlinear industrial  process, a  polyethylene reactor, is  presented.
Coastal Navigation with Mobile Robots
Authors: Sebastian Thrun Nicholas Roy
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  problem  that  we  address  in  this paper  is  how  a mobile robot can  plan  in  order  to  arrive  at  its  goal  with  minimum  uncertainty.  Traditional  motion  planning algo(cid:173) rithms often assume that a mobile robot can track its position reliably, however, in real  world situations, reliable localization may not always be feasible.  Partially Observable  Markov Decision Processes (POMDPs) provide one way to maximize the certainty of  reaching  the goal  state,  but at the cost of computational  intractability for large state  spaces.  The  method  we propose  explicitly  models  the  uncertainty of the  robot's position as  a  state  variable,  and  generates  trajectories  through  the  augmented  pose-uncertainty  space.  By  minimizing  the  positional  uncertainty  at  the  goal,  the  robot reduces  the  likelihood  it  becomes  lost.  We  demonstrate  experimentally  that  coastal  navigation  reduces the uncertainty at the goal, especially with degraded localization.
Boosting with Multi-Way Branching in Decision Trees
Authors: David A. McAllester Yishay Mansour
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: It  is  known  that  decision  tree  learning  can  be  viewed  as  a  form  of boosting.  However,  existing boosting theorems for  decision tree  learning allow only binary-branching trees and the generalization to  multi-branching trees  is  not immediate.  Practical decision tree al(cid:173) gorithms, such  as CART and C4.5, implement a  trade-off between  the  number  of branches  and  the  improvement  in  tree  quality  as  measured  by an index function.  Here  we  give a  boosting justifica(cid:173) tion for a particular quantitative trade-off curve.  Our main theorem  states,  in  essence,  that if we  require  an  improvement proportional  to the  log  of the  number of branches  then  top-down  greedy  con(cid:173) struction of decision trees  remains an effective boosting algorithm.
Neural Representation of Multi-Dimensional Stimuli
Authors: Stefan D. Wilke, Helmut Schwegler Christian W. Eurich
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Spatial  information comes in two forms:  direct spatial information (for  example, retinal position) and indirect temporal contiguity information,  since objects encountered sequentially are in general spatially close. The  acquisition  of spatial  information  by  a  neural  network  is  investigated  here. Given a spatial layout of several objects, networks are trained on a  prediction task.  Networks using temporal sequences with no direct spa(cid:173) tial information are found to  develop internal representations that show  distances correlated with distances in the external layout.  The influence  of spatial information is analyzed by providing direct spatial information  to the system during training that is  either consistent with the layout or  inconsistent with  it.  This  approach  allows  examination of the relative  contributions of spatial and temporal contiguity.
Model Selection for Support Vector Machines
Authors: Vladimir Vapnik Olivier Chapelle
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: New functionals for parameter (model) selection of Support Vector Ma(cid:173) chines are introduced based on the concepts of the span of support vec(cid:173) tors and rescaling of the feature space.  It is shown that using these func(cid:173) tionals, one can both predict the best choice of parameters of the model  and the relative quality of performance for any value of parameter.
Memory Capacity of Linear vs. Nonlinear Models of Dendritic Integration
Authors: Bartlett W. Mel Panayiota Poirazi
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Previous biophysical modeling work showed that nonlinear interac(cid:173) tions among nearby synapses located on active dendritic trees can  provide a large boost in the memory capacity of a cell  (Mel,  1992a,  1992b).  The aim of our present work is  to quantify  this  boost by  estimating  the  capacity  of  (1)  a  neuron  model  with  passive  den(cid:173) dritic  integration  where  inputs  are  combined  linearly  across  the  entire cell  followed  by a  single  global  threshold,  and  (2)  an active  dendrite  model  in  which  a  threshold  is  applied  separately  to  the  output of each branch, and the branch subtotals are combined lin(cid:173) early.  We focus  here on the limiting case of binary-valued synaptic  weights,  and derive expressions which  measure model  capacity by  estimating the number of distinct input-output functions  available  to both neuron types.  We  show that  (1)  the application of a fixed  nonlinearity to each dendritic compartment substantially increases  the model's flexibility,  (2) for a neuron of realistic size, the capacity  of the nonlinear cell can exceed that of the same-sized linear cell by  more than an order of magnitude, and (3)  the largest capacity boost  occurs for cells with a relatively large number of dendritic subunits  of relatively  small  size.  We  validated  the analysis  by  empirically  measuring memory  capacity with  randomized two-class  classifica(cid:173) tion problems, where a stochastic delta rule was used to train both  linear and nonlinear models.  We  found  that large capacity boosts  predicted for  the nonlinear  dendritic  model  were  readily  achieved  in practice. 
-http://lnc.usc.edu 
158 
P.  Poirazi and B.  W.  Mel
State Abstraction in MAXQ Hierarchical Reinforcement Learning
Authors: Thomas G. Dietterich
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Many researchers have explored methods for hierarchical reinforce(cid:173) ment  learning  (RL)  with  temporal  abstractions,  in  which  abstract  actions are defined that can perform many primitive actions before  terminating.  However, little is known about learning with state ab(cid:173) stractions, in which aspects of the state space are ignored.  In previ(cid:173) ous work, we  developed the MAXQ method for  hierarchical RL.  In  this paper, we  define five  conditions under which state abstraction  can  be  combined  with  the  MAXQ  value  function  decomposition.  We  prove  that  the  MAXQ-Q  learning  algorithm  converges  under  these conditions and show experimentally that state abstraction is  important for  the successful  application of MAXQ-Q learning.
Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks
Authors: Samy Bengio Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The curse of dimensionality is  severe when modeling high-dimensional  discrete data:  the number of possible combinations of the variables ex(cid:173) plodes exponentially.  In  this  paper we propose a  new  architecture for  modeling high-dimensional data that requires resources (parameters and  computations) that grow only at most as the square of the number of vari(cid:173) ables,  using a multi-layer neural  network to represent the joint distribu(cid:173) tion of the variables as the product of conditional distributions. The neu(cid:173) ral  network can be interpreted as  a graphical model  without hidden ran(cid:173) dom variables, but in which the conditional distributions are tied through  the hidden units. The connectivity of the neural network can be pruned by  using dependency tests between the variables. Experiments on modeling  the distribution of several discrete data sets show statistically significant  improvements over other methods such as  naive Bayes and comparable  Bayesian networks,  and show  that significant improvements can be ob(cid:173) tained by pruning the network.
A Generative Model for Attractor Dynamics
Authors: Michael Mozer Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Attractor networks, which map an input space to a discrete out(cid:173) put space, are useful for pattern completion. However, designing  a net to have a given set of attractors is notoriously tricky; training  procedures are CPU intensive and often produce spurious afuac(cid:173) tors and ill-conditioned attractor basins.  These difficulties  occur  because each connection in the network participates in the encod(cid:173) ing of multiple attractors.  We describe an alternative formulation  of attractor networks in which the encoding of knowledge is local,  not distributed. Although localist attractor networks have similar  dynamics to their distributed counterparts, they are much easier  to work with and interpret. We propose a statistical formulation of  localist attract or net dynamics, which yields a convergence proof  and a mathematical interpretation of model parameters. 
Attractor networks map an input space,  usually continuous,  to a  sparse output  space composed of a discrete set of alternatives.  Attractor networks have a long  history in neural network research.  Attractor networks are often used for pattern completion, which involves filling  in  missing,  noisy,  or incorrect features  in an input pattern.  The  initial state of the  attractor net is  typically determined by the input pattern.  Over time, the state is  drawn to  one of a  predefined set of states-the attractors.  Attractor net dynam(cid:173) ics can be described by a state trajectory (Figure 1a).  An attractor net is generally  implemented by a set of visible units whose activity represents the instantaneous  state, and optionally, a set of hidden units that assist in the computation. Attractor  dynamics arise from interactions among the units.  In most formulations of afuac(cid:173) tor nets,2,3  the dynamics can be characterized by gradient descent in an energy  landscape, allowing one to partition the output space into attractor basins.  Instead  of homogeneous attractor basins, it is often desirable to sculpt basins that depend  on the recent history of the network and the arrangement of attractors in the space.  In psychological models of human cognition, for example, priming is fundamental:  after the model visits an attractor, it should be faster to fall into the same attractor  in the near future, i.e., the attractor basin should be broadened. 1 ,6  Another property of attractor nets is key to explaining behavioral data in psycho(cid:173) logical  and neurobiological  models:  the gang  effect,  in  which  the strength  of an  attractor is influenced by other attractors in its neighborhood. Figure 1b illustrates  the gang effect:  the proximity of the two rightmost afuactors creates a deeper at(cid:173) tractor basin, so that if the input starts at the origin it will get pulled to the right. 
A  Generative Model for Attractor Dynamics 
81
An Environment Model for Nonstationary Reinforcement Learning
Authors: Dit-Yan Yeung, Nevin Lianwen Zhang Samuel P. M. Choi
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Reinforcement learning in nonstationary environments is  generally  regarded  as  an  important  and  yet  difficult  problem.  This  paper  partially addresses the problem by formalizing a subclass of nonsta(cid:173) tionary environments.  The environment model, called hidden-mode  Markov  decision  process  (HM-MDP),  assumes that environmental  changes  are  always  confined  to  a  small  number  of hidden  modes.  A  mode  basically  indexes  a  Markov  decision  process  (MDP)  and  evolves  with  time according to a  Markov  chain.  While  HM-MDP  is  a  special  case  of partially  observable  Markov  decision  processes  (POMDP), modeling an HM-MDP environment via the more gen(cid:173) eral POMDP  model  unnecessarily increases the problem complex(cid:173) ity.  A variant of the Baum-Welch algorithm is  developed for model  learning requiring less  data and time.
Predictive App roaches for Choosing Hyperparameters in Gaussian Processes
Authors: S. Sathiya Keerthi S. Sundararajan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Gaussian  Processes  are  powerful  regression  models  specified  by  parametrized mean and covariance functions.  Standard approaches  to  estimate  these  parameters  (known  by  the  name  Hyperparam(cid:173) eters)  are  Maximum  Likelihood  (ML)  and  Maximum  APosterior  (MAP)  approaches.  In this paper, we  propose and investigate pre(cid:173) dictive  approaches,  namely,  maximization  of  Geisser's  Surrogate  Predictive Probability (GPP) and minimization of mean square er(cid:173) ror with  respect to GPP  (referred to as  Geisser's  Predictive mean  square  Error  (GPE))  to  estimate  the  hyperparameters.  We  also  derive  results  for  the  standard  Cross-Validation  (CV)  error  and  make  a  comparison.  These approaches are tested on  a  number of  problems and experimental results show that these approaches are  strongly competitive to existing approaches.
Acquisition in Autoshaping
Authors: Peter Dayan Sham Kakade
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Quantitative data on the speed with which animals acquire behav(cid:173) ioral responses during classical conditioning experiments should  provide strong constraints on models of learning.  However, most  models have simply ignored these data; the few that have attempt(cid:173) ed to address them have failed by at least an order of magnitude.  We discuss key data on the speed of acquisition, and show how to  account for them using a statistically sound model of learning, in  which differential reliabilities of stimuli playa crucial role.
Transductive Inference for Estimating Values of Functions
Authors: Vladimir Vapnik, Jason Weston Olivier Chapelle
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce an algorithm for  estimating the values of a  function  at a set of test points  Xe+!, ... , xl+m  given a set of training points  (XI,YI), ... ,(xe,Ye)  without  estimating  (as  an  intermediate  step)  the regression function .  We demonstrate that this direct (transduc(cid:173) ti ve)  way  for  estimating  values  of the  regression  (or  classification  in  pattern  recognition)  can  be  more  accurate  than  the  tradition(cid:173) alone  based  on  two  steps,  first  estimating  the  function  and  then  calculating the values of this function  at the  points of interest.
Effective Learning Requires Neuronal Remodeling of Hebbian Synapses
Authors: Isaac Meilijson, Eytan Ruppin Gal Chechik
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper revisits the classical neuroscience paradigm of Hebbian  learning.  We  find  that  a  necessary  requirement  for  effective  as(cid:173) sociative  memory  learning  is  that  the  efficacies  of  the  incoming  synapses  should  be  uncorrelated.  This  requirement  is  difficult  to  achieve in  a  robust  manner by  Hebbian synaptic learning,  since it  depends on network level information.  Effective learning can yet be  obtained by a neuronal process that maintains a zero sum of the in(cid:173) coming synaptic efficacies.  This normalization drastically improves  the  memory  capacity of associative networks,  from  an  essentially  bounded capacity to one that linearly scales with the network's size.  It also enables the effective storage of patterns with heterogeneous  coding levels in a single network.  Such neuronal normalization can  be successfully carried out by activity-dependent homeostasis of the  neuron's synaptic efficacies, which was recently observed in cortical  tissue.  Thus,  our findings  strongly suggest  that  effective  associa(cid:173) tive learning with Hebbian synapses alone is  biologically implausi(cid:173) ble and that Hebbian synapses must be continuously remodeled by  neuronally-driven regulatory processes in  the brain.
The Relevance Vector Machine
Authors: Michael E. Tipping
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The support vector machine  (SVM)  is  a state-of-the-art technique  for regression and classification, combining excellent generalisation  properties  with  a  sparse  kernel  representation.  However,  it  does  suffer from a number of disadvantages, notably the absence of prob(cid:173) abilistic outputs, the requirement to estimate a trade-off parameter  and the need to utilise  'Mercer' kernel functions.  In this paper we  introduce the Relevance  Vector Machine  (RVM),  a Bayesian treat(cid:173) ment  of a  generalised  linear  model  of identical  functional  form  to  the SVM.  The RVM  suffers  from  none of the above disadvantages,  and examples demonstrate that for  comparable generalisation per(cid:173) formance,  the RVM  requires dramatically fewer  kernel functions. 
1
Dual Estimation and the Unscented Transformation
Authors: Rudolph van der Merwe, Alex T. Nelson Eric A. Wan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Dual estimation  refers  to  the problem of simultaneously  estimating the  state of a dynamic system and the model which gives rise  to the dynam(cid:173) ics.  Algorithms  include  expectation-maximization (EM), dual  Kalman  filtering,  and joint Kalman methods.  These methods have recently  been  explored in  the context of nonlinear modeling,  where a  neural network  is  used  as  the functional form of the  unknown model.  Typically, an  ex(cid:173) tended  Kalman  filter  (EKF)  or smoother  is  used  for  the  part of the  al(cid:173) gorithm that estimates the clean state given the current estimated model.  An  EKF may  also be used to estimate the weights of the  network.  This  paper points  out the flaws  in  using  the  EKF,  and  proposes an  improve(cid:173) ment based on a new approach called the unscented transformation (UT)  [3].  A  substantial performance gain is  achieved  with  the same order of  computational complexity as  that of the standard EKF.  The approach is  illustrated on several dual estimation methods.
A Multi-class Linear Learning Algorithm Related to Winnow
Authors: Chris Mesterharm
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this  paper,  we  present  Committee,  a  new  multi-class  learning  algo(cid:173) rithm related  to the Winnow family  of algorithms.  Committee is  an  al(cid:173) gorithm for combining the predictions of a set of sub-experts in  the  on(cid:173) line mistake-bounded model oflearning. A sub-expert is a special type of  attribute that predicts with a distribution over a finite  number of classes.  Committee learns a linear function  of sub-experts and uses this function  to make class predictions.  We  provide bounds for Committee that show  it performs  well  when  the  target can  be represented  by  a  few  relevant  sub-experts.  We  also  show how Committee can  be  used  to solve more  traditional problems composed  of attributes.  This leads to a natural ex(cid:173) tension  that learns on  multi-class problems that contain both traditional  attributes and sub-experts.