mtybilly/Instruct2DS huggingface dataset:
Learning from User Feedback in Image Retrieval Systems
Authors: Andrew Lippman Nuno Vasconcelos
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  formulate  the  problem  of retrieving  images  from  visual  databases  as  a problem of Bayesian inference.  This leads to  natural and effective  solutions for two of the most challenging issues in the design of a retrieval  system:  providing  support  for  region-based  queries  without  requiring  prior  image  segmentation,  and  accounting  for  user-feedback  during  a  retrieval  session.  We  present  a  new  learning  algorithm  that  relies  on  belief propagation to account for both positive and negative examples of  the user's interests.
Online Independent Component Analysis with Local Learning Rate Adaptation
Authors: Xavier Giannakopoulos Nicol N. Schraudolph
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Stochastic meta-descent (SMD) is a new technique for online adap(cid:173) tation  of local  learning  rates  in  arbitrary twice-differentiable  sys(cid:173) tems.  Like matrix momentum it uses full  second-order information  while  retaining  O(n)  computational  complexity  by  exploiting  the  efficient  computation  of Hessian-vector  products.  Here  we  apply  SMD  to independent  component  analysis,  and  employ  the  result(cid:173) ing  algorithm  for  the  blind  separation  of time-varying  mixtures.  By matching individual learning rates to the rate of change in each  source signal's  mixture coefficients,  our technique is  capable of si(cid:173) multaneously tracking sources that move at very different,  a priori  unknown speeds.
A Variational Baysian Framework for Graphical Models
Authors: Hagai Attias
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper presents a novel practical framework for Bayesian model  averaging  and  model  selection  in  probabilistic  graphical  models.  Our approach approximates full  posterior distributions over model  parameters and structures, as well as latent variables, in an analyt(cid:173) ical  manner.  These  posteriors fall  out of a  free-form  optimization  procedure,  which  naturally  incorporates  conjugate  priors.  Unlike  in  large sample  approximations,  the posteriors are  generally  non(cid:173) Gaussian and no Hessian needs to be computed.  Predictive quanti(cid:173) ties  are obtained analytically.  The resulting algorithm generalizes  the standard Expectation Maximization algorithm, and its conver(cid:173) gence  is  guaranteed.  We  demonstrate  that  this  approach  can  be  applied  to  a  large  class  of  models  in  several  domains,  including  mixture models and source separation.
Algebraic Analysis for Non-regular Learning Machines
Authors: Sumio Watanabe
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Hierarchical learning machines are non-regular and non-identifiable  statistical models, whose true parameter sets are analytic sets with  singularities.  Using  algebraic  analysis,  we  rigorously  prove  that  the  stochastic  complexity  of  a  non-identifiable  learning  machine  (ml  - 1) log log n  + const.,  is  asymptotically  equal  to  >'1  log n  - where n is the number of training samples.  Moreover we show that  the rational  number >'1  and the integer ml  can be algorithmically  calculated  using  resolution  of singularities  in  algebraic  geometry.  Also we obtain inequalities 0 < >'1  ~ d/2 and 1  ~ ml  ~ d,  where d  is  the number of parameters.
Model Selection in Clustering by Uniform Convergence Bounds
Authors: Marcus Held Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Unsupervised  learning  algorithms  are  designed  to  extract  struc(cid:173) ture from  data samples.  Reliable  and robust  inference  requires  a  guarantee that extracted structures are typical for the data source,  Le.,  similar  structures  have  to be  inferred  from  a  second  sample  set of the same data source.  The overfitting phenomenon in max(cid:173) imum  entropy  based  annealing  algorithms  is  exemplarily  studied  for  a  class of histogram clustering  models.  Bernstein's  inequality  for  large deviations is used to determine the maximally achievable  approximation  quality  parameterized  by  a  minimal  temperature.  Monte Carlo simulations support the proposed model selection cri(cid:173) terion by finite  temperature annealing.
Unmixing Hyperspectral Data
Authors: Clay Spence, Paul Sajda, Andreas Ziehe, Klaus-Robert Müller Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In hyperspectral imagery one pixel  typically  consists of a  mixture  of the  reflectance  spectra of several  materials,  where  the  mixture  coefficients  correspond  to  the  abundances of the  constituting  ma(cid:173) terials.  We  assume linear combinations of reflectance spectra with  some additive normal sensor noise and derive a  probabilistic MAP  framework  for  analyzing  hyperspectral  data.  As  the  material re(cid:173) flectance characteristics are not know  a priori, we face  the problem  of  unsupervised  linear  unmixing.  The  incorporation  of  different  prior  information  (e.g.  positivity  and  normalization  of the  abun(cid:173) dances)  naturally  leads  to  a  family  of  interesting  algorithms,  for  example  in  the  noise-free  case  yielding  an  algorithm  that  can  be  understood as constrained independent component analysis (ICA).  Simulations underline the usefulness of our theory.
Some Theoretical Results Concerning the Convergence of Compositions of Regularized Linear Functions
Authors: Tong Zhang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Recently,  sample complexity bounds have been derived for problems in(cid:173) volving linear functions such as neural networks and support vector ma(cid:173) chines.  In this paper,  we extend some theoretical results in this area by  deriving dimensional independent covering number bounds for regular(cid:173) ized  linear functions under certain regularization conditions.  We  show  that such bounds lead to a class of new methods for training linear clas(cid:173) sifiers with similar theoretical advantages of the support vector machine.  Furthermore,  we also present a theoretical analysis for these new meth(cid:173) ods from the asymptotic statistical point of view.  This technique provides  better description for large sample behaviors of these algorithms.
Inference for the Generalization Error
Authors: Yoshua Bengio Claude Nadeau
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In order to to compare learning algorithms, experimental results reported  in  the  machine  learning  litterature often  use  statistical  tests  of signifi(cid:173) cance.  Unfortunately,  most of these  tests  do  not take  into account the  variability  due  to  the  choice  of training  set.  We  perform  a  theoretical  investigation of the variance of the cross-validation estimate of the gen(cid:173) eralization error that takes into account the  variability due to  the choice  of training  sets.  This  allows  us  to  propose  two  new  ways  to  estimate  this variance. We show, via simulations, that these new statistics perform  well relative to the statistics considered by Dietterich (Dietterich, 1998).
LTD Facilitates Learning in a Noisy Environment
Authors: Gerardina Hernández Paul W. Munro
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Long-term potentiation (LTP) has long been held as a biological  substrate for associative learning. Recently, evidence has emerged  that long-term depression (LTD) results when the presynaptic cell  fires after the postsynaptic cell. The computational utility of LTD  is explored here. Synaptic modification kernels for both LTP and  LTD have been proposed by other laboratories based studies of one  postsynaptic unit. Here, the interaction between time-dependent  LTP and LTD is studied in small networks.
An Oculo-Motor System with Multi-Chip Neuromorphic Analog VLSI Control
Authors: Steve Gyger Oliver Landolt
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A system emulating the functionality of a moving eye-hence the name  oculo-motor system-has been built and successfully tested.  It is  made  of an optical device for shifting the field of view of an image sensor by up  to 45 ° in  any direction, four neuromorphic analog VLSI circuits imple(cid:173) menting an oculo-motor control loop, and some off-the-shelf electronics.  The custom integrated circuits communicate with each other primarily by  non-arbitrated address-event buses.  The system  implements the behav(cid:173) iors of saliency-based saccadic exploration, and smooth pursuit of light  spots.  The duration of saccades ranges from  45 ms  to  100 ms,  which  is  comparable to human eye performance. Smooth pursuit operates on light  sources moving at up to 50 0 /s in the visual field.
Understanding Stepwise Generalization of Support Vector Machines: a Toy Model
Authors: Mirta B. Gordon Sebastian Risau-Gusman
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this  article  we  study the effects  of introducing structure in the  input distribution of the data to be learnt by a  simple perceptron.  We  determine the learning curves within the framework of Statis(cid:173) tical  Mechanics.  Stepwise  generalization  occurs  as  a  function  of  the number of examples when the distribution of patterns is highly  anisotropic.  Although  extremely simple,  the model  seems  to cap(cid:173) ture  the  relevant  features  of  a  class  of  Support  Vector  Machines  which was  recently shown to present this behavior.
A Geometric Interpretation of v-SVM Classifiers
Authors: Christopher J. C. Burges David J. Crisp
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We show that the recently proposed variant of the Support Vector  machine  (SVM)  algorithm,  known  as  v-SVM,  can  be  interpreted  as a maximal separation between subsets of the convex hulls of the  data,  which  we  call  soft  convex  hulls.  The  soft  convex  hulls  are  controlled by  choice of the parameter v.  If the intersection of the  convex hulls is empty, the hyperplane is positioned halfway between  them such that the distance between convex hulls, measured along  the normal, is  maximized; and if it is not, the hyperplane's normal  is  similarly determined  by  the  soft  convex  hulls,  but  its  position  (perpendicular  distance  from  the  origin)  is  adjusted  to  minimize  the  error  sum.  The  proposed  geometric  interpretation of v-SVM  also leads to necessary and sufficient conditions for the existence of  a choice of v  for  which  the v-SVM solution is nontrivial.
Robust Learning of Chaotic Attractors
Authors: Jaap C. Schouten, Marc-Olivier Coppens, Floris Takens, C. Lee Giles, Cor M. van den Bleek Rembrandt Bakker
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A fundamental problem with the modeling of chaotic time series data is that  minimizing short-term prediction errors does not guarantee a match  between the reconstructed attractors of model and experiments. We  introduce a modeling paradigm that simultaneously learns to short-tenn  predict and to locate the outlines of the attractor by a new way of nonlinear  principal component analysis. Closed-loop predictions are constrained to  stay within these outlines, to prevent divergence from the attractor. Learning  is exceptionally fast: parameter estimation for the 1000 sample laser data  from the 1991 Santa Fe time series competition took less than a minute on  a 166 MHz Pentium PC.
Greedy Importance Sampling
Authors: Dale Schuurmans
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: I present a simple variation of importance sampling that explicitly search(cid:173) es for  important regions in the target distribution.  I prove that the tech(cid:173) nique yields unbiased estimates,  and show empirically it can reduce the  variance  of standard Monte Carlo estimators.  This is achieved  by con(cid:173) centrating samples in more significant regions of the sample space.
Recurrent Cortical Competition: Strengthen or Weaken?
Authors: Lars Schwabe, Christian Piepenbrock, Klaus Obermayer Péter Adorján
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We investigate the short term .dynamics of the recurrent competition and  neural  activity in the primary visual cortex in terms of information pro(cid:173) cessing and in the context of orientation selectivity.  We propose that af(cid:173) ter stimulus onset, the strength of the recurrent excitation decreases due  to  fast  synaptic depression.  As  a consequence,  the network shifts from  an  initially highly  nonlinear to a  more  linear operating  regime.  Sharp  orientation tuning is established in the first highly competitive phase.  In  the second and less competitive phase, precise signaling of multiple ori(cid:173) entations and long range modulation, e.g.,  by intra- and inter-areal con(cid:173) nections becomes possible (surround effects).  Thus the network first ex(cid:173) tracts  the  salient  features  from  the  stimulus,  and  then  starts  to  process  the details.  We  show  that  this  signal  processing  strategy  is  optimal  if  the neurons have limited bandwidth and their objective is to transmit the  maximum amount of information in any time interval beginning with the  stimulus onset.
Approximate Inference A lgorithms for Two-Layer Bayesian Networks
Authors: Michael I. Jordan Andrew Y. Ng
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present  a  class  of approximate  inference  algorithms  for  graphical  models  of the  QMR-DT type.  We  give  convergence rates  for  these  al(cid:173) gorithms and  for  the  Jaakkola and  Jordan  (1999) algorithm,  and  verify  these  theoretical predictions empirically.  We  also present empirical re(cid:173) sults on the difficult QMR-DT network problem, obtaining performance  of the  new  algorithms roughly  comparable  to  the  Jaakkola and  Jordan  algorithm.
Perceptual Organization Based on Temporal Dynamics
Authors: DeLiang L. Wang Xiuwen Liu
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A figure-ground  segregation  network is  proposed  based on a  novel  boundary  pair  representation.  Nodes  in  the  network  are  bound(cid:173) ary  segments  obtained  through  local  grouping.  Each  node  is  ex(cid:173) citatorily  coupled  with  the  neighboring  nodes  that  belong  to  the  same region, and inhibitorily coupled with the corresponding paired  node.  Gestalt grouping rules are incorporated by  modulating con(cid:173) nections.  The  status  of  a  node  represents  its  probability  being  figural  and  is  updated  according  to  a  differential  equation.  The  system solves the figure-ground segregation problem through tem(cid:173) poral  evolution.  Different  perceptual  phenomena,  such  as  modal  and  amodal completion,  virtual contours,  grouping and  shape de(cid:173) composition are then explained through local diffusion.  The system  eliminates combinatorial optimization and accounts for  many psy(cid:173) chophysical results with a  fixed  set of parameters.
Learning Informative Statistics: A Nonparametnic Approach
Authors: Alexander T. Ihler, Paul A. Viola John W. Fisher III
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We discuss an information theoretic approach for categorizing and mod(cid:173) eling dynamic processes. The approach can learn a compact and informa(cid:173) tive statistic which summarizes past states to predict future observations.  Furthermore, the  uncertainty of the prediction is characterized nonpara(cid:173) metrically by a joint density over the learned statistic and present obser(cid:173) vation.  We  discuss the  application of the technique to both noise driven  dynamical systems and random processes sampled from a density which  is conditioned on the past. In the first case we show results in which both  the  dynamics of random walk and the  statistics of the  driving noise  are  captured.  In the second case we  present results in  which a summarizing  statistic  is  learned  on  noisy  random telegraph  waves  with  differing de(cid:173) pendencies on past states.  In both cases the algorithm yields a principled  approach for discriminating processes with differing dynamics and/or de(cid:173) pendencies.  The method is  grounded in  ideas  from  information theory  and nonparametric statistics.
Rules and Similarity in Concept Learning
Authors: Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper argues that two apparently distinct modes of generalizing con(cid:173) cepts - abstracting rules and computing similarity to exemplars - should  both be seen as special cases of a more general Bayesian learning frame(cid:173) work.  Bayes explains the specific workings of these two modes - which  rules are abstracted,  how similarity is measured - as  well  as  why gener(cid:173) alization should appear rule- or similarity-based in different situations.  This  analysis also suggests why  the rules/similarity distinction, even  if  not computationally fundamental,  may  still be useful at the algorithmic  level as part of a principled approximation to fully Bayesian learning.
Support Vector Method for Novelty Detection
Authors: Robert C. Williamson, Alex J. Smola, John Shawe-Taylor, John C. Platt Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Suppose you are given some dataset drawn from an underlying probabil(cid:173) ity distribution P  and you want to estimate a "simple" subset S  of input  space such that the probability that a test point drawn from P lies outside  of S equals some a priori specified l/ between 0 and 1.  We  propose a method to  approach this  problem by trying to estimate a  function f  which is positive on S  and negative on the complement.  The  functional form of f  is given by a kernel expansion in terms of a poten(cid:173) tially small subset of the training data; it is regularized by controlling the  length of the weight vector in  an associated feature space.  We provide a  theoretical analysis of the statistical performance of our algorithm.  The algorithm is  a natural extension of the  support vector algorithm to  the case of unlabelled data.
Generalized Model Selection for Unsupervised Learning in High Dimensions
Authors: Byron Dom Shivakumar Vaithyanathan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We describe a Bayesian approach to  model  selection in unsupervised  learning  that  determines  both  the  feature  set  and  the  number  of  clusters. We then evaluate this scheme (based on marginal likelihood)  and  one  based  on  cross-validated  likelihood.  For  the  Bayesian  scheme  we  derive  a  closed-form  solution  of the  marginal  likelihood  by  assuming  appropriate  forms  of the  likelihood  function  and  prior.  Extensive  experiments  compare these  approaches  and  all  results  are  verified  by comparison against ground truth.  In  these experiments the  Bayesian scheme using our objective function gave better results than  cross-validation.
An Improved Decomposition Algorithm for Regression Support Vector Machines
Authors: Pavel Laskov
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  new  decomposition  algorithm  for  training  regression  Support  Vector  Machines  (SVM)  is  presented.  The  algorithm  builds  on  the  basic  principles  of decomposition  proposed  by  Osuna et.  al.,  and addresses the issue of optimal working set  selection.  The new  criteria for  testing optimality  of a  working set are derived.  Based  on these  criteria, the principle of  "maximal inconsistency"  is  pro(cid:173) posed to form (approximately) optimal working sets.  Experimental  results show superior performance of the new algorithm in compar(cid:173) ison with traditional training of regression SVM without decompo(cid:173) sition.  Similar results have been previously reported on decomposi(cid:173) tion algorithms for  pattern recognition SVM. The new algorithm is  also applicable to advanced SVM formulations based on regression,  such as density estimation  and integral equation SVM. 
1
An Analog VLSI Model of Periodicity Extraction
Authors: André van Schaik
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: that extracts 
This paper presents an electronic system  the  periodicity of a sound. It uses three analogue VLSI building  blocks: a silicon cochlea, two inner-hair-cell circuits and two  spiking neuron chips. The silicon cochlea consists of a cascade of  filters. Because of the delay between two outputs from the silicon  cochlea, spike trains created at these outputs are synchronous only  for a narrow range of periodicities. In contrast to traditional band(cid:173) pass filters, where an increase in' selectivity has to be traded off  against a decrease in response time, the proposed system responds  quickly, independent of selectivity.
From Coexpression to Coregulation: An Approach to Inferring Transcriptional Regulation among Gene Classes from Large-Scale Expression Data
Authors: Tobias Mann, Rebecca Castaño, Barbara J. Wold Eric Mjolsness
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: small-scale  gene 
regulation  networks 
We  provide  preliminary  evidence  that  eXlstmg  algorithms  for  inferring  from  gene  expression  data  can  be  adapted  to  large-scale  gene  expression  data  coming  from  hybridization  microarrays.  The essential  steps are  (1)  clustering  many  genes  by  their  expression  time-course  data  into  a  minimal  set  of  clusters  of  co-expressed  genes,  (2)  theoretically  modeling  the  various  conditions  under  which  the  time-courses  are  measured  using  a  continious-time  analog  recurrent  neural  network  for  the  cluster  mean  time-courses,  (3)  fitting  such  a  regulatory  model  to  the  cluster  mean  time  courses  by  simulated  annealing  with  weight  decay,  and  (4)  analysing  several  such  fits  for  commonalities  the  connection  matrices.  This  procedure  can  be  used  to  assess  the  adequacy  of existing  and  future  gene  expression  time-course  data  sets  for  determ ining  transcriptional  regulatory  relationships  such  as  coregulation . 
the  circuit  parameter  sets
Data Visualization and Feature Selection: New Algorithms for Nongaussian Data
Authors: John Moody Howard Hua Yang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Data  visualization  and  feature  selection  methods  are  proposed  based on the )oint mutual information and ICA.  The visualization  methods can find  many good 2-D  projections for  high dimensional  data interpretation,  which  cannot be easily found by  the other ex(cid:173) isting methods.  The new  variable selection  method is found  to be  better in eliminating redundancy in the inputs than other methods  based  on  simple mutual information.  The efficacy  of the  methods  is illustrated on a radar signal analysis problem to find  2-D viewing  coordinates for  data visualization and to select  inputs for  a  neural  network  classifier.  Keywords:  feature  selection,  joint mutual information,  ICA,  vi(cid:173) sualization, classification.
An Information-Theoretic Framework for Understanding Saccadic Eye Movements
Authors: Stella X. Yu Tai Sing Lee
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this paper,  we  propose that information maximization can  pro(cid:173) vide  a  unified  framework  for  understanding  saccadic  eye  move(cid:173) ments.  In  this framework,  the mutual information among the cor(cid:173) tical  representations  of the  retinal  image,  the  priors  constructed  from  our  long  term  visual  experience,  and  a  dynamic  short-term  internal  representation  constructed  from  recent  saccades  provides  a  map for  guiding eye  navigation .  By  directing  the  eyes  to loca(cid:173) tions  of maximum complexity in  neuronal  ensemble  responses  at  each  step,  the  automatic saccadic  eye  movement system  greedily  collects information about the external world, while modifying the  neural  representations  in  the  process.  This  framework  attempts  to  connect  several  psychological  phenomena,  such  as  pop-out  and  inhibition of return,  to long term visual experience  and short term  working  memory.  It  also  provides  an  interesting  perspective  on  contextual computation and formation of neural  representation  in  the visual system.
Noisy Neural Networks and Generalizations
Authors: Alexander Roitershtein, Asa Ben-Hur Hava T. Siegelmann
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this paper we  define  a  probabilistic computational model which  generalizes many noisy neural network models, including the recent  work  of Maass and Sontag [5].  We identify weak  ergodicjty  as  the  mechanism responsible for  restriction  of the computational power  of  probabilistic  models  to  definite  languages,  independent  of the  characteristics  of the noise:  whether  it is  discrete  or analog,  or if  it  depends  on  the  input  or  not,  and  independent  of whether  the  variables  are  discrete  or continuous.  We  give examples of weakly  ergodic  models including  noisy  computational systems  with noise  depending on the current state and inputs, aggregate models,  and  computational systems which update in continuous time.
The Nonnegative Boltzmann Machine
Authors: David J. C. MacKay, Daniel D. Lee Oliver B. Downs
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The nonnegative Boltzmann machine (NNBM) is a recurrent neural net(cid:173) work model that can describe multimodal nonnegative data.  Application  of maximum likelihood estimation to this model gives a learning rule that  is analogous to the binary Boltzmann machine. We examine the utility of  the mean field  approximation for the  NNBM,  and describe  how Monte  Carlo sampling techniques can be  used to  learn its parameters.  Reflec(cid:173) tive  slice  sampling  is  particularly well-suited  for  this  distribution,  and  can efficiently be implemented to sample the  distribution.  We  illustrate  learning of the NNBM on a transiationally invariant distribution, as well  as on a generative model for images of human faces.
Boosting Algorithms as Gradient Descent
Authors: Jonathan Baxter, Peter L. Bartlett, Marcus R. Frean Llew Mason
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  provide an abstract characterization of boosting algorithms as  gradient  decsent  on  cost-functionals  in  an  inner-product  function  space.  We  prove convergence  of these functional-gradient-descent  algorithms under quite  weak conditions.  Following previous theo(cid:173) retical  results  bounding the generalization  performance of convex  combinations of classifiers in terms of general cost functions of the  margin,  we  present  a  new  algorithm  (DOOM  II)  for  performing a  gradient descent  optimization of such cost functions.  Experiments  on  several  data sets  from  the  UC  Irvine  repository  demonstrate  that DOOM II generally outperforms AdaBoost, especially in high  noise situations, and that the overfitting behaviour of AdaBoost is  predicted by our cost functions.
Local Probability Propagation for Factor Analysis
Authors: Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Ever since Pearl's probability propagation algorithm in graphs with  cycles  was  shown  to  produce excellent  results  for  error-correcting  decoding  a  few  years  ago,  we  have  been  curious  about  whether  local  probability  propagation  could  be  used  successfully  for  ma(cid:173) chine  learning.  One  of the simplest  adaptive models  is  the factor  analyzer,  which  is  a  two-layer  network  that  models  bottom  layer  sensory inputs as a  linear combination of top layer factors  plus in(cid:173) dependent  Gaussian  sensor noise.  We  show  that local probability  propagation in the factor analyzer network usually takes just a few  iterations to perform accurate inference, even in networks with 320  sensors and 80 factors.  We derive an expression for  the algorithm's  fixed  point and show that this fixed  point matches the exact solu(cid:173) tion in a variety of networks, even when the fixed point is  unstable.  We also show that this method can be used successfully to perform  inference for approximate EM and we give results on an online face  recognition task.  1  Factor analysis  A  simple way  to encode  input patterns is  to suppose that  each input  can  be  well(cid:173) approximated by a linear combination of component vectors,  where the amplitudes  of the vectors are modulated to match the input.  For a given training set,  the most  appropriate set  of component vectors  will  depend  on  how  we  expect  the  modula(cid:173) tion levels  to  behave  and how  we  measure the distance  between  the input  and its  approximation.  These  effects  can  be  captured  by  a  generative  probabilit~ model  that  specifies  a  distribution  p(z)  over  modulation  levels  z  =  (Zl, ... ,ZK)  and  a  distribution  p(xlz)  over  sensors  x  =  (Xl, ... ,XN)T  given  the  modulation  levels.  Principal component analysis,  independent component analysis and factor analysis  can be viewed as maximum likelihood learning in a model of this type, where we as(cid:173) sume that over the training set, the appropriate modulation levels  are independent  and the overall distortion is  given  by  the sum of the individual sensor distortions. 
In  factor  analysis,  the  modulation  levels  are  called  factors  and  the  distributions  have the following  form: 
p(Zk)  =  N(Zk; 0,1), 
p(z)  =  nf=lP(Zk) =  N(z; 0, I), 
p(xnl z) =  N(xn; E~=l AnkZk, 'l/Jn), 
(1)  The parameters of this model  are the factor  loading matrix A,  with elements  Ank,  and the diagonal sensor noise  covariance matrix  'It,  with  diagonal elements 'l/Jn.  A  belief network for  the factor analyzer is  shown in Fig.  1a.  The likelihood is 
p(xlz) =  n:=IP(xnlz) =  N(x; Az, 'It). 
p(x) = 1 N(z; 0, I)N(x; Az, 'It)dz = N(x; 0, AA T  + 'It),
A MCMC Approach to Hierarchical Mixture Modelling
Authors: Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: There  are  many  hierarchical  clustering  algorithms  available,  but these  lack a firm  statistical basis.  Here we  set up a hierarchical probabilistic  mixture model, where data is  generated in  a hierarchical tree-structured  manner. Markov chain Monte Carlo (MCMC) methods are demonstrated  which can  be used to  sample from  the  posterior distribution  over trees  containing variable numbers of hidden units.
The Infinite Gaussian Mixture Model
Authors: Carl Edward Rasmussen
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In a Bayesian mixture model it is not necessary a priori to limit the num(cid:173) ber of components to be finite.  In this paper an infinite Gaussian mixture  model is  presented which neatly sidesteps the difficult problem of find(cid:173) ing the "right" number of mixture components. Inference in the model is  done using an efficient parameter-free Markov Chain that relies entirely  on Gibbs sampling.
Learning the Similarity of Documents: An Information-Geometric Approach to Document Retrieval and Categorization
Authors: Thomas Hofmann
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  project  pursued  in  this  paper  is  to  develop  from  first  information-geometric  principles  a  general  method  for  learning  the  similarity  between  text  documents.  Each  individual  docu(cid:173) ment  is  modeled  as  a  memoryless  information source.  Based  on  a  latent  class  decomposition of the  term-document  matrix, a  low(cid:173) dimensional  (curved)  multinomial subfamily is  learned.  From this  model a  canonical similarity function - known as the Fisher  kernel  - is  derived.  Our  approach  can  be  applied  for  unsupervised  and  supervised  learning problems alike.  This in  particular covers inter(cid:173) esting  cases  where  both,  labeled and  unlabeled  data are  available.  Experiments in  automated indexing and text categorization verify  the advantages of the proposed  method.
Bayesian Reconstruction of 3D Human Motion from Single-Camera Video
Authors: Michael E. Leventon, William T. Freeman Nicholas R. Howe
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The three-dimensional motion of humans is  underdetermined when  the  observation is  limited to a single camera, due to the inherent 3D ambi(cid:173) guity of 2D video.  We present a system that reconstructs the 3D motion  of human subjects from single-camera video, relying on prior knowledge  about  human motion,  learned  from  training  data,  to  resolve  those am(cid:173) biguities.  After initialization in  2D, the tracking and 3D reconstruction  is  automatic;  we show results  for several  video sequences.  The results  show the power of treating 3D body tracking as an inference problem.
Mixture Density Estimation
Authors: Andrew R. Barron Jonathan Q. Li
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Gaussian mixtures (or so-called radial basis function networks)  for  density estimation provide a natural counterpart to sigmoidal neu(cid:173) ral networks for function fitting and approximation.  In both cases,  it is  possible to give  simple expressions for  the iterative improve(cid:173) ment of performance as components of the network are introduced  one at a time.  In particular, for mixture density estimation we show  that a k-component mixture estimated by maximum likelihood  (or  by an iterative likelihood improvement that we introduce) achieves  log-likelihood  within order  1/k of the log-likelihood  achievable by  any convex combination.  Consequences for  approximation and es(cid:173) timation  using  Kullback-Leibler  risk  are  also  given.  A  Minimum  Description Length principle selects the optimal number of compo(cid:173) nents k that minimizes the risk bound.
Information Capacity and Robustness of Stochastic Neuron Models
Authors: Idan Segev, Naftali Tishby Elad Schneidman
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  reliability  and  accuracy  of  spike  trains  have  been  shown  to  depend  on  the  nature  of  the  stimulus  that  the  neuron  encodes.  Adding  ion  channel  stochasticity  to  neuronal  models  results  in  a  macroscopic behavior that replicates the input-dependent reliabili(cid:173) ty and precision of real neurons.  We  calculate the amount of infor(cid:173) mation that an ion channel based stochastic Hodgkin-Huxley (HH)  neuron model can encode about a wide set of stimuli.  We show that  both the information  rate and the information  per  spike  of the s(cid:173) tochastic model are similar to the values  reported experimentally.  Moreover,  the  amount  of information  that  the  neuron  encodes  is  correlated with the amplitude of fluctuations in the input, and less  so with the average firing rate of the neuron.  We also show that for  the HH  ion  channel density,  the information capacity is  robust  to  changes  in  the density  of ion  channels  in  the  membrane,  whereas  changing  the  ratio  between  the  Na+  and  K+  ion  channels  has  a  considerable effect on the information that the neuron can encode.  Finally,  we  suggest  that neurons  may maximize  their information  capacity by appropriately balancing the density of the different ion  channels that underlie neuronal excitability.
Bayesian Transduction
Authors: Ralf Herbrich, Klaus Obermayer Thore Graepel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Transduction  is  an  inference  principle  that  takes  a  training sam(cid:173) ple  and aims at estimating the values of a function  at given points  contained in the so-called working sample as  opposed  to the whole  of input space  for  induction.  Transduction  provides  a  confidence  measure  on  single  predictions  rather  than  classifiers  - a  feature  particularly important for  risk-sensitive  applications.  The possibly  infinite number of functions  is  reduced  to a finite  number of equiv(cid:173) alence classes on the working sample.  A rigorous Bayesian analysis  reveals  that for  standard classification loss  we  cannot benefit  from  considering  more  than  one  test  point  at  a  time.  The  probability  of  the  label  of a  given  test  point  is  determined  as  the  posterior  measure of the corresponding subset  of hypothesis  space.  We  con(cid:173) sider the PAC setting of binary classification by linear discriminant  functions  (perceptrons)  in kernel space such that the probability of  labels  is  determined  by  the  volume  ratio  in  version  space.  We  suggest  to sample this  region  by  an  ergodic  billiard.  Experimen(cid:173) tal  results  on  real  world data indicate that Bayesian Transduction  compares  favourably  to  the  well-known  Support  Vector  Machine,  in  particular  if the  posterior  probability  of labellings  is  used  as  a  confidence  measure to exclude test  points of low  confidence.
Constructing Heterogeneous Committees Using Input Feature Grouping: Application to Economic Forecasting
Authors: John E. Moody Yuansong Liao
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  committee  approach  has  been  proposed  for  reducing  model  uncertainty  and  improving  generalization  performance.  The  ad(cid:173) vantage of committees depends  on  (1)  the performance of individ(cid:173) ual members  and  (2)  the correlational structure of errors between  members.  This paper presents an input grouping technique for  de(cid:173) signing a  heterogeneous  committee.  With  this  technique,  all input  variables are first grouped based on their mutual information.  Sta(cid:173) tistically  similar  variables  are  assigned  to  the  same  group.  Each  member's  input  set  is  then  formed  by  input  variables  extracted  from different groups.  Our designed  committees have less error cor(cid:173) relation between its members, since each member observes different  input variable combinations.  The individual member's feature sets  contain less redundant information, because highly correlated vari(cid:173) ables will not be combined together.  The member feature sets con(cid:173) tain almost complete information, since each set contains a feature  from  each  information group.  An  empirical study for  a  noisy  and  nonstationary  economic  forecasting  problem  shows  that  commit(cid:173) tees constructed by our proposed technique outperform committees  formed using several existing techniques.
An Analysis of Turbo Decoding with Gaussian Densities
Authors: Benjamin Van Roy Paat Rusmevichientong
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  provide  an  analysis  of  the  turbo  decoding  algorithm  (TDA)  in  a  setting involving  Gaussian  densities.  In  this  context,  we  are  able  to show  that  the  algorithm  converges  and  that  - somewhat  surprisingly - though the density generated by the TDA may differ  significantly from  the desired posterior density,  the means of these  two densities  coincide. 
1
Probabilistic Methods for Support Vector Machines
Authors: Peter Sollich
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: I  describe  a  framework  for  interpreting Support  Vector  Machines  (SVMs)  as  maximum  a  posteriori  (MAP)  solutions  to  inference  problems with Gaussian Process priors.  This can provide intuitive  guidelines  for  choosing  a  'good'  SVM  kernel.  It  can  also  assign  (by  evidence  maximization)  optimal values  to parameters such  as  the noise level C which cannot be determined unambiguously from  properties of the MAP  solution alone  (such as  cross-validation er(cid:173) ror) . I illustrate this using a simple approximate expression for the  SVM  evidence.  Once  C  has  been determined,  error bars on SVM  predictions can also  be obtained. 
1  Support  Vector Machines:  A  probabilistic framework 
Support  Vector  Machines  (SVMs)  have  recently  been  the  subject  of  intense  re(cid:173) search  activity  within  the  neural  networks  community;  for  tutorial  introductions  and overviews of recent developments see  [1,  2,  3].  One of the open questions that  remains is  how  to set the 'tunable' parameters of an SVM  algorithm:  While meth(cid:173) ods for  choosing the width of the kernel function and the noise parameter C  (which  controls  how  closely  the  training  data  are  fitted)  have  been  proposed  [4,  5]  (see  also, very recently,  [6]), the effect of the overall shape of the kernel function remains  imperfectly understood [1].  Error bars  (class probabilities) for  SVM  predictions - important for safety-critical applications, for example - are also difficult to obtain.  In this paper I suggest that a probabilistic interpretation of SVMs  could be used to  tackle these problems.  It shows that the SVM  kernel defines  a  prior over functions  on the input space, avoiding the need to think in terms of high-dimensional feature  spaces.  It also allows one to define quantities such as the evidence  (likelihood) for a  set of hyperparameters (C,  kernel amplitude Ko  etc).  I give a simple approximation  to the  evidence  which  can  then  be  maximized  to set  such  hyperparameters.  The  evidence is sensitive to the values of C and Ko individually, in contrast to properties  (such  as  cross-validation  error)  of the  deterministic  solution,  which  only  depends  on the product  CKo.  It can thfrefore be used  to  assign an unambiguous  value  to  C, from  which  error bars can be derived. 
350
Neural System Model of Human Sound Localization
Authors: Simon Carlile Craig T. Jin
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper examines the role of biological constraints in the human audi(cid:173) tory localization process.  A psychophysical and neural system modeling  approach  was  undertaken  in which  performance  comparisons between  competing  models  and  a  human  subject  explore  the  relevant  biologi(cid:173) cally plausible  "realism  constraints".  The  directional  acoustical  cues,  upon which sound localization is  based,  were  derived  from  the  human  subject's  head-related transfer functions  (HRTFs).  Sound stimuli  were  generated by convolving bandpass noise with the HRTFs and were pre(cid:173) sented to both the subject and the model.  The input stimuli to the model  was processed using the Auditory Image Model of cochlear processing.  The  cochlear  data  was  then  analyzed  by  a  time-delay  neural  network  which integrated temporal and spectral information to determine the spa(cid:173) tial  location  of the  sound  source.  The  combined  cochlear  model  and  neural network provided a system model of the sound localization pro(cid:173) cess.  Human-like  localization performance  was  qualitatively  achieved  for broadband and bandpass stimuli when the model architecture incor(cid:173) porated frequency division (or tonotopicity), and was trained using vari(cid:173) able bandwidth and center-frequency sounds.
A Neuromorphic VLSI System for Modeling the Neural Control of Axial Locomotion
Authors: Edgar A. Brown, Stephen P. DeWeerth Girish N. Patel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We have developed and tested an analog/digital VLSI system that mod(cid:173) els the coordination of biological segmental oscillators underlying axial  locomotion in animals such as leeches and lampreys. In its current form  the system consists of a chain of twelve pattern generating circuits that  are capable of arbitrary contralateral inhibitory synaptic coupling. Each  pattern generating circuit is implemented with two independent silicon  Morris-Lecar neurons with a total of 32 programmable (floating-gate  based) inhibitory synapses, and an asynchronous address-event inter(cid:173) connection element that provides synaptic connectivity and implements  axonal delay. We describe and analyze the data from a set of experi(cid:173) ments exploring the system behavior in terms of synaptic coupling.
Graded Grammaticality in Prediction Fractal Machines
Authors: Peter Tiño, Georg Dorffner Shan Parfitt
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce  a  novel  method  of constructing  language  models,  which  avoids some of the problems associated  with recurrent  neu(cid:173) ral networks.  The method of creating a Prediction Fractal Machine  (PFM)  [1]  is  briefly described  and some experiments are presented  which  demonstrate the suitability of PFMs for  language modeling.  PFMs  distinguish  reliably  between  minimal  pairs,  and  their  be(cid:173) havior is  consistent  with the hypothesis  [4]  that wellformedness  is  'graded' not absolute.  A discussion of their potential to offer fresh  insights into language acquisition and processing follows.