mtybilly/Instruct2DS huggingface dataset:
Learning Joint Statistical Models for Audio-Visual Fusion and Segregation
Authors: John W. Fisher III, Trevor Darrell, William T. Freeman, Paul A. Viola
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: People can  understand complex auditory  and  visual  information,  often  using one to disambiguate the other.  Automated analysis, even at a low(cid:173) level,  faces  severe challenges, including  the  lack of accurate  statistical  models  for  the  signals,  and  their high-dimensionality  and  varied  sam(cid:173) pling rates.  Previous approaches  [6]  assumed simple parametric models  for the joint distribution which, while tractable, cannot capture the com(cid:173) plex signal relationships. We learn the joint distribution of the visual and  auditory  signals using a non-parametric approach.  First,  we project the  data into  a maximally  informative,  low-dimensional subspace,  suitable  for density estimation.  We  then  model the  complicated stochastic rela(cid:173) tionships between the  signals  using  a nonparametric density  estimator.  These  learned densities  allow  processing across  signal  modalities.  We  demonstrate,  on  synthetic and  real  signals,  localization  in  video  of the  face that is  speaking in  audio, and,  conversely, audio enhancement of a  particular speaker selected from the video.
Emergence of Movement Sensitive Neurons' Properties by Learning a Sparse Code for Natural Moving Images
Authors: Rafal Bogacz, Malcolm W. Brown, Christophe G. Giraud-Carrier
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Olshausen  &  Field  demonstrated  that  a  learning  algorithm  that  attempts  to  generate  a  sparse  code  for  natural  scenes  develops  a  complete  family  of localised,  oriented,  bandpass  receptive  fields,  similar  to  those  of  'simple  cells'  in  VI.  This  paper  describes  an  algorithm  which  finds  a  sparse  code  for  sequences  of images  that  preserves information  about the  input.  This  algorithm when  trained  on  natural  video  sequences  develops  bases  representing  the  movement in particular directions  with  particular speeds,  similar to  the  receptive  fields  of  the  movement-sensitive  cells  observed  in  cortical  visual  areas.  Furthermore,  to  previous  approaches  to  learning direction  selectivity,  the  timing  of neuronal  activity  encodes  the  phase  of the  movement,  so  the  precise  timing  of spikes is crucially important to  the information encoding.
A Support Vector Method for Clustering
Authors: Asa Ben-Hur, David Horn, Hava T. Siegelmann, Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present a novel  method for clustering using the support vector ma(cid:173) chine approach.  Data points are  mapped to  a high  dimensional  feature  space, where support vectors are used to define a sphere enclosing them.  The boundary of the sphere forms in data space a set of closed contours  containing the data.  Data points enclosed by each contour are defined as a  cluster. As the width parameter of the Gaussian kernel is decreased, these  contours fit  the data more tightly and  splitting of contours occurs.  The  algorithm  works  by  separating clusters  according  to  valleys  in  the un(cid:173) derlying probability distribution,  and  thus clusters can take on  arbitrary  geometrical shapes.  As in other SV algorithms, outliers can be dealt with  by introducing a soft margin constant leading to smoother cluster bound(cid:173) aries.  The  structure of the data is  explored by  varying the two parame(cid:173) ters.  We  investigate the dependence of our method on  these parameters  and apply it to several data sets.
Incremental and Decremental Support Vector Machine Learning
Authors: Gert Cauwenberghs, Tomaso Poggio
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: An on-line recursive algorithm for training support vector machines, one  vector  at  a  time,  is  presented.  Adiabatic  increments  retain  the  Kuhn(cid:173) Tucker  conditions  on  all  previously  seen  training  data,  in  a  number  of steps  each computed analytically.  The  incremental procedure is  re(cid:173) versible, and decremental "unlearning" offers an  efficient method to ex(cid:173) actly  evaluate  leave-one-out generalization performance.  Interpretation  of decremental unlearning in feature space sheds light on the relationship  between generalization and geometry of the data.
Active Inference in Concept Learning
Authors: Jonathan D. Nelson, Javier R. Movellan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: People  are  active  experimenters,  not  just  passive  observers,  constantly  seeking  new  information  relevant  to  their  goals.  A  reasonable  approach  to  active  information  gathering  is  to  ask  questions  and  conduct  experiments  that  maximize  the  expected  information  gain,  given  current  beliefs  (Fedorov  1972,  MacKay  1992,  Oaksford  &  Chater  1994).  In  this  paper  we  present  results  on  an  exploratory  experiment  designed  to  study  people's  active  information  gathering  behavior  on  a  concept  task  (Tenenbaum  2000).  The  results  of the  experiment are  analyzed  in  terms  of the  expected  information  gain  of the  questions  asked  by  subjects.
Sex with Support Vector Machines
Authors: Baback Moghaddam, Ming-Hsuan Yang
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Nonlinear  Support  Vector  Machines  (SVMs)  are  investigated  for  visual sex classification with low resolution  "thumbnail"  faces  (21- by-12  pixels)  processed  from  1,755  images  from  the  FE RET face  database.  The  performance  of SVMs  is  shown  to  be  superior  to  traditional pattern classifiers (Linear, Quadratic, Fisher Linear Dis(cid:173) criminant,  Nearest-Neighbor)  as  well  as  more  modern  techniques  such as Radial Basis Function (RBF) classifiers and large ensemble(cid:173) RBF  networks.  Furthermore, the  SVM  performance  (3.4%  error)  is  currently the best result reported in the open literature.
Recognizing Hand-written Digits Using Hierarchical Products of Experts
Authors: Guy Mayraz, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The  product  of  experts  learning  procedure  [1]  can  discover  a  set  of  stochastic binary features that constitute a non-linear generative model of  handwritten images of digits.  The quality of generative models learned  in this way can be assessed by learning a separate model for each class of  digit and  then comparing the unnormalized probabilities of test images  under the  10 different class-specific models.  To  improve discriminative  performance, it is helpful to learn a hierarchy of separate models for each  digit class.  Each model in the hierarchy has one layer of hidden units and  the nth level model is trained on data that consists of the activities of the  hidden  units  in  the  already  trained  (n  - l)th  level  model.  After train(cid:173) ing, each level produces a separate, unnormalized log probabilty score.  With a three-level hierarchy for each of the 10 digit classes, a test image  produces 30 scores  which  can  be used  as  inputs to  a supervised,  logis(cid:173) tic classification network that is trained on separate data.  On the MNIST  database, our system is comparable with current state-of-the-art discrimi(cid:173) native methods, demonstrating that the product of experts learning proce(cid:173) dure can produce effective generative models of high-dimensional data. 
1  Learning products of stochastic binary experts 
Hinton [1] describes a learning algorithm for probabilistic generative models that are com(cid:173) posed  of a  number of experts.  Each  expert specifies  a probability  distribution  over  the  visible variables and the experts are combined by  multiplying these distributions together  and renormalizing. 
(1) 
where d  is  a data vector in a discrete space, Om  is  all  the parameters of individual model  m, Pm(dIOm) is  the probability of d under model m, and  c  is  an  index over all  possible  vectors in  the data space. 
A Restricted Boltzmann machine [2,  3]  is  a special case of a product of experts in  which  each expert is  a single,  binary  stochastic  hidden unit that has  symmetrical connections to  a  set of visible units,  and connections between the hidden units are  forbidden.  Inference  in an RBM is  much easier than in a general Boltzmann machine and it is also much easier 
than in a causal belief net because there is  no explaining away.  There is  therefore no need  to  perform any iteration to determine the activities of the hidden units.  The hidden states,  Sj ,  are conditionally independent given the visible states,  Si,  and the distribution of Sj  is  given by the standard logistic function :
APRICODD: Approximate Policy Construction Using Decision Diagrams
Authors: Robert St-Aubin, Jesse Hoey, Craig Boutilier
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We propose a method of approximate dynamic programming for Markov  decision processes (MDPs) using algebraic decision diagrams  (ADDs).  We produce near-optimal value functions and policies with much lower  time  and  space  requirements  than  exact  dynamic  programming.  Our  method reduces  the  sizes  of the  intermediate value functions  generated  during value iteration by replacing the values at the terminals of the ADD  with  ranges  of values.  Our method is  demonstrated on  a class  of large  MDPs (with up to 34 billion states), and we compare the results with the  optimal value functions.
Four-legged Walking Gait Control Using a Neuromorphic Chip Interfaced to a Support Vector Learning Algorithm
Authors: Susanne Still, Bernhard Schölkopf, Klaus Hepp, Rodney J. Douglas
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: To  control  the  walking gaits  of a four-legged robot we  present a novel  neuromorphic  VLSI  chip  that  coordinates  the  relative  phasing  of the  robot's legs similar to how spinal Central Pattern Generators are believed  to  control  vertebrate  locomotion  [3].  The  chip  controls  the  leg  move(cid:173) ments by  driving motors  with time varying voltages which are  the  out(cid:173) puts of a small network of coupled oscillators.  The characteristics of the  chip's  output voltages  depend  on  a  set  of input parameters.  The rela(cid:173) tionship between input parameters and output voltages can be computed  analytically for an idealized system.  In  practice, however,  this  ideal  re(cid:173) lationship is only approximately true due to transistor mismatch and off(cid:173) sets.  Fine tuning of the chip's input parameters is done automatically by  the robotic system, using an unsupervised Support Vector (SV) learning  algorithm introduced  recently  [7].  The learning  requires  only  that  the  description of the desired output is  given.  The machine learns from (un(cid:173) labeled) examples how to set the parameters to the chip in order to obtain  a desired motor behavior.
A Mathematical Programming Approach to the Kernel Fisher Algorithm
Authors: Sebastian Mika, Gunnar Rätsch, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We investigate a new kernel-based classifier:  the Kernel Fisher Discrim(cid:173) inant (KFD). A mathematical programming formulation based on the ob(cid:173) servation that KFD maximizes the average margin permits an interesting  modification of the original KFD algorithm yielding the sparse KFD. We  find  that  both,  KFD  and  the  proposed sparse  KFD,  can  be  understood  in an unifying probabilistic context.  Furthermore, we  show connections  to Support Vector Machines and Relevance Vector Machines.  From this  understanding,  we  are  able  to  outline  an  interesting  kernel-regression  technique based upon the KFD  algorithm.  Simulations support the  use(cid:173) fulness  of our approach.
Using Free Energies to Represent Q-values in a Multiagent Reinforcement Learning Task
Authors: Brian Sallans, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The problem of reinforcement learning in large factored Markov decision  processes is explored. The Q-value of a state-action pair is approximated  by the free energy of a product of experts network.  Network parameters  are learned on-line using a modified SARSA algorithm which minimizes  the inconsistency of the Q-values of consecutive state-action pairs.  Ac(cid:173) tions are chosen based on the current value estimates by fixing the current  state and sampling actions from the network using Gibbs sampling. The  algorithm is  tested  on  a co-operative multi-agent task.  The product of  experts model is found to perform comparably to table-based Q-Iearning  for small instances of the task,  and  continues to perform well  when the  problem becomes too large for a table-based representation.
Dopamine Bonuses
Authors: Sham Kakade, Peter Dayan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Substantial data support a temporal difference (TO) model of  dopamine (OA) neuron activity in which the cells provide a global  error signal for reinforcement learning. However, in certain cir(cid:173) cumstances, OA activity seems anomalous under the TO model,  responding to non-rewarding stimuli. We address these anoma(cid:173) lies by suggesting that OA cells multiplex information about re(cid:173) ward bonuses, including Sutton's exploration bonuses and Ng et  al's non-distorting shaping bonuses. We interpret this additional  role for OA in terms of the unconditional attentional and psy(cid:173) chomotor effects of dopamine, having the computational role of  guiding exploration.
Sparse Greedy Gaussian Process Regression
Authors: Alex J. Smola, Peter L. Bartlett
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present  a  simple  sparse  greedy  technique  to  approximate  the  maximum a  posteriori estimate of Gaussian Processes  with much  improved  scaling  behaviour  in  the  sample  size  m.  In  particular,  computational  requirements  are  O(n2m),  storage  is  O(nm),  the  cost  for  prediction  is  0 ( n)  and  the  cost  to  compute  confidence  bounds  is  O(nm),  where  n  «:  m.  We  show  how  to  compute  a  stopping  criterion,  give  bounds  on  the  approximation  error,  and  show applications to large scale problems.
Large Scale Bayes Point Machines
Authors: Ralf Herbrich, Thore Graepel
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: also known as the Bayes point -
The concept of averaging over classifiers is fundamental to the  Bayesian analysis of learning. Based on this viewpoint, it has re(cid:173) cently been demonstrated for linear classifiers that the centre of  mass of version space (the set of all classifiers consistent with the  training set) - exhibits excel(cid:173) lent generalisation abilities. However, the billiard algorithm as pre(cid:173) sented in [4] is restricted to small sample size because it requires  o (m 2 ) of memory and 0 (N . m2 ) computational steps where m  is the number of training patterns and N is the number of random  draws from the posterior distribution. In this paper we present a  method based on the simple perceptron learning algorithm which  allows to overcome this algorithmic drawback. The method is al(cid:173) gorithmically simple and is easily extended to the multi-class case.  We present experimental results on the MNIST data set of hand(cid:173) written digits which show that Bayes point machines (BPMs) are  competitive with the current world champion, the support vector  machine. In addition, the computational complexity of BPMs can  be tuned by varying the number of samples from the posterior.  Finally, rejecting test points on the basis of their (approximative)  posterior probability leads to a rapid decrease in generalisation er(cid:173) ror, e.g. 0.1% generalisation error for a given rejection rate of 10%.
Efficient Learning of Linear Perceptrons
Authors: Shai Ben-David, Hans-Ulrich Simon
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  consider  the  existence  of efficient  algorithms for  learning the  class of half-spaces in ~n in the agnostic learning model (Le.,  mak(cid:173) ing no prior assumptions on the example-generating distribution).  The resulting combinatorial problem - finding  the best  agreement  half-space  over  an  input  sample - is  NP  hard  to  approximate to  within some constant factor.  We  suggest  a  way to circumvent this  theoretical bound by introducing a new measure of success for  such  algorithms.  An  algorithm is  IL-margin  successful  if the agreement  ratio of the half-space it outputs is as good as that of any half-space  once training points that are inside the IL-margins of its separating  hyper-plane are  disregarded.  We  prove  crisp  computational com(cid:173) plexity results with respect  to this success measure:  On one hand,  for  every positive IL,  there exist efficient  (poly-time) IL-margin suc(cid:173) cessful  learning  algorithms.  On  the  other  hand,  we  prove  that  unless  P=NP, there is  no  algorithm that runs  in time polynomial  in  the  sample  size  and  in  1/ IL  that  is  IL-margin  successful  for  all  IL> O.
Gaussianization
Authors: Scott Saobing Chen, Ramesh A. Gopinath
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: High dimensional data modeling is difficult mainly because the so-called  "curse of dimensionality". We propose a technique called "Gaussianiza(cid:173) tion" for high dimensional density estimation, which alleviates the curse  of dimensionality by exploiting the independence structures in  the data.  Gaussianization is  motivated from  recent developments in  the statistics  literature:  projection pursuit, independent component analysis and Gaus(cid:173) sian  mixture  models  with  semi-tied  covariances.  We  propose  an  iter(cid:173) ative  Gaussianization  procedure  which  converges  weakly:  at  each  it(cid:173) eration,  the  data is  first  transformed  to  the  least dependent coordinates  and then each coordinate is  marginally Gaussianized by univariate tech(cid:173) niques.  Gaussianization offers density estimation sharper than traditional  kernel  methods and radial  basis function  methods.  Gaussianization can  be viewed as efficient solution of nonlinear independent component anal(cid:173) ysis and high dimensional projection pursuit.
Error-correcting Codes on a Bethe-like Lattice
Authors: Renato Vicente, David Saad, Yoshiyuki Kabashima
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  analyze Gallager codes by employing a simple mean-field approxi(cid:173) mation that distorts the model geometry and preserves important interac(cid:173) tions between sites.  The method naturally recovers the probability prop(cid:173) agation decoding algorithm as  an extremization of a proper free-energy.  We  find  a thermodynamic phase transition that coincides with informa(cid:173) tion theoretical upper-bounds and explain the practical code performance  in terms of the free-energy landscape.
Homeostasis in a Silicon Integrate and Fire Neuron
Authors: Shih-Chii Liu, Bradley A. Minch
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In this work, we explore homeostasis in a silicon integrate-and-fire neu(cid:173) ron.  The neuron adapts its firing rate over long time periods on the order  of seconds or minutes so that it returns to its spontaneous firing rate after  a  lasting  perturbation.  Homeostasis  is  implemented  via  two  schemes.  One  scheme  looks  at  the  presynaptic  activity  and  adapts  the  synaptic  weight depending  on  the  presynaptic  spiking rate.  The  second scheme  adapts the synaptic "threshold" depending on the neuron's activity.  The  threshold is  lowered if the  neuron's activity  decreases over a long  time  and  is  increased  for  prolonged  increase  in  postsynaptic  activity.  Both  these  mechanisms  for  adaptation use  floating-gate  technology.  The re(cid:173) sults shown here are measured from  a chip fabricated in  a 2-J.lm  CMOS  process.
Incorporating Second-Order Functional Knowledge for Better Option Pricing
Authors: Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, René Garcia
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Incorporating prior knowledge of a particular task into the  architecture  of a learning algorithm can greatly improve generalization performance.  We  study  here  a case where we  know  that the  function  to be learned is  non-decreasing in two of its  arguments and convex in  one of them.  For  this purpose we propose a class of functions similar to multi-layer neural  networks but (1) that has those properties, (2) is a universal approximator  of continuous functions  with  these  and  other properties.  We  apply  this  new class  of functions  to  the task of modeling the price of call  options.  Experiments show improvements on regressing the price of call options  using the new types of function classes that incorporate the a priori con(cid:173) straints.
Spike-Timing-Dependent Learning for Oscillatory Networks
Authors: Silvia Scarpetta, Zhaoping Li, John A. Hertz
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  apply to oscillatory networks a  class of learning rules in which  synaptic weights change proportional to pre- and post-synaptic ac(cid:173) tivity,  with  a  kernel  A(r) measuring the effect  for  a  postsynaptic  spike a time r  after the presynaptic one.  The resulting synaptic ma(cid:173) trices have an outer-product form in which the oscillating patterns  are  represented  as  complex  vectors.  In  a  simple  model,  the even  part of A(r) enhances the resonant response to learned stimulus by  reducing the effective  damping, while the odd part determines the  frequency of oscillation.  We relate our model to the olfactory cortex  and hippocampus and their presumed roles  in forming  associative  memories and input representations.
On Reversing Jensen's Inequality
Authors: Tony Jebara, Alex Pentland
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Jensen's inequality is a powerful mathematical tool and one of the  workhorses in statistical learning. Its applications therein include the EM  algorithm, Bayesian estimation and Bayesian inference. Jensen com(cid:173) putes simple lower bounds on otherwise intractable quantities such as  products of sums and latent log-likelihoods. This simplification then per(cid:173) mits operations like integration and maximization. Quite often (i.e. in  discriminative learning) upper bounds are needed as well. We derive and  prove an efficient analytic inequality that provides such variational upper  bounds. This inequality holds for latent variable mixtures of exponential  family distributions and thus spans a wide range of contemporary statis(cid:173) tical models. We also discuss applications of the upper bounds including  maximum conditional likelihood, large margin discriminative models and  conditional Bayesian inference. Convergence, efficiency and prediction  results are shown. 1
From Mixtures of Mixtures to Adaptive Transform Coding
Authors: Cynthia Archer, Todd K. Leen
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We establish a principled framework for adaptive transform cod(cid:173) ing. Transform coders are often constructed by concatenating an ad  hoc choice of transform with suboptimal bit allocation and quan(cid:173) tizer design. Instead, we start from a probabilistic latent variable  model in the form of a mixture of constrained Gaussian mixtures.  From this model we derive a transform coding algorithm, which is  a constrained version of the generalized Lloyd algorithm for vector  quantizer design. A byproduct of our derivation is the introduc(cid:173) tion of a new transform basis, which unlike other transforms (PCA,  DCT, etc.) is explicitly optimized for coding. Image compression  experiments show adaptive transform coders designed with our al(cid:173) gorithm improve compressed image signal-to-noise ratio up to 3 dB  compared to global transform coding and 0.5 to 2 dB compared to  other adaptive transform coders.
Algorithmic Stability and Generalization Performance
Authors: Olivier Bousquet, André Elisseeff
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present a novel way of obtaining PAC-style bounds on the gen(cid:173) eralization error of learning algorithms, explicitly using their stabil(cid:173) ity properties.  A stable learner is one for which the learned solution  does  not  change much with small  changes in the training set.  The  bounds we  obtain do not depend on any measure of the complexity  of the hypothesis space  (e.g.  VC  dimension)  but rather depend on  how  the  learning  algorithm  searches  this  space,  and  can  thus  be  applied  even when the VC  dimension is  infinite.  We  demonstrate  that regularization networks possess the required stability property  and apply our method to obtain new bounds on their generalization  performance.
Higher-Order Statistical Properties Arising from the Non-Stationarity of Natural Signals
Authors: Lucas C. Parra, Clay Spence, Paul Sajda
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present  evidence  that  several  higher-order  statistical  proper(cid:173) ties of natural images and signals can be explained by a  stochastic  model  which  simply  varies  scale of an otherwise  stationary Gaus(cid:173) sian  process.  We  discuss  two  interesting  consequences.  The first  is  that  a  variety of natural signals  can be related  through a  com(cid:173) mon  model  of spherically invariant  random  processes,  which  have  the attractive property that the joint densities can be constructed  from the one dimensional marginal.  The second is that in some cas(cid:173) es the non-stationarity assumption and only second order methods  can be explicitly exploited to find  a  linear basis that is  equivalent  to  independent  components  obtained  with  higher-order  methods.  This is  demonstrated on spectro-temporal components of speech.
Probabilistic Semantic Video Indexing
Authors: Milind R. Naphade, Igor Kozintsev, Thomas S. Huang
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  propose a  novel probabilistic framework for  semantic video in(cid:173) dexing.  We  define  probabilistic  multimedia  objects  (multijects)  to  map  low-level  media  features  to  high-level  semantic  labels.  A  graphical network of such multijects (multinet) captures scene con(cid:173) text  by discovering intra-frame  as  well  as  inter-frame  dependency  relations  between  the  concepts.  The  main contribution is  a  novel  application  of  a  factor  graph  framework  to  model  this  network.  We  model  relations  between  semantic  concepts  in  terms  of their  co-occurrence as well  as the temporal dependencies  between these  concepts  within video  shots.  Using  the sum-product algorithm  [1]  for  approximate or exact inference in these factor graph multinets,  we  attempt  to  correct  errors made  during isolated  concept  detec(cid:173) tion by  forcing  high-level constraints.  This results  in  a  significant  improvement in the overall detection performance.
Accumulator Networks: Suitors of Local Probability Propagation
Authors: Brendan J. Frey, Anitha Kannan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: One  way  to  approximate  inference  in  richly-connected  graphical  models  is  to  apply  the  sum-product  algorithm  (a.k.a.  probabil(cid:173) ity propagation algorithm), while  ignoring the fact  that the graph  has cycles.  The sum-product  algorithm can  be directly applied in  Gaussian networks  and in  graphs for  coding,  but for  many condi(cid:173) tional probability functions  - including the sigmoid function  - di(cid:173) rect  application of the sum-product  algorithm is  not possible.  We  introduce  "accumulator networks"  that  have low  local  complexity  (but exponential global complexity) so  the sum-product algorithm  can be directly applied.  In an accumulator network, the probability  of a child given its parents is  computed by accumulating the inputs  from the parents in a Markov chain or more generally a tree.  After  giving  expressions  for  inference  and  learning  in  accumulator  net(cid:173) works,  we  give  results  on the  "bars problem"  and on the problem  of extracting translated, overlapping faces  from  an image.
Sparse Representation for Gaussian Process Models
Authors: Lehel Csató, Manfred Opper
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We develop an approach for a sparse representation for Gaussian Process  (GP) models in order to overcome the limitations of GPs caused by large  data sets.  The method is based on a combination of a Bayesian online al(cid:173) gorithm together with a sequential construction of a relevant subsample  of the  data  which  fully  specifies  the  prediction  of the  model.  Experi(cid:173) mental results on toy examples and large real-world data sets indicate the  efficiency of the approach.
Hippocampally-Dependent Consolidation in a Hierarchical Model of Neocortex
Authors: Szabolcs Káli, Peter Dayan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In  memory consolidation, declarative memories  which  initially  require  the  hippocampus  for  their recall,  ultimately  become independent of it.  Consolidation has  been the focus of numerous experimental and qualita(cid:173) tive modeling studies, but only little quantitative exploration. We present  a consolidation model  in  which  hierarchical  connections in  the  cortex,  that  initially  instantiate  purely  semantic  information  acquired  through  probabilistic  unsupervised learning,  come  to  instantiate episodic  infor(cid:173) mation  as  well.  The hippocampus is  responsible  for  helping complete  partial input patterns before consolidation is  complete, while also train(cid:173) ing the cortex to perform appropriate completion by itself.
Automated State Abstraction for Options using the U-Tree Algorithm
Authors: Anders Jonsson, Andrew G. Barto
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Learning  a  complex  task  can  be  significantly  facilitated  by  defining  a  hierarchy  of subtasks.  An  agent  can  learn  to  choose  between  various  temporally abstract actions, each solving an assigned subtask, to accom(cid:173) plish the overall task.  In this paper, we study hierarchical learning using  the  framework of options.  We  argue  that to take full  advantage  of hier(cid:173) archical  structure,  one  should perform option-specific  state  abstraction,  and  that if this  is  to  scale to larger tasks,  state abstraction should be au(cid:173) tomated.  We adapt McCallum's U-Tree algorithm to automatically build  option-specific  representations  of the  state  feature  space,  and  we  illus(cid:173) trate  the  resulting  algorithm  using  a  simple  hierarchical  task.  Results  suggest  that  automated  option-specific  state  abstraction  is  an  attractive  approach to making hierarchical learning systems more effective.
Structure Learning in Human Causal Induction
Authors: Joshua B. Tenenbaum, Thomas L. Griffiths
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We use graphical models to explore the question of how people learn sim(cid:173) ple causal relationships from data.  The two leading psychological theo(cid:173) ries  can both be seen as  estimating the parameters  of a fixed  graph.  We  argue  that a complete account of causal  induction should  also consider  how people learn the underlying causal graph structure, and we propose  to model this inductive process as  a Bayesian inference.  Our argument is  supported through the discussion of three data sets.
Minimum Bayes Error Feature Selection for Continuous Speech Recognition
Authors: George Saon, Mukund Padmanabhan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We consider the problem of designing a linear transformation ()  E lRPx n,  of rank p  ~ n, which  projects the features of a classifier x  E  lRn  onto  y  =  ()x  E  lRP  such  as  to  achieve  minimum Bayes  error (or probabil(cid:173) ity  of misclassification).  Two  avenues  will  be  explored:  the  first  is  to  maximize the ()-average divergence between the  class  densities  and  the  second is  to minimize the union Bhattacharyya bound in the range of ().  While both  approaches yield  similar performance in  practice,  they  out(cid:173) perform standard  LDA  features  and  show  a  10%  relative improvement  in  the  word  error rate  over state-of-the-art cepstral  features  on  a  large  vocabulary telephony speech recognition task.
Bayesian Video Shot Segmentation
Authors: Nuno Vasconcelos, Andrew Lippman
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Prior knowledge about video  structure  can be used both as  a means  to  improve the peiformance of content analysis and to extract features  that  allow  semantic  classification.  We  introduce  statistical models for two  important components of this  structure,  shot duration and activity,  and  demonstrate  the  usefulness  of these  models  by introducing  a  Bayesian  formulation for  the  shot segmentation problem.  The  new formulations  is  shown  to  extend standard  thresholding  methods  in  an  adaptive  and  intuitive way,  leading to  improved segmentation accuracy.
Kernel Expansions with Unlabeled Examples
Authors: Martin Szummer, Tommi Jaakkola
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Modern  classification  applications  necessitate  supplementing  the  few  available  labeled examples  with  unlabeled examples  to  improve classi(cid:173) fication performance.  We  present a new  tractable algorithm for exploit(cid:173) ing unlabeled examples in discriminative classification.  This is achieved  essentially by expanding the input vectors into longer feature vectors via  both labeled and unlabeled examples. The resulting classification method  can be interpreted as a discriminative kernel density estimate and is read(cid:173) ily trained via the EM algorithm, which in this case is both discriminative  and achieves the optimal solution.  We provide, in addition, a purely dis(cid:173) criminative formulation  of the  estimation  problem by  appealing  to  the  maximum entropy  framework.  We  demonstrate  that  the  proposed  ap(cid:173) proach requires very few  labeled examples for high classification accu(cid:173) racy.
Universality and Individuality in a Neural Code
Authors: Elad Schneidman, Naama Brenner, Naftali Tishby, Robert R. de Ruyter van Steveninck, William Bialek
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The  problem  of neural  coding  is  to  understand  how  sequences  of  action potentials (spikes)  are related to sensory stimuli, motor out(cid:173) puts,  or  (ultimately)  thoughts  and  intentions.  One clear  question  is whether the same coding rules are used  by  different  neurons, or  by  corresponding  neurons  in  different  individuals.  We  present  a  quantitative formulation of this problem using ideas from  informa(cid:173) tion theory, and apply this approach to the analysis of experiments  in the fly  visual  system.  We  find  significant  individual  differences  in  the  structure of the  code,  particularly  in  the  way  that tempo(cid:173) ral  patterns of spikes  are  used  to  convey information  beyond that  available from  variations in spike  rate.  On the other hand,  all  the  flies  in our ensemble exhibit  a  high coding efficiency,  so  that every  spike carries the same amount of information in all the individuals.  Thus  the  neural  code  has  a  quantifiable  mixture  of individuality  and universality.
Generalized Belief Propagation
Authors: Jonathan S. Yedidia, William T. Freeman, Yair Weiss
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Belief propagation  (BP)  was  only  supposed  to  work  for  tree-like  networks but works surprisingly well in many applications involving  networks  with  loops,  including  turbo  codes.  However,  there  has  been  little  understanding  of  the  algorithm  or  the  nature  of  the  solutions it finds  for  general graphs.  We  show  that  BP  can  only  converge  to  a  stationary  point  of an  approximate free  energy,  known as the Bethe free  energy in statis(cid:173) tical physics.  This result  characterizes BP fixed-points  and makes  connections with variational approaches to approximate inference.  More importantly, our analysis lets us  build  on the progress made  in  statistical  physics  since  Bethe's  approximation was  introduced  in 1935.  Kikuchi and others have shown how to construct more ac(cid:173) curate free energy approximations, of which Bethe's approximation  is  the  simplest.  Exploiting  the  insights  from  our analysis,  we  de(cid:173) rive generalized belief propagation (GBP) versions ofthese Kikuchi  approximations.  These  new  message  passing  algorithms  can  be  significantly more accurate than ordinary BP, at an adjustable in(cid:173) crease in complexity.  We  illustrate such  a  new  GBP algorithm on  a grid Markov network and show that it gives much more accurate  marginal probabilities than those found  using ordinary BP.
Multiple Timescales of Adaptation in a Neural Code
Authors: Adrienne L. Fairhall, Geoffrey D. Lewen, William Bialek, Robert R. de Ruyter van Steveninck
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Many neural systems extend their dynamic range by  adaptation.  We ex(cid:173) amine the  timescales  of adaptation  in  the  context of dynamically  mod(cid:173) ulated rapidly-varying stimuli,  and demonstrate in the  fly  visual system  that  adaptation  to  the  statistical  ensemble  of the  stimulus  dynamically  maximizes information transmission about the time-dependent stimulus.  Further, while the rate response has  long transients, the adaptation takes  place on timescales consistent with optimal variance estimation.
Speech Denoising and Dereverberation Using Probabilistic Models
Authors: Hagai Attias, John C. Platt, Alex Acero, Li Deng
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: This paper presents a unified probabilistic framework for denoising and  dereverberation of speech signals. The framework transforms the denois(cid:173) ing and dereverberation problems into Bayes-optimal signal estimation.  The  key  idea is  to  use  a  strong  speech  model  that  is  pre-trained on  a  large data set of clean speech.  Computational efficiency is  achieved by  using variational EM, working in  the frequency domain, and employing  conjugate priors.  The framework covers both single and multiple micro(cid:173) phones.  We  apply this  approach to noisy reverberant speech signals and  get results substantially better than standard methods.
Modelling Spatial Recall, Mental Imagery and Neglect
Authors: Suzanna Becker, Neil Burgess
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We present a computational model of the neural mechanisms in the pari(cid:173) etal and  temporal lobes that support spatial navigation, recall of scenes  and  imagery  of the  products  of recall.  Long  term  representations  are  stored  in  the  hippocampus,  and  are  associated  with  local  spatial  and  object-related features  in the parahippocampal region.  Viewer-centered  representations are dynamically generated from long term memory in the  parietal part of the  model.  The model thereby  simulates recall  and  im(cid:173) agery  of locations and  objects in  complex environments.  After parietal  damage,  the  model exhibits hemispatial neglect in  mental imagery that  rotates  with  the  imagined perspective of the  observer,  as  in  the  famous  Milan  Square experiment  [1].  Our model  makes  novel  predictions for  the  neural representations  in  the  parahippocampal and  parietal  regions  and for behavior in healthy volunteers and neuropsychological patients.
Text Classification using String Kernels
Authors: Huma Lodhi, John Shawe-Taylor, Nello Cristianini, Christopher J. C. H. Watkins
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  introduce  a  novel  kernel  for  comparing two  text  documents.  The  kernel  is  an  inner  product  in  the  feature  space  consisting  of  all  subsequences  of  length  k.  A  subsequence  is  any  ordered  se(cid:173) quence  of k characters occurring in the text though not necessarily  contiguously.  The subsequences  are  weighted  by  an exponentially  decaying factor  of their  full  length  in  the  text,  hence  emphasising  those  occurrences  which  are  close  to  contiguous.  A  direct  compu(cid:173) tation of this feature vector  would involve a  prohibitive amount of  computation  even  for  modest  values  of k,  since  the  dimension  of  the feature space grows exponentially with k.  The paper describes  how despite this fact the inner product can be efficiently evaluated  by a dynamic programming technique.  A preliminary experimental  comparison of the performance of the kernel compared with a  stan(cid:173) dard  word  feature  space  kernel  results. 
[6]  is  made showing  encouraging
Fast Training of Support Vector Classifiers
Authors: Fernando Pérez-Cruz, Pedro Luis Alarcón-Diana, Angel Navia-Vázquez, Antonio Artés-Rodríguez
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In this communication we present a new algorithm for solving Support  Vector Classifiers (SVC) with large training data sets. The new algorithm  is based on an Iterative Re-Weighted Least Squares procedure which is  used to  optimize the SVc. Moreover,  a novel sample selection strategy  for the working set is presented,  which randomly chooses  the working  set among the training  samples that do  not fulfill  the stopping criteria.  The validity  of both proposals,  the  optimization procedure and sample  selection  strategy,  is  shown  by  means  of computer experiments  using  well-known data sets.
Beyond Maximum Likelihood and Density Estimation: A Sample-Based Criterion for Unsupervised Learning of Complex Models
Authors: Sepp Hochreiter, Michael Mozer
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The goal of many unsupervised learning procedures is to bring two  probability  distributions  into  alignment.  Generative  models  such  as  Gaussian mixtures and Boltzmann machines can be cast in this  light,  as  can recoding models  such  as ICA  and projection pursuit.  We propose a novel sample-based error measure for these classes of  models, which applies even in situations where maximum likelihood  (ML)  and  probability  density  estimation-based  formulations  can(cid:173) not be applied,  e.g.,  models that are nonlinear  or have intractable  posteriors.  Furthermore,  our  sample-based  error  measure  avoids  the difficulties of approximating a  density function.  We  prove that  with  an unconstrained  model,  (1)  our approach  converges  on  the  correct solution as the number of samples goes to infinity,  and  (2)  the expected solution of our approach in the generative framework  is  the  ML  solution.  Finally,  we  evaluate our approach via simula(cid:173) tions of linear and nonlinear models  on  mixture of Gaussians and  ICA problems.  The experiments show the broad applicability and  generality of our approach.
A Neural Probabilistic Language Model
Authors: Yoshua Bengio, Réjean Ducharme, Pascal Vincent
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A goal  of statistical language modeling is  to  learn  the joint probability  function of sequences of words.  This is intrinsically difficult because of  the curse of dimensionality:  we propose to fight it with its own weapons.  In the proposed approach one learns simultaneously (1) a distributed rep(cid:173) resentation for each word (i.e.  a similarity between words) along with (2)  the probability function for word sequences, expressed with these repre(cid:173) sentations.  Generalization is  obtained because a sequence of words that  has  never been seen before gets  high probability if it is  made of words  that are similar to words forming an already seen sentence.  We report on  experiments using neural networks for the probability function, showing  on  two  text  corpora that  the  proposed approach  very  significantly  im(cid:173) proves on a state-of-the-art trigram model.
Support Vector Novelty Detection Applied to Jet Engine Vibration Spectra
Authors: Paul M. Hayton, Bernhard Schölkopf, Lionel Tarassenko, Paul Anuzis
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A system has  been developed to  extract diagnostic information from jet  engine carcass vibration data.  Support Vector Machines applied to nov(cid:173) elty  detection provide a measure  of how  unusual  the  shape  of a  vibra(cid:173) tion signature is, by learning a representation of normality.  We describe  a novel method for  Support Vector Machines  of including information  from a second class for novelty detection and give results from the appli(cid:173) cation to Jet Engine vibration analysis.