mtybilly/Instruct2DS huggingface dataset:
Higher-Order Statistical Properties Arising from the Non-Stationarity of Natural Signals
Authors: Clay Spence, Paul Sajda Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present  evidence  that  several  higher-order  statistical  proper(cid:173) ties of natural images and signals can be explained by a  stochastic  model  which  simply  varies  scale of an otherwise  stationary Gaus(cid:173) sian  process.  We  discuss  two  interesting  consequences.  The first  is  that  a  variety of natural signals  can be related  through a  com(cid:173) mon  model  of spherically invariant  random  processes,  which  have  the attractive property that the joint densities can be constructed  from the one dimensional marginal.  The second is that in some cas(cid:173) es the non-stationarity assumption and only second order methods  can be explicitly exploited to find  a  linear basis that is  equivalent  to  independent  components  obtained  with  higher-order  methods.  This is  demonstrated on spectro-temporal components of speech.
Probabilistic Semantic Video Indexing
Authors: Igor Kozintsev, Thomas S. Huang Milind R. Naphade
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  propose a  novel probabilistic framework for  semantic video in(cid:173) dexing.  We  define  probabilistic  multimedia  objects  (multijects)  to  map  low-level  media  features  to  high-level  semantic  labels.  A  graphical network of such multijects (multinet) captures scene con(cid:173) text  by discovering intra-frame  as  well  as  inter-frame  dependency  relations  between  the  concepts.  The  main contribution is  a  novel  application  of  a  factor  graph  framework  to  model  this  network.  We  model  relations  between  semantic  concepts  in  terms  of their  co-occurrence as well  as the temporal dependencies  between these  concepts  within video  shots.  Using  the sum-product algorithm  [1]  for  approximate or exact inference in these factor graph multinets,  we  attempt  to  correct  errors made  during isolated  concept  detec(cid:173) tion by  forcing  high-level constraints.  This results  in  a  significant  improvement in the overall detection performance.
Accumulator Networks: Suitors of Local Probability Propagation
Authors: Anitha Kannan Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: One  way  to  approximate  inference  in  richly-connected  graphical  models  is  to  apply  the  sum-product  algorithm  (a.k.a.  probabil(cid:173) ity propagation algorithm), while  ignoring the fact  that the graph  has cycles.  The sum-product  algorithm can  be directly applied in  Gaussian networks  and in  graphs for  coding,  but for  many condi(cid:173) tional probability functions  - including the sigmoid function  - di(cid:173) rect  application of the sum-product  algorithm is  not possible.  We  introduce  "accumulator networks"  that  have low  local  complexity  (but exponential global complexity) so  the sum-product algorithm  can be directly applied.  In an accumulator network, the probability  of a child given its parents is  computed by accumulating the inputs  from the parents in a Markov chain or more generally a tree.  After  giving  expressions  for  inference  and  learning  in  accumulator  net(cid:173) works,  we  give  results  on the  "bars problem"  and on the problem  of extracting translated, overlapping faces  from  an image.
Sparse Representation for Gaussian Process Models
Authors: Manfred Opper Lehel Csató
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We develop an approach for a sparse representation for Gaussian Process  (GP) models in order to overcome the limitations of GPs caused by large  data sets.  The method is based on a combination of a Bayesian online al(cid:173) gorithm together with a sequential construction of a relevant subsample  of the  data  which  fully  specifies  the  prediction  of the  model.  Experi(cid:173) mental results on toy examples and large real-world data sets indicate the  efficiency of the approach.
Hippocampally-Dependent Consolidation in a Hierarchical Model of Neocortex
Authors: Peter Dayan Szabolcs Káli
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In  memory consolidation, declarative memories  which  initially  require  the  hippocampus  for  their recall,  ultimately  become independent of it.  Consolidation has  been the focus of numerous experimental and qualita(cid:173) tive modeling studies, but only little quantitative exploration. We present  a consolidation model  in  which  hierarchical  connections in  the  cortex,  that  initially  instantiate  purely  semantic  information  acquired  through  probabilistic  unsupervised learning,  come  to  instantiate episodic  infor(cid:173) mation  as  well.  The hippocampus is  responsible  for  helping complete  partial input patterns before consolidation is  complete, while also train(cid:173) ing the cortex to perform appropriate completion by itself.
Automated State Abstraction for Options using the U-Tree Algorithm
Authors: Andrew G. Barto Anders Jonsson
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Learning  a  complex  task  can  be  significantly  facilitated  by  defining  a  hierarchy  of subtasks.  An  agent  can  learn  to  choose  between  various  temporally abstract actions, each solving an assigned subtask, to accom(cid:173) plish the overall task.  In this paper, we study hierarchical learning using  the  framework of options.  We  argue  that to take full  advantage  of hier(cid:173) archical  structure,  one  should perform option-specific  state  abstraction,  and  that if this  is  to  scale to larger tasks,  state abstraction should be au(cid:173) tomated.  We adapt McCallum's U-Tree algorithm to automatically build  option-specific  representations  of the  state  feature  space,  and  we  illus(cid:173) trate  the  resulting  algorithm  using  a  simple  hierarchical  task.  Results  suggest  that  automated  option-specific  state  abstraction  is  an  attractive  approach to making hierarchical learning systems more effective.
Structure Learning in Human Causal Induction
Authors: Thomas L. Griffiths Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We use graphical models to explore the question of how people learn sim(cid:173) ple causal relationships from data.  The two leading psychological theo(cid:173) ries  can both be seen as  estimating the parameters  of a fixed  graph.  We  argue  that a complete account of causal  induction should  also consider  how people learn the underlying causal graph structure, and we propose  to model this inductive process as  a Bayesian inference.  Our argument is  supported through the discussion of three data sets.
Minimum Bayes Error Feature Selection for Continuous Speech Recognition
Authors: Mukund Padmanabhan George Saon
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We consider the problem of designing a linear transformation ()  E lRPx n,  of rank p  ~ n, which  projects the features of a classifier x  E  lRn  onto  y  =  ()x  E  lRP  such  as  to  achieve  minimum Bayes  error (or probabil(cid:173) ity  of misclassification).  Two  avenues  will  be  explored:  the  first  is  to  maximize the ()-average divergence between the  class  densities  and  the  second is  to minimize the union Bhattacharyya bound in the range of ().  While both  approaches yield  similar performance in  practice,  they  out(cid:173) perform standard  LDA  features  and  show  a  10%  relative improvement  in  the  word  error rate  over state-of-the-art cepstral  features  on  a  large  vocabulary telephony speech recognition task.
Bayesian Video Shot Segmentation
Authors: Andrew Lippman Nuno Vasconcelos
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Prior knowledge about video  structure  can be used both as  a means  to  improve the peiformance of content analysis and to extract features  that  allow  semantic  classification.  We  introduce  statistical models for two  important components of this  structure,  shot duration and activity,  and  demonstrate  the  usefulness  of these  models  by introducing  a  Bayesian  formulation for  the  shot segmentation problem.  The  new formulations  is  shown  to  extend standard  thresholding  methods  in  an  adaptive  and  intuitive way,  leading to  improved segmentation accuracy.
Kernel Expansions with Unlabeled Examples
Authors: Tommi Jaakkola Martin Szummer
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Modern  classification  applications  necessitate  supplementing  the  few  available  labeled examples  with  unlabeled examples  to  improve classi(cid:173) fication performance.  We  present a new  tractable algorithm for exploit(cid:173) ing unlabeled examples in discriminative classification.  This is achieved  essentially by expanding the input vectors into longer feature vectors via  both labeled and unlabeled examples. The resulting classification method  can be interpreted as a discriminative kernel density estimate and is read(cid:173) ily trained via the EM algorithm, which in this case is both discriminative  and achieves the optimal solution.  We provide, in addition, a purely dis(cid:173) criminative formulation  of the  estimation  problem by  appealing  to  the  maximum entropy  framework.  We  demonstrate  that  the  proposed  ap(cid:173) proach requires very few  labeled examples for high classification accu(cid:173) racy.
Universality and Individuality in a Neural Code
Authors: Naama Brenner, Naftali Tishby, Robert R. de Ruyter van Steveninck, William Bialek Elad Schneidman
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The  problem  of neural  coding  is  to  understand  how  sequences  of  action potentials (spikes)  are related to sensory stimuli, motor out(cid:173) puts,  or  (ultimately)  thoughts  and  intentions.  One clear  question  is whether the same coding rules are used  by  different  neurons, or  by  corresponding  neurons  in  different  individuals.  We  present  a  quantitative formulation of this problem using ideas from  informa(cid:173) tion theory, and apply this approach to the analysis of experiments  in the fly  visual  system.  We  find  significant  individual  differences  in  the  structure of the  code,  particularly  in  the  way  that tempo(cid:173) ral  patterns of spikes  are  used  to  convey information  beyond that  available from  variations in spike  rate.  On the other hand,  all  the  flies  in our ensemble exhibit  a  high coding efficiency,  so  that every  spike carries the same amount of information in all the individuals.  Thus  the  neural  code  has  a  quantifiable  mixture  of individuality  and universality.
Generalized Belief Propagation
Authors: William T. Freeman, Yair Weiss Jonathan S. Yedidia
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Belief propagation  (BP)  was  only  supposed  to  work  for  tree-like  networks but works surprisingly well in many applications involving  networks  with  loops,  including  turbo  codes.  However,  there  has  been  little  understanding  of  the  algorithm  or  the  nature  of  the  solutions it finds  for  general graphs.  We  show  that  BP  can  only  converge  to  a  stationary  point  of an  approximate free  energy,  known as the Bethe free  energy in statis(cid:173) tical physics.  This result  characterizes BP fixed-points  and makes  connections with variational approaches to approximate inference.  More importantly, our analysis lets us  build  on the progress made  in  statistical  physics  since  Bethe's  approximation was  introduced  in 1935.  Kikuchi and others have shown how to construct more ac(cid:173) curate free energy approximations, of which Bethe's approximation  is  the  simplest.  Exploiting  the  insights  from  our analysis,  we  de(cid:173) rive generalized belief propagation (GBP) versions ofthese Kikuchi  approximations.  These  new  message  passing  algorithms  can  be  significantly more accurate than ordinary BP, at an adjustable in(cid:173) crease in complexity.  We  illustrate such  a  new  GBP algorithm on  a grid Markov network and show that it gives much more accurate  marginal probabilities than those found  using ordinary BP.
Multiple Timescales of Adaptation in a Neural Code
Authors: Geoffrey D. Lewen, William Bialek, Robert R. de Ruyter van Steveninck Adrienne L. Fairhall
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Many neural systems extend their dynamic range by  adaptation.  We ex(cid:173) amine the  timescales  of adaptation  in  the  context of dynamically  mod(cid:173) ulated rapidly-varying stimuli,  and demonstrate in the  fly  visual system  that  adaptation  to  the  statistical  ensemble  of the  stimulus  dynamically  maximizes information transmission about the time-dependent stimulus.  Further, while the rate response has  long transients, the adaptation takes  place on timescales consistent with optimal variance estimation.
Speech Denoising and Dereverberation Using Probabilistic Models
Authors: John C. Platt, Alex Acero, Li Deng Hagai Attias
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: This paper presents a unified probabilistic framework for denoising and  dereverberation of speech signals. The framework transforms the denois(cid:173) ing and dereverberation problems into Bayes-optimal signal estimation.  The  key  idea is  to  use  a  strong  speech  model  that  is  pre-trained on  a  large data set of clean speech.  Computational efficiency is  achieved by  using variational EM, working in  the frequency domain, and employing  conjugate priors.  The framework covers both single and multiple micro(cid:173) phones.  We  apply this  approach to noisy reverberant speech signals and  get results substantially better than standard methods.
Modelling Spatial Recall, Mental Imagery and Neglect
Authors: Neil Burgess Suzanna Becker
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We present a computational model of the neural mechanisms in the pari(cid:173) etal and  temporal lobes that support spatial navigation, recall of scenes  and  imagery  of the  products  of recall.  Long  term  representations  are  stored  in  the  hippocampus,  and  are  associated  with  local  spatial  and  object-related features  in the parahippocampal region.  Viewer-centered  representations are dynamically generated from long term memory in the  parietal part of the  model.  The model thereby  simulates recall  and  im(cid:173) agery  of locations and  objects in  complex environments.  After parietal  damage,  the  model exhibits hemispatial neglect in  mental imagery that  rotates  with  the  imagined perspective of the  observer,  as  in  the  famous  Milan  Square experiment  [1].  Our model  makes  novel  predictions for  the  neural representations  in  the  parahippocampal and  parietal  regions  and for behavior in healthy volunteers and neuropsychological patients.
Adaptive Object Representation with Hierarchically-Distributed Memory Sites
Authors: Bosco S. Tjan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Theories  of  object  recognition  often  assume  that  only  one  representa(cid:173) tion  scheme  is  used  within  one  visual-processing  pathway.  Versatility  of  the  visual  system  comes  from  having  multiple  visual-processing  pathways,  each  specialized  in  a  different  category  of  objects.  We  propose  a  theoretically  simpler  alternative,  capable  of  explaining  the  same set of data and  more.  A  single primary visual-processing pathway,  loosely  modular,  is  assumed.  Memory  modules  are  attached  to  sites  along  this  pathway.  Object-identity  decision  is  made  independently  at  each  site.  A  site's  response  time  is  a  monotonic-decreasing  function  of  its  confidence  regarding  its  decision.  An  observer's  response  is  the  first-arriving  response  from  any  site.  The  effective  representation(s)  of  such  a system,  determined empirically,  can appear to  be  specialized for  different  tasks  and  stimuli,  consistent  with  recent  clinical  and  functional-imaging  findings.  This,  however,  merely  reflects  a  decision  being  made  at  its  appropriate  level  of abstraction.  The  system  itself is  intrinsically flexible and adaptive.
Text Classification using String Kernels
Authors: John Shawe-Taylor, Nello Cristianini, Christopher J. C. H. Watkins Huma Lodhi
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  introduce  a  novel  kernel  for  comparing two  text  documents.  The  kernel  is  an  inner  product  in  the  feature  space  consisting  of  all  subsequences  of  length  k.  A  subsequence  is  any  ordered  se(cid:173) quence  of k characters occurring in the text though not necessarily  contiguously.  The subsequences  are  weighted  by  an exponentially  decaying factor  of their  full  length  in  the  text,  hence  emphasising  those  occurrences  which  are  close  to  contiguous.  A  direct  compu(cid:173) tation of this feature vector  would involve a  prohibitive amount of  computation  even  for  modest  values  of k,  since  the  dimension  of  the feature space grows exponentially with k.  The paper describes  how despite this fact the inner product can be efficiently evaluated  by a dynamic programming technique.  A preliminary experimental  comparison of the performance of the kernel compared with a  stan(cid:173) dard  word  feature  space  kernel  results. 
[6]  is  made showing  encouraging
Fast Training of Support Vector Classifiers
Authors: Pedro Luis Alarcón-Diana, Angel Navia-Vázquez, Antonio Artés-Rodríguez Fernando Pérez-Cruz
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In this communication we present a new algorithm for solving Support  Vector Classifiers (SVC) with large training data sets. The new algorithm  is based on an Iterative Re-Weighted Least Squares procedure which is  used to  optimize the SVc. Moreover,  a novel sample selection strategy  for the working set is presented,  which randomly chooses  the working  set among the training  samples that do  not fulfill  the stopping criteria.  The validity  of both proposals,  the  optimization procedure and sample  selection  strategy,  is  shown  by  means  of computer experiments  using  well-known data sets.
Beyond Maximum Likelihood and Density Estimation: A Sample-Based Criterion for Unsupervised Learning of Complex Models
Authors: Michael Mozer Sepp Hochreiter
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The goal of many unsupervised learning procedures is to bring two  probability  distributions  into  alignment.  Generative  models  such  as  Gaussian mixtures and Boltzmann machines can be cast in this  light,  as  can recoding models  such  as ICA  and projection pursuit.  We propose a novel sample-based error measure for these classes of  models, which applies even in situations where maximum likelihood  (ML)  and  probability  density  estimation-based  formulations  can(cid:173) not be applied,  e.g.,  models that are nonlinear  or have intractable  posteriors.  Furthermore,  our  sample-based  error  measure  avoids  the difficulties of approximating a  density function.  We  prove that  with  an unconstrained  model,  (1)  our approach  converges  on  the  correct solution as the number of samples goes to infinity,  and  (2)  the expected solution of our approach in the generative framework  is  the  ML  solution.  Finally,  we  evaluate our approach via simula(cid:173) tions of linear and nonlinear models  on  mixture of Gaussians and  ICA problems.  The experiments show the broad applicability and  generality of our approach.
A Neural Probabilistic Language Model
Authors: Réjean Ducharme, Pascal Vincent Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A goal  of statistical language modeling is  to  learn  the joint probability  function of sequences of words.  This is intrinsically difficult because of  the curse of dimensionality:  we propose to fight it with its own weapons.  In the proposed approach one learns simultaneously (1) a distributed rep(cid:173) resentation for each word (i.e.  a similarity between words) along with (2)  the probability function for word sequences, expressed with these repre(cid:173) sentations.  Generalization is  obtained because a sequence of words that  has  never been seen before gets  high probability if it is  made of words  that are similar to words forming an already seen sentence.  We report on  experiments using neural networks for the probability function, showing  on  two  text  corpora that  the  proposed approach  very  significantly  im(cid:173) proves on a state-of-the-art trigram model.
Support Vector Novelty Detection Applied to Jet Engine Vibration Spectra
Authors: Bernhard Schölkopf, Lionel Tarassenko, Paul Anuzis Paul M. Hayton
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A system has  been developed to  extract diagnostic information from jet  engine carcass vibration data.  Support Vector Machines applied to nov(cid:173) elty  detection provide a measure  of how  unusual  the  shape  of a  vibra(cid:173) tion signature is, by learning a representation of normality.  We describe  a novel method for  Support Vector Machines  of including information  from a second class for novelty detection and give results from the appli(cid:173) cation to Jet Engine vibration analysis.
`N-Body' Problems in Statistical Learning
Authors: Andrew W. Moore Alexander G. Gray
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present  efficient  algorithms for  all-point-pairs  problems ,  or  'N(cid:173) body '-like problems, which  are  ubiquitous in statistical learning.  We  focus on six examples, including nearest-neighbor classification, kernel  density  estimation,  outlier  detection ,  and  the  two-point  correlation.  These include any problem which abstractly requires  a comparison of  each  of the  N  points  in  a  dataset  with  each  other  point  and  would  naively  be  solved  using  N 2  distance  computations.  In  practice  N  is  often  large  enough  to  make this  infeasible.  We  present  a  suite  of  new  geometric  t echniques  which  are  applicable  in  principle  to  any  'N-body'  computation  including  large-scale  mixtures  of  Gaussians,  RBF neural networks,  and HMM 's.  Our  algorithms exhibit favorable  asymptotic  scaling  and  are  empirically  several  orders  of magnitude  faster  than  the  naive  computation,  even  for  small datasets.  We  are  aware of no exact  algorithms for  these  problems which  are  more effi(cid:173) cient  either  empirically  or  theoretically.  In  addition,  our framework  yields  simple and  elegant  algorithms.  It also  permits two  important  generalizations  beyond  the  standard  all-point-pairs  problems,  which  are  more difficult.  These  are  represented  by  our final  examples,  the  multiple two-point correlation  and the  notorious n-point correlation.
A Silicon Primitive for Competitive Learning
Authors: Miguel Figueroa, Chris Diorio David Hsu
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Competitive  learning  is  a  technique  for  training  classification  and  clustering  networks.  We  have  designed  and  fabricated  an  11- transistor primitive,  that  we  term  an  automaximizing  bump  circuit,  that  implements  competitive  learning  dynamics.  The  circuit  per(cid:173) forms  a  similarity  computation,  affords  nonvolatile  storage,  and  implements  simultaneous  local  adaptation  and  computation.  We  show  that  our  primitive  is  suitable  for  implementing  competitive  learning  in  VLSI,  and  demonstrate  its  effectiveness  in  a  standard  clustering task.
Automatic Choice of Dimensionality for PCA
Authors: Thomas P. Minka
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A central issue  in  principal component analysis  (PCA)  is  choosing the  number of principal components to be retained.  By  interpreting PCA  as  density estimation, we show how to  use Bayesian model selection to es(cid:173) timate the true dimensionality of the data.  The resulting estimate is sim(cid:173) ple to  compute yet guaranteed to pick the correct dimensionality, given  enough data.  The estimate involves an integral over the Steifel manifold  of k-frames, which is difficult to compute exactly.  But after choosing an  appropriate parameterization  and  applying  Laplace's  method,  an  accu(cid:173) rate and practical estimator is obtained. In simulations, it is convincingly  better than cross-validation and  other proposed algorithms,  plus it runs  much faster.
Propagation Algorithms for Variational Bayesian Learning
Authors: Matthew J. Beal Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Variational  approximations  are  becoming  a  widespread  tool  for  Bayesian learning of graphical models.  We  provide  some theoret(cid:173) ical  results for  the variational updates in a  very  general family  of  conjugate-exponential graphical  models.  We  show  how  the  belief  propagation  and  the junction  tree  algorithms  can  be  used  in  the  inference step of variational Bayesian learning.  Applying these re(cid:173) sults to the Bayesian analysis of linear-Gaussian state-space models  we  obtain a  learning procedure that exploits the Kalman smooth(cid:173) ing propagation, while  integrating over  all  model parameters.  We  demonstrate how  this can be used to infer the hidden state dimen(cid:173) sionality of the state-space model in a variety of synthetic problems  and one  real high-dimensional data set.
An Adaptive Metric Machine for Pattern Classification
Authors: Jing Peng, Dimitrios Gunopulos Carlotta Domeniconi
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Nearest neighbor  classification assumes locally constant class con(cid:173) ditional  probabilities.  This  assumption  becomes  invalid  in  high  dimensions with finite  samples due to the curse of dimensionality.  Severe  bias  can be  introduced  under these  conditions  when  using  the  nearest  neighbor  rule.  We  propose  a  locally  adaptive  nearest  neighbor  classification  method to try to  minimize  bias.  We  use  a  Chi-squared distance analysis to compute a flexible  metric for pro(cid:173) ducing neighborhoods that are elongated along less relevant feature  dimensions and constricted along most influential ones.  As a result,  the class conditional probabilities tend to be smoother in the mod(cid:173) ified neighborhoods, whereby better classification performance can  be achieved.  The efficacy of our method is validated and compared  against other techniques using a variety of real world data.
On Iterative Krylov-Dogleg Trust-Region Steps for Solving Neural Networks Nonlinear Least Squares Problems
Authors: James Demmel Eiji Mizutani
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: This  paper  describes  a  method of dogleg  trust-region  steps,  or re(cid:173) stricted  Levenberg-Marquardt  steps,  based  on  a  projection  pro(cid:173) cess  onto the  Krylov subspaces  for  neural  networks  nonlinear least  squares problems.  In particular, the linear conjugate gradient (CG)  method  works  as  the  inner iterative  algorithm for  solving  the  lin(cid:173) earized  Gauss-Newton normal equation, whereas  the  outer nonlin(cid:173) ear algorithm repeatedly  takes so-called  "Krylov-dogleg"  steps, re(cid:173) lying only on matrix-vector multiplication without explicitly form(cid:173) ing the Jacobian matrix or the Gauss-Newton model Hessian.  That  is,  our  iterative  dogleg  algorithm  can  reduce  both  operational  counts  and  memory space  by  a  factor  of O(n)  (the number of pa(cid:173) rameters)  in comparison with a  direct  linear-equation solver.  This  memory-less property is  useful  for  large-scale  problems.
Stability and Noise in Biochemical Switches
Authors: William Bialek
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Many processes in biology, from the regulation of gene expression in  bacteria to memory in the brain, involve switches constructed from  networks of biochemical reactions.  Crucial molecules are present in  small numbers, raising questions about noise and stability.  Analysis  of noise in simple reaction schemes indicates that switches stable for  years  and  switchable  in  milliseconds  can be  built  from  fewer  than  one hundred molecules.  Prospects for direct tests of this prediction,  as well  as implications, are discussed.
Computing with Finite and Infinite Networks
Authors: Ole Winther
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Using  statistical mechanics  results, I calculate learning curves  (average  generalization error) for Gaussian processes  (GPs)  and Bayesian  neural  networks  (NNs) used for regression.  Applying the results to learning a  teacher defined by a two-layer network, I can  directly compare GP and  Bayesian NN learning. I find that a GP in general requires CJ (d S )-training  examples  to  learn  input features  of order  s  (d  is  the  input dimension),  whereas  a  NN  can  learn  the  task  with  order  the  number of adjustable  weights training examples.  Since a GP can be considered as  an  infinite  NN, the results show that even  in the Bayesian approach, it is important  to limit the complexity of the learning machine.  The theoretical findings  are confirmed in simulations with analytical GP learning and a NN mean  field algorithm.
Hierarchical Memory-Based Reinforcement Learning
Authors: Sridhar Mahadevan Natalia Hernandez-Gardiol
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Sridhar  Mahadevan 
Department of Computer Science 
Michigan State University  East Lansing,  MI  48824  mahadeva@cse.msu.edu 
A  key  challenge  for  reinforcement  learning  is  scaling  up  to  large  partially  observable  domains.  In this paper,  we  show  how  a  hier(cid:173) archy of behaviors can be used to create and select among variable  length short-term memories  appropriate for  a task.  At  higher lev(cid:173) els  in  the  hierarchy,  the  agent  abstracts  over  lower-level  details  and  looks  back  over  a  variable  number  of high-level  decisions  in  time.  We  formalize  this  idea  in  a  framework  called  Hierarchical  Suffix Memory (HSM).  HSM  uses a memory-based SMDP learning  method  to  rapidly  propagate  delayed  reward  across  long  decision  sequences.  We  describe  a  detailed  experimental  study  comparing  memory  vs.  hierarchy  using  the  HSM  framework  on  a  realistic  corridor navigation task.
Second Order Approximations for Probability Models
Authors: Wim Wiegerinck Hilbert J. Kappen
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In  this  paper,  we  derive  a  second  order mean  field  theory  for  directed  graphical  probability  models.  By  using  an  information  theoretic  argu(cid:173) ment  it  is  shown  how  this  can  be  done  in  the  absense  of a  partition  function.  This method is  a direct generalisation of the well-known TAP  approximation for  Boltzmann Machines.  In  a numerical  example,  it is  shown  that the  method  greatly  improves  the  first  order mean  field  ap(cid:173) proximation.  For a restricted class of graphical models, so-called single  overlap graphs, the second order method has comparable complexity to  the first order method.  For sigmoid belief networks, the method is shown  to be particularly fast and effective.
Feature Selection for SVMs
Authors: Sayan Mukherjee, Olivier Chapelle, Massimiliano Pontil, Tomaso Poggio, Vladimir Vapnik Jason Weston
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We introduce a method of feature selection for Support Vector Machines.  The method is based upon finding those features which minimize bounds  on  the leave-one-out error.  This  search can be efficiently performed via  gradient descent.  The resulting  algorithms  are  shown to  be  superior to  some standard feature selection algorithms on both toy data and real-life  problems of face  recognition, pedestrian detection  and  analyzing DNA  micro array data.
Direct Classification with Indirect Data
Authors: Timothy X. Brown
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We classify an input space according to the outputs of a real-valued  function.  The  function  is  not  given,  but  rather  examples  of the  function.  We  contribute a  consistent classifier that  avoids the un(cid:173) necessary complexity of estimating the function.
Constrained Independent Component Analysis
Authors: Jagath C. Rajapakse Wei Lu
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The  paper  presents  a  novel  technique  of constrained  independent  component analysis  (CICA)  to introduce constraints into the clas(cid:173) sical ICA and solve the constrained optimization problem by using  Lagrange  multiplier  methods.  This  paper  shows  that  CICA  can  be used to order the resulted independent components in a specific  manner and normalize the demixing matrix in the signal separation  procedure.  It can systematically eliminate the ICA's indeterminacy  on permutation and dilation.  The experiments demonstrate the use  of CICA  in  ordering of  independent  components  while  providing  normalized demixing processes.  Keywords:  Independent  component  analysis, constrained indepen(cid:173) dent component analysis, constrained optimization, Lagrange mul(cid:173) tiplier methods
The Early Word Catches the Weights
Authors: Garrison W. Cottrell, Karen L. Anderson Mark A. Smith
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: The strong correlation between the frequency of words and their naming  latency has been well documented.  However, as  early as  1973, the  Age  of Acquisition (AoA) of a word was  alleged to be the actual  variable of  interest, but these studies  seem to  have been ignored in  most of the lit(cid:173) erature.  Recently, there has been a resurgence of interest in AoA. While  some studies have shown that frequency has no effect when AoA is con(cid:173) trolled for,  more recent studies have found independent contributions of  frequency and AoA. Connectionist models have repeatedly shown strong  effects  of frequency,  but  little  attention  has  been  paid  to  whether they  can also  show  AoA effects.  Indeed,  several researchers have explicitly  claimed  that they  cannot  show  AoA  effects.  In  this  work,  we  explore  these claims using a simple feed forward neural network.  We find  a sig(cid:173) nificant  contribution  of AoA  to  naming  latency,  as  well  as  conditions  under which frequency provides an independent contribution.
The Manhattan World Assumption: Regularities in Scene Statistics which Enable Bayesian Inference
Authors: Alan L. Yuille James M. Coughlan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Preliminary work by the authors made use of the so-called  "Man(cid:173) hattan  world"  assumption  about  the  scene  statistics  of  city  and  indoor scenes.  This assumption stated that such  scenes were built  on a  cartesian grid which led to regularities in the image edge gra(cid:173) dient  statistics.  In this paper we  explore the general applicability  of this  assumption  and show  that,  surprisingly,  it holds  in a  large  variety of less structured environments including rural scenes.  This  enables us, from  a single image, to determine the orientation of the  viewer relative to the scene structure and also to detect target ob(cid:173) jects  which  are  not  aligned  with  the  grid.  These  inferences  are  performed  using  a  Bayesian  model  with  probability  distributions  (e.g.  on the image gradient statistics)  learnt from  real data.
Processing of Time Series by Neural Circuits with Biologically Realistic Synaptic Dynamics
Authors: Wolfgang Maass, Eduardo D. Sontag, Anthony M. Zador Thomas Natschläger
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: Experimental data show that biological synapses behave quite differently  from the symbolic synapses in common artificial neural network models.  Biological synapses are dynamic, i.e., their "weight" changes on  a short  time  scale  by  several  hundred percent in  dependence of the  past  input  to  the  synapse.  In  this  article  we  explore  the  consequences  that  these  synaptic  dynamics  entail  for  the  computational  power  of feedforward  neural  networks. We  show that gradient descent suffices to  approximate  a given  (quadratic) filter  by  a rather small  neural  system  with  dynamic  synapses.  We  also  compare our network model to  artificial  neural net(cid:173) works  designed  for  time  series  processing.  Our  numerical  results  are  complemented by  theoretical analysis which  show  that even with just a  single hidden layer such networks can approximate a surprisingly large  large  class  of nonlinear filters:  all  filters  that  can  be  characterized  by  Volterra series. This result is robust with regard to various changes in the  model for synaptic dynamics.
Machine Learning for Video-Based Rendering
Authors: Irfan A. Essa Arno Schödl
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We  present  techniques  for  rendering  and  animation  of  realistic  scenes  by  analyzing  and  training  on  short  video  sequences.  This  work extends the new paradigm for computer animation, video  tex(cid:173) tures,  which  uses  recorded video  to generate  novel  animations  by  replaying the  video  samples  in  a  new  order.  Here  we  concentrate  on  video  sprites,  which  are  a  special  type  of  video  texture.  In  video  sprites,  instead of storing whole  images,  the  object  of inter(cid:173) est  is  separated  from  the  background  and  the  video  samples  are  stored as  a sequence of alpha-matted sprites with associated veloc(cid:173) ity information.  They can be  rendered anywhere  on the screen to  create a novel animation of the object.  We present methods to cre(cid:173) ate  such  animations  by  finding  a  sequence  of sprite  samples  that  is  both  visually  smooth  and  follows  a  desired  path.  To  estimate  visual  smoothness,  we  train  a  linear  classifier  to  estimate  visual  similarity  between video  samples.  If the  motion  path is  known in  advance,  we  use  beam search to find  a  good sample sequence.  We  can specify the motion interactively by precomputing the sequence  cost  function  using Q-Iearning.
Discovering Hidden Variables: A Structure-Based Approach
Authors: Noam Lotner, Nir Friedman, Daphne Koller Gal Elidan
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: A serious problem in learning probabilistic models is the presence of hid(cid:173) den variables. These variables are not observed, yet interact with several  of the observed variables.  As  such,  they induce seemingly complex de(cid:173) pendencies  among  the  latter.  In  recent years,  much  attention  has  been  devoted to  the  development of algorithms for  learning parameters,  and  in  some cases  structure, in  the presence of hidden variables.  In this  pa(cid:173) per,  we  address  the  related  problem of detecting  hidden  variables  that  interact with the observed variables.  This problem is of interest both for  improving our understanding of the domain and as a preliminary step that  guides the learning procedure towards promising models.  A very natural  approach is  to  search for "structural  signatures" of hidden variables - substructures in the learned network that tend to  suggest the presence of  a hidden variable.  We  make this  basic  idea concrete, and  show  how  to  integrate it with structure-search algorithms.  We evaluate this method on  several synthetic and real-life datasets, and show that it performs surpris(cid:173) ingly well.
Regularized Winnow Methods
Authors: Tong Zhang
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In theory, the Winnow multiplicative update has certain advantages over  the Perceptron additive update when there are many irrelevant attributes.  Recently,  there has  been much effort on enhancing the Perceptron algo(cid:173) rithm by  using regularization,  leading to  a class  of linear classification  methods called support vector machines.  Similarly, it is also possible to  apply the regularization idea to the Winnow algorithm, which gives meth(cid:173) ods  we  call regularized Winnows.  We  show  that the resulting methods  compare with the basic Winnows in  a similar way  that a support vector  machine  compares  with  the  Perceptron.  We  investigate  algorithmic is(cid:173) sues and learning properties of the derived methods.  Some experimental  results will also be provided to illustrate different methods.
FaceSync: A Linear Operator for Measuring Synchronization of Video Facial Images and Audio Tracks
Authors: Michele Covell Malcolm Slaney
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: FaceSync  is  an  optimal  linear  algorithm  that  finds  the  degree  of syn(cid:173) chronization  between  the  audio  and  image  recordings  of  a  human  speaker.  Using canonical correlation,  it finds  the best direction  to com(cid:173) bine  all  the  audio  and  image data,  projecting  them  onto  a  single  axis.  FaceSync uses  Pearson's correlation to  measure the degree of synchro(cid:173) nization between the audio and image data. We derive the optimal linear  transform to combine the audio and visual information and describe an  implementation that avoids  the  numerical problems caused by comput(cid:173) ing the correlation matrices. 
1  Motivation  In many applications, we want to know about the synchronization between an audio signal  and  the corresponding image data.  In a teleconferencing system,  we might want to know  which of the several people imaged by a camera is  heard by the microphones; then, we can  direct the camera to the speaker. In post-production for a film,  clean audio dialog is  often  dubbed over the  video;  we  want to  adjust the  audio  signal  so  that the lip-sync is  perfect.  When analyzing a film,  we want to know when the person talking is in the shot, instead of  off camera. When evaluating the quality of dubbed films,  we can measure of how  well  the  translated words and audio fit  the actor's face. 
This paper describes an algorithm, FaceSync, that measures the degree of synchronization  between the video image of a face and the associated audio signal. We can do this task by  synthesizing the talking face,  using techniques  such  as Video  Rewrite  [1], and then com(cid:173) paring the synthesized video with the test video. That process, however, is expensive. Our  solution finds  a linear operator that, when applied to the audio and video signals, generates  an  audio-video-synchronization-error  signal.  The  linear  operator  gathers  information  from throughout the image and thus allows us to do the computation inexpensively.  Hershey and Movellan  [2]  describe an approach based on measuring the mutual informa(cid:173) tion between the audio signal and individual pixels in the  video. The correlation between  the audio signal, x, and one pixel in the image y,  is  given by Pearson's correlation,  r.  The  mutual information between these two  variables is  given by f(x,y)  = -1/2  log(l-?). They  create movies that show the regions of the video that have high correlation with the audio; 

Currently at IBM Almaden Research, 650 Harry Road, San Jose, CA 95120.  2.  Currently at Yes Video. com, 2192 Fortune Drive, San Jose, CA 95131.
Mixtures of Gaussian Processes
Authors: Volker Tresp
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: We introduce the mixture of Gaussian processes (MGP) model which is  useful for applications in which the optimal bandwidth of a map is input  dependent.  The MGP is  derived from the mixture of experts model and  can also be used for modeling general conditional probability densities.  We discuss how  Gaussian processes -in particular in form of Gaussian  process classification, the support vector machine and the MGP model(cid:173) can be used for quantifying the dependencies in graphical models.
What Can a Single Neuron Compute?
Authors: Adrienne L. Fairhall, William Bialek Blaise Agüera y Arcas
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: In this  paper  we  formulate  a  description  of the  computation  per(cid:173) formed  by  a  neuron  as  a  combination  of  dimensional  reduction  and nonlinearity.  We  implement this description for  the Hodgkin(cid:173) Huxley model,  identify the most  relevant  dimensions  and find  the  nonlinearity.  A  two  dimensional  description  already  captures  a  significant  fraction  of the information that  spikes  carry about  dy(cid:173) namic inputs.  This description  also shows that computation in the  Hodgkin-Huxley  model  is  more  complex  than  a  simple  integrate(cid:173) and-fire or perceptron model.