mtybilly/Instruct2DS huggingface dataset:
Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks
Authors: Yoshua Bengio, Samy Bengio
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The curse of dimensionality is  severe when modeling high-dimensional  discrete data:  the number of possible combinations of the variables ex(cid:173) plodes exponentially.  In  this  paper we propose a  new  architecture for  modeling high-dimensional data that requires resources (parameters and  computations) that grow only at most as the square of the number of vari(cid:173) ables,  using a multi-layer neural  network to represent the joint distribu(cid:173) tion of the variables as the product of conditional distributions. The neu(cid:173) ral  network can be interpreted as  a graphical model  without hidden ran(cid:173) dom variables, but in which the conditional distributions are tied through  the hidden units. The connectivity of the neural network can be pruned by  using dependency tests between the variables. Experiments on modeling  the distribution of several discrete data sets show statistically significant  improvements over other methods such as  naive Bayes and comparable  Bayesian networks,  and show  that significant improvements can be ob(cid:173) tained by pruning the network.
A Generative Model for Attractor Dynamics
Authors: Richard S. Zemel, Michael Mozer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Attractor networks, which map an input space to a discrete out(cid:173) put space, are useful for pattern completion. However, designing  a net to have a given set of attractors is notoriously tricky; training  procedures are CPU intensive and often produce spurious afuac(cid:173) tors and ill-conditioned attractor basins.  These difficulties  occur  because each connection in the network participates in the encod(cid:173) ing of multiple attractors.  We describe an alternative formulation  of attractor networks in which the encoding of knowledge is local,  not distributed. Although localist attractor networks have similar  dynamics to their distributed counterparts, they are much easier  to work with and interpret. We propose a statistical formulation of  localist attract or net dynamics, which yields a convergence proof  and a mathematical interpretation of model parameters. 
Attractor networks map an input space,  usually continuous,  to a  sparse output  space composed of a discrete set of alternatives.  Attractor networks have a long  history in neural network research.  Attractor networks are often used for pattern completion, which involves filling  in  missing,  noisy,  or incorrect features  in an input pattern.  The  initial state of the  attractor net is  typically determined by the input pattern.  Over time, the state is  drawn to  one of a  predefined set of states-the attractors.  Attractor net dynam(cid:173) ics can be described by a state trajectory (Figure 1a).  An attractor net is generally  implemented by a set of visible units whose activity represents the instantaneous  state, and optionally, a set of hidden units that assist in the computation. Attractor  dynamics arise from interactions among the units.  In most formulations of afuac(cid:173) tor nets,2,3  the dynamics can be characterized by gradient descent in an energy  landscape, allowing one to partition the output space into attractor basins.  Instead  of homogeneous attractor basins, it is often desirable to sculpt basins that depend  on the recent history of the network and the arrangement of attractors in the space.  In psychological models of human cognition, for example, priming is fundamental:  after the model visits an attractor, it should be faster to fall into the same attractor  in the near future, i.e., the attractor basin should be broadened. 1 ,6  Another property of attractor nets is key to explaining behavioral data in psycho(cid:173) logical  and neurobiological  models:  the gang  effect,  in  which  the strength  of an  attractor is influenced by other attractors in its neighborhood. Figure 1b illustrates  the gang effect:  the proximity of the two rightmost afuactors creates a deeper at(cid:173) tractor basin, so that if the input starts at the origin it will get pulled to the right. 
A  Generative Model for Attractor Dynamics 
81
An Environment Model for Nonstationary Reinforcement Learning
Authors: Samuel P. M. Choi, Dit-Yan Yeung, Nevin Lianwen Zhang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Reinforcement learning in nonstationary environments is  generally  regarded  as  an  important  and  yet  difficult  problem.  This  paper  partially addresses the problem by formalizing a subclass of nonsta(cid:173) tionary environments.  The environment model, called hidden-mode  Markov  decision  process  (HM-MDP),  assumes that environmental  changes  are  always  confined  to  a  small  number  of hidden  modes.  A  mode  basically  indexes  a  Markov  decision  process  (MDP)  and  evolves  with  time according to a  Markov  chain.  While  HM-MDP  is  a  special  case  of partially  observable  Markov  decision  processes  (POMDP), modeling an HM-MDP environment via the more gen(cid:173) eral POMDP  model  unnecessarily increases the problem complex(cid:173) ity.  A variant of the Baum-Welch algorithm is  developed for model  learning requiring less  data and time.
Predictive App roaches for Choosing Hyperparameters in Gaussian Processes
Authors: S. Sundararajan, S. Sathiya Keerthi
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Gaussian  Processes  are  powerful  regression  models  specified  by  parametrized mean and covariance functions.  Standard approaches  to  estimate  these  parameters  (known  by  the  name  Hyperparam(cid:173) eters)  are  Maximum  Likelihood  (ML)  and  Maximum  APosterior  (MAP)  approaches.  In this paper, we  propose and investigate pre(cid:173) dictive  approaches,  namely,  maximization  of  Geisser's  Surrogate  Predictive Probability (GPP) and minimization of mean square er(cid:173) ror with  respect to GPP  (referred to as  Geisser's  Predictive mean  square  Error  (GPE))  to  estimate  the  hyperparameters.  We  also  derive  results  for  the  standard  Cross-Validation  (CV)  error  and  make  a  comparison.  These approaches are tested on  a  number of  problems and experimental results show that these approaches are  strongly competitive to existing approaches.
Acquisition in Autoshaping
Authors: Sham Kakade, Peter Dayan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Quantitative data on the speed with which animals acquire behav(cid:173) ioral responses during classical conditioning experiments should  provide strong constraints on models of learning.  However, most  models have simply ignored these data; the few that have attempt(cid:173) ed to address them have failed by at least an order of magnitude.  We discuss key data on the speed of acquisition, and show how to  account for them using a statistically sound model of learning, in  which differential reliabilities of stimuli playa crucial role.
Transductive Inference for Estimating Values of Functions
Authors: Olivier Chapelle, Vladimir Vapnik, Jason Weston
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce an algorithm for  estimating the values of a  function  at a set of test points  Xe+!, ... , xl+m  given a set of training points  (XI,YI), ... ,(xe,Ye)  without  estimating  (as  an  intermediate  step)  the regression function .  We demonstrate that this direct (transduc(cid:173) ti ve)  way  for  estimating  values  of the  regression  (or  classification  in  pattern  recognition)  can  be  more  accurate  than  the  tradition(cid:173) alone  based  on  two  steps,  first  estimating  the  function  and  then  calculating the values of this function  at the  points of interest.
Effective Learning Requires Neuronal Remodeling of Hebbian Synapses
Authors: Gal Chechik, Isaac Meilijson, Eytan Ruppin
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper revisits the classical neuroscience paradigm of Hebbian  learning.  We  find  that  a  necessary  requirement  for  effective  as(cid:173) sociative  memory  learning  is  that  the  efficacies  of  the  incoming  synapses  should  be  uncorrelated.  This  requirement  is  difficult  to  achieve in  a  robust  manner by  Hebbian synaptic learning,  since it  depends on network level information.  Effective learning can yet be  obtained by a neuronal process that maintains a zero sum of the in(cid:173) coming synaptic efficacies.  This normalization drastically improves  the  memory  capacity of associative networks,  from  an  essentially  bounded capacity to one that linearly scales with the network's size.  It also enables the effective storage of patterns with heterogeneous  coding levels in a single network.  Such neuronal normalization can  be successfully carried out by activity-dependent homeostasis of the  neuron's synaptic efficacies, which was recently observed in cortical  tissue.  Thus,  our findings  strongly suggest  that  effective  associa(cid:173) tive learning with Hebbian synapses alone is  biologically implausi(cid:173) ble and that Hebbian synapses must be continuously remodeled by  neuronally-driven regulatory processes in  the brain.
Dual Estimation and the Unscented Transformation
Authors: Eric A. Wan, Rudolph van der Merwe, Alex T. Nelson
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Dual estimation  refers  to  the problem of simultaneously  estimating the  state of a dynamic system and the model which gives rise  to the dynam(cid:173) ics.  Algorithms  include  expectation-maximization (EM), dual  Kalman  filtering,  and joint Kalman methods.  These methods have recently  been  explored in  the context of nonlinear modeling,  where a  neural network  is  used  as  the functional form of the  unknown model.  Typically, an  ex(cid:173) tended  Kalman  filter  (EKF)  or smoother  is  used  for  the  part of the  al(cid:173) gorithm that estimates the clean state given the current estimated model.  An  EKF may  also be used to estimate the weights of the  network.  This  paper points  out the flaws  in  using  the  EKF,  and  proposes an  improve(cid:173) ment based on a new approach called the unscented transformation (UT)  [3].  A  substantial performance gain is  achieved  with  the same order of  computational complexity as  that of the standard EKF.  The approach is  illustrated on several dual estimation methods.
Recognizing Evoked Potentials in a Virtual Environment
Authors: Jessica D. Bayliss, Dana H. Ballard
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Virtual  reality  (VR)  provides  immersive  and  controllable  experimen(cid:173) tal  environments.  It expands  the  bounds  of possible  evoked  potential  (EP)  experiments by  providing complex,  dynamic  environments in  or(cid:173) der  to  study  cognition  without  sacrificing  environmental  control.  VR  also serves as a safe dynamic testbed for brain-computer .interface (BCl)  research.  However, there has been some concern about detecting EP sig(cid:173) nals in a complex VR environment.  This paper shows that EPs exist at  red,  green, and yellow stop lights in  a virtual driving environment.  Ex(cid:173) perimental results show  the existence of the  P3  EP at  "go" and  "stop"  lights and  the contingent negative variation (CNY)  EP  at "slow down"  lights.  In order to  test the  feasibility  of on-line recognition in  VR,  we  looked at recognizing the P3 EP at red stop tights and the absence of this  signal at yellow slow down lights.  Recognition results show that the P3  may successfully be used to control the brakes of a VR car at stop lights.
Means, Correlations and Bounds
Authors: Martijn Leisink, Bert Kappen
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: The  partition function  for  a  Boltzmann  machine  can  be  bounded  from  above  and  below.  We  can  use  this  to  bound  the  means  and  the  correlations.  For  networks  with  small  weights,  the  values  of  these statistics can be restricted to non-trivial regions  (i.e.  a subset  of  [-1 , 1]).  Experimental  results  show  that  reasonable  bounding  occurs  for  weight sizes  where  mean field  expansions  generally  give  good  results.
Fragment Completion in Humans and Machines
Authors: David Jacobs, Bas Rokers, Archisman Rudra, Zili Liu
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Partial information can trigger a complete memory. At the same time, human memory is not perfect. A cue can contain enough information to specify an item in memory, but fail to trigger that item. In the context of word memory, we present experiments that demonstrate some basic patterns in human memory errors. We use cues that consist of word frag- ments. We show that short and long cues are completed more accurately than medium length ones and study some of the factors that lead to this behavior. We then present a novel computational model that shows some of the ﬂexibility and patterns of errors that occur in human memory. This model iterates between bottom-up and top-down computations. These are tied together using a Markov model of words that allows memory to be accessed with a simple feature set, and enables a bottom-up process to compute a probability distribution of possible completions of word frag- ments, in a manner similar to models of visual perceptual completion.
Eye movements and the maturation of cortical orientation selectivity
Authors: Antonino Casile, Michele Rucci
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Neural activity appears to be a crucial component for shaping the recep- tive ﬁelds of cortical simple cells into adjacent, oriented subregions alter- nately receiving ON- and OFF-center excitatory geniculate inputs. It is known that the orientation selective responses of V1 neurons are reﬁned by visual experience. After eye opening, the spatiotemporal structure of neural activity in the early stages of the visual pathway depends both on the visual environment and on how the environment is scanned. We have used computational modeling to investigate how eye movements might affect the reﬁnement of the orientation tuning of simple cells in the pres- ence of a Hebbian scheme of synaptic plasticity. Levels of correlation be- tween the activity of simulated cells were examined while natural scenes were scanned so as to model sequences of saccades and ﬁxational eye movements, such as microsaccades, tremor and ocular drift. The speciﬁc patterns of activity required for a quantitatively accurate development of simple cell receptive ﬁelds with segregated ON and OFF subregions were observed during ﬁxational eye movements, but not in the presence of saccades or with static presentation of natural visual input. These re- sults suggest an important role for the eye movements occurring during visual ﬁxation in the reﬁnement of orientation selectivity.
Orientation-Selective aVLSI Spiking Neurons
Authors: Shih-Chii Liu, Jörg Kramer, Giacomo Indiveri, Tobi Delbrück, Rodney J. Douglas
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We describe a programmable multi-chip VLSI neuronal system that can be used for exploring spike-based information processing models. The system consists of a silicon retina, a PIC microcontroller, and a transceiver chip whose integrate-and-ﬁre neurons are connected in a soft winner-take-all architecture. The circuit on this multi-neuron chip ap- proximates a cortical microcircuit. The neurons can be conﬁgured for different computational properties by the virtual connections of a se- lected set of pixels on the silicon retina. The virtual wiring between the different chips is effected by an event-driven communication pro- tocol that uses asynchronous digital pulses, similar to spikes in a neu- ronal system. We used the multi-chip spike-based system to synthe- size orientation-tuned neurons using both a feedforward model and a feedback model. The performance of our analog hardware spiking model matched the experimental observations and digital simulations of continuous-valued neurons. The multi-chip VLSI system has advantages over computer neuronal models in that it is real-time, and the computa- tional time does not scale with the size of the neuronal network.
On the Generalization Ability of On-Line Learning Algorithms
Authors: Nicolò Cesa-bianchi, Alex Conconi, Claudio Gentile
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: In this paper we show that on-line algorithms for classiﬁcation and re- gression can be naturally used to obtain hypotheses with good data- dependent tail bounds on their risk. Our results are proven without re- quiring complicated concentration-of-measure arguments and they hold for arbitrary on-line learning algorithms. Furthermore, when applied to concrete on-line algorithms, our results yield tail bounds that in many cases are comparable or better than the best known bounds.
Analysis of Sparse Bayesian Learning
Authors: Anita C. Faul, Michael E. Tipping
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: The recent introduction of the 'relevance vector machine' has effec(cid:173) tively  demonstrated  how  sparsity  may  be obtained  in  generalised  linear  models  within  a  Bayesian  framework.  Using  a  particular  form  of Gaussian  parameter prior,  'learning'  is  the  maximisation,  with respect  to hyperparameters, of the  marginal  likelihood  of the  data.  This  paper  studies  the  properties  of  that  objective  func(cid:173) tion,  and  demonstrates  that  conditioned  on  an  individual  hyper(cid:173) parameter,  the  marginal  likelihood  has  a  unique  maximum which  is  computable in closed form.  It is  further  shown that if a  derived  'sparsity criterion'  is  satisfied,  this maximum is  exactly  equivalent  to  'pruning' the corresponding parameter from  the model.
A Hierarchical Model of Complex Cells in Visual Cortex for the Binocular Perception of Motion-in-Depth
Authors: Silvio P. Sabatini, Fabio Solari, Giulia Andreani, Chiara Bartolozzi, Giacomo M. Bisio
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: A cortical model for  motion-in-depth selectivity of complex cells in  the  visual  cortex  is  proposed.  The  model  is  based  on  a  time  ex(cid:173) tension of the phase-based techniques for  disparity estimation.  We  consider  the  computation  of the  total  temporal  derivative  of the  time-varying disparity through the combination of the responses of  disparity energy units.  To take into account the physiological plau(cid:173) sibility, the model  is  based on the  combinations of binocular cells  characterized by different  ocular dominance indices.  The resulting  cortical units  of the model show  a  sharp selectivity for  motion-in(cid:173) depth that has been compared with that reported in the literature  for  real cortical cells.
Categorization by Learning and Combining Object Parts
Authors: Bernd Heisele, Thomas Serre, Massimiliano Pontil, Thomas Vetter, Tomaso Poggio
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We describe an algorithm for automatically learning discriminative com- ponents of objects with SVM classiﬁers. It is based on growing image parts by minimizing theoretical bounds on the error probability of an SVM. Component-based face classiﬁers are then combined in a second stage to yield a hierarchical SVM classiﬁer. Experimental results in face classiﬁcation show considerable robustness against rotations in depth and suggest performance at signiﬁcantly better level than other face detection systems. Novel aspects of our approach are: a) an algorithm to learn component-based classiﬁcation experts and their combination, b) the use of 3-D morphable models for training, and c) a maximum operation on the output of each component classiﬁer which may be relevant for bio- logical models of visual recognition.
Group Redundancy Measures Reveal Redundancy Reduction in the Auditory Pathway
Authors: Gal Chechik, Amir Globerson, M. J. Anderson, E. D. Young, Israel Nelken, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: The  way  groups  of auditory  neurons  interact  to  code  acoustic  in(cid:173) formation is  investigated using an information theoretic approach.  We develop measures of redundancy among groups of neurons, and  apply  them  to  the  study  of collaborative  coding  efficiency  in  two  processing stations in the auditory pathway:  the inferior colliculus  (IC)  and the primary auditory cortex (AI).  Under two schemes for  the  coding  of the  acoustic  content,  acoustic  segments  coding  and  stimulus  identity  coding,  we  show  differences  both in  information  content and group redundancies between IC and AI neurons.  These  results  provide for  the first  time  a  direct  evidence for  redundancy  reduction  along  the  ascending  auditory  pathway,  as  has  been  hy(cid:173) pothesized for  theoretical considerations  [Barlow  1959,2001].  The  redundancy effects under the single-spikes coding scheme are signif(cid:173) icant only for  groups larger than ten cells, and cannot be revealed  with  the  redundancy  measures  that  use  only  pairs  of cells.  The  results  suggest  that the  auditory system transforms low  level  rep(cid:173) resentations that contain redundancies due to the statistical struc(cid:173) ture of natural stimuli, into a  representation in which cortical neu(cid:173) rons extract rare and independent component of complex acoustic  signals,  that are useful  for  auditory scene analysis.
Probabilistic Inference of Hand Motion from Neural Activity in Motor Cortex
Authors: Yun Gao, Michael J. Black, Elie Bienenstock, Shy Shoham, John P. Donoghue
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Statistical learning and probabilistic inference techniques are used to in- fer the hand position of a subject from multi-electrode recordings of neu- ral activity in motor cortex. First, an array of electrodes provides train- ing data of neural ﬁring conditioned on hand kinematics. We learn a non- parametric representation of this ﬁring activity using a Bayesian model and rigorously compare it with previous models using cross-validation. Second, we infer a posterior probability distribution over hand motion conditioned on a sequence of neural test data using Bayesian inference. The learned ﬁring models of multiple cells are used to deﬁne a non- Gaussian likelihood term which is combined with a prior probability for the kinematics. A particle ﬁltering method is used to represent, update, and propagate the posterior distribution over time. The approach is com- pared with traditional linear ﬁltering methods; the results suggest that it may be appropriate for neural prosthetic applications.
The Noisy Euclidean Traveling Salesman Problem and Learning
Authors: Mikio L. Braun, Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We  consider  noisy  Euclidean  traveling  salesman  problems  in  the  plane,  which  are  random combinatorial problems  with  underlying  structure. Gibbs sampling is  used to compute average trajectories,  which  estimate  the  underlying structure common  to all  instances.  This procedure requires identifying the exact relationship between  permutations and tours.  In  a  learning setting,  the  average trajec(cid:173) tory  is  used  as  a  model  to  construct  solutions  to  new  instances  sampled from the same source. Experimental results show that the  average trajectory can in fact estimate the underlying structure and  that overfitting effects occur if the trajectory adapts too closely to  a  single instance.
Incorporating Invariances in Non-Linear Support Vector Machines
Authors: Olivier Chapelle, Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: The  choice  of an  SVM  kernel  corresponds  to  the  choice  of a  rep(cid:173) resentation  of  the  data  in  a  feature  space  and,  to  improve  per(cid:173) formance,  it  should  therefore incorporate prior knowledge  such  as  known  transformation invariances.  We  propose  a  technique which  extends earlier work and aims at incorporating invariances in  non(cid:173) linear  kernels.  We  show  on  a  digit  recognition  task that the  pro(cid:173) posed  approach is  superior to the Virtual Support Vector method,  which  previously had been the method of choice.
Sampling Techniques for Kernel Methods
Authors: Dimitris Achlioptas, Frank Mcsherry, Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We propose randomized techniques for speeding up Kernel Principal Component Analysis on three levels: sampling and quantization of the Gram matrix in training, randomized rounding in evaluating the kernel expansions, and random projections in evaluating the kernel itself. In all three cases, we give sharp bounds on the accuracy of the obtained ap- proximations. Rather intriguingly, all three techniques can be viewed as instantiations of the following idea: replace the kernel function  by a “randomized kernel” which behaves like
Reinforcement Learning and Time Perception -- a Model of Animal Experiments
Authors: Jonathan L. Shapiro, J. Wearden
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Animal  data on delayed-reward conditioning experiments  shows  a  striking property - the  data for  different  time intervals  collapses  into  a  single  curve  when  the  data is  scaled  by  the  time  interval.  This is  called the scalar property of interval timing.  Here a  simple  model of a  neural  clock is  presented and shown to give  rise  to the  scalar  property.  The model  is  an accumulator consisting of noisy,  linear  spiking  neurons.  It  is  analytically  tractable  and  contains  only three parameters.  When coupled with reinforcement learning  it simulates peak procedure experiments, producing both the scalar  property and the pattern of single  trial covariances.
Hyperbolic Self-Organizing Maps for Semantic Navigation
Authors: Jorg Ontrup, Helge Ritter
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We introduce a new type of Self-Organizing Map (SOM) to navigate in the Semantic Space of large text collections. We propose a “hyper- bolic SOM” (HSOM) based on a regular tesselation of the hyperbolic plane, which is a non-euclidean space characterized by constant negative gaussian curvature. The exponentially increasing size of a neighborhood around a point in hyperbolic space provides more freedom to map the complex information space arising from language into spatial relations. We describe experiments, showing that the HSOM can successfully be applied to text categorization tasks and yields results comparable to other state-of-the-art methods.
Probabilistic Abstraction Hierarchies
Authors: Eran Segal, Daphne Koller, Dirk Ormoneit
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Many domains are naturally organized in an abstraction hierarchy or taxonomy, where the instances in “nearby” classes in the taxonomy are similar. In this pa- per, we provide a general probabilistic framework for clustering data into a set of classes organized as a taxonomy, where each class is associated with a prob- abilistic model from which the data was generated. The clustering algorithm simultaneously optimizes three things: the assignment of data instances to clus- ters, the models associated with the clusters, and the structure of the abstraction hierarchy. A unique feature of our approach is that it utilizes global optimization algorithms for both of the last two steps, reducing the sensitivity to noise and the propensity to local maxima that are characteristic of algorithms such as hierarchi- cal agglomerative clustering that only take local steps. We provide a theoretical analysis for our algorithm, showing that it converges to a local maximum of the joint likelihood of model and data. We present experimental results on synthetic data, and on real data in the domains of gene expression and text.
Fast and Robust Classification using Asymmetric AdaBoost and a Detector Cascade
Authors: Paul Viola, Michael Jones
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: This paper develops a new approach for extremely fast detection in do- mains where the distribution of positive and negative examples is highly skewed (e.g. face detection or database retrieval). In such domains a cascade of simple classiﬁers each trained to achieve high detection rates and modest false positive rates can yield a ﬁnal detector with many desir- able features: including high detection rates, very low false positive rates, and fast performance. Achieving extremely high detection rates, rather than low error, is not a task typically addressed by machine learning al- gorithms. We propose a new variant of AdaBoost as a mechanism for training the simple classiﬁers used in the cascade. Experimental results in the domain of face detection show the training algorithm yields sig- niﬁcant improvements in performance over conventional AdaBoost. The ﬁnal face detection system can process 15 frames per second, achieves over 90% detection, and a false positive rate of 1 in a 1,000,000.
Agglomerative Multivariate Information Bottleneck
Authors: Noam Slonim, Nir Friedman, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: The information bottleneck method  is  an  unsupervised  model  independent data  organization  technique.  Given  a joint distribution  peA, B),  this  method  con(cid:173) structs a new variable T  that extracts partitions, or clusters, over the values of A  that are informative about B.  In  a recent paper,  we introduced a general princi(cid:173) pled framework for multivariate extensions of the information bottleneck method  that allows us to consider multiple systems of data partitions that are inter-related.  In this  paper,  we  present  a  new  family  of simple  agglomerative  algorithms  to  construct such systems of inter-related clusters.  We analyze the behavior of these  algorithms and apply them to several real-life datasets.
K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms
Authors: Pascal Vincent, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Guided by an initial idea of building a complex (non linear) decision surface with maximal local margin in input space, we give a possible geometrical intuition as to why K-Nearest Neighbor (KNN) algorithms often perform more poorly than SVMs on classiﬁcation tasks. We then propose modiﬁed K-Nearest Neighbor algorithms to overcome the per- ceived problem. The approach is similar in spirit to Tangent Distance, but with invariances inferred from the local neighborhood rather than prior knowledge. Experimental results on real world classiﬁcation tasks sug- gest that the modiﬁed KNN algorithms often give a dramatic improve- ment over standard KNN and perform as well or better than SVMs.
Using Vocabulary Knowledge in Bayesian Multinomial Estimation
Authors: Thomas L. Griffiths, Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Estimating the parameters of sparse multinomial distributions is an important component of many statistical learning tasks. Recent approaches have used uncertainty over the vocabulary of symbols in a multinomial distribution as a means of accounting for sparsity. We present a Bayesian approach that allows weak prior knowledge, in the form of a small set of approximate candidate vocabularies, to be used to dramatically improve the resulting estimates. We demonstrate these improvements in applications to text compres(cid:173) sion and estimating distributions over words in newsgroup data.
Bayesian time series classification
Authors: Peter Sykacek, Stephen J. Roberts
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: This paper proposes an approach to classiﬁcation of adjacent segments of a time series as being either of  classes. We use a hierarchical model that consists of a feature extraction stage and a generative classiﬁer which is built on top of these features. Such two stage approaches are often used in signal and image processing. The novel part of our work is that we link these stages probabilistically by using a latent feature space. To use one joint model is a Bayesian requirement, which has the advantage to fuse information according to its certainty. The classiﬁer is implemented as hidden Markov model with Gaussian and Multinomial observation distributions deﬁned on a suitably chosen representation of autoregressive models. The Markov dependency is mo- tivated by the assumption that successive classiﬁcations will be corre- lated. Inference is done with Markov chain Monte Carlo (MCMC) tech- niques. We apply the proposed approach to synthetic data and to classi- ﬁcation of EEG that was recorded while the subjects performed different cognitive tasks. All experiments show that using a latent feature space results in a signiﬁcant improvement in generalization accuracy. Hence we expect that this idea generalizes well to other hierarchical models.
Playing is believing: The role of beliefs in multi-agent learning
Authors: Yu-Han Chang, Leslie Pack Kaelbling
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We propose a new classiﬁcation for multi-agent learning algorithms, with each league of players characterized by both their possible strategies and possible beliefs. Using this classiﬁcation, we review the optimality of ex- isting algorithms, including the case of interleague play. We propose an incremental improvement to the existing algorithms that seems to achieve average payoffs that are at least the Nash equilibrium payoffs in the long- run against fair opponents.
Probabilistic principles in unsupervised learning of visual structure: human data and a model
Authors: Shimon Edelman, Benjamin P. Hiles, Hwajin Yang, Nathan Intrator
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: To ﬁnd out how the representations of structured visual objects depend on the co-occurrence statistics of their constituents, we exposed subjects to a set of composite images with tight control exerted over (1) the condi- tional probabilities of the constituent fragments, and (2) the value of Bar- low’s criterion of “suspicious coincidence” (the ratio of joint probability to the product of marginals). We then compared the part veriﬁcation re- sponse times for various probe/target combinations before and after the exposure. For composite probes, the speedup was much larger for tar- gets that contained pairs of fragments perfectly predictive of each other, compared to those that did not. This effect was modulated by the sig- niﬁcance of their co-occurrence as estimated by Barlow’s criterion. For lone-fragment probes, the speedup in all conditions was generally lower than for composites. These results shed light on the brain’s strategies for unsupervised acquisition of structural information in vision.
Predictive Representations of State
Authors: Michael L. Littman, Richard S. Sutton
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We show that states of a dynamical system can be usefully repre(cid:173) sented by multi-step, action-conditional predictions of future ob(cid:173) servations. State representations that are grounded in data in this way may be easier to learn, generalize better, and be less depen(cid:173) dent on accurate prior models than, for example, POMDP state representations. Building on prior work by Jaeger and by Rivest and Schapire, in this paper we compare and contrast a linear spe(cid:173) cialization of the predictive approach with the state representa(cid:173) tions used in POMDPs and in k-order Markov models. Ours is the first specific formulation of the predictive idea that includes both stochasticity and actions (controls). We show that any system has a linear predictive state representation with number of predictions no greater than the number of states in its minimal POMDP model.
In predicting or controlling a sequence of observations, the concepts of state and state estimation inevitably arise. There have been two dominant approaches. The generative-model approach, typified by research on partially observable Markov de(cid:173) cision processes (POMDPs), hypothesizes a structure for generating observations and estimates its state and state dynamics. The history-based approach, typified by k-order Markov methods, uses simple functions of past observations as state, that is, as the immediate basis for prediction and control. (The data flow in these two ap(cid:173) proaches are diagrammed in Figure 1.) Of the two, the generative-model approach is more general. The model's internal state gives it temporally unlimited memory(cid:173) the ability to remember an event that happened arbitrarily long ago--whereas a history-based approach can only remember as far back as its history extends. The bane of generative-model approaches is that they are often strongly dependent on a good model of the system's dynamics. Most uses of POMDPs, for example, assume a perfect dynamics model and attempt only to estimate state. There are algorithms for simultaneously estimating state and dynamics (e.g., Chrisman, 1992), analogous to the Baum-Welch algorithm for the uncontrolled case (Baum et al., 1970), but these are only effective at tuning parameters that are already approximately cor(cid:173) rect (e.g., Shatkay & Kaelbling, 1997).
observations (and actions)
On Kernel-Target Alignment
Authors: Nello Cristianini, John Shawe-Taylor, André Elisseeff, Jaz S. Kandola
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We  introduce the notion of kernel-alignment, a  measure of similar(cid:173) ity  between two  kernel functions  or between a  kernel and a  target  function.  This quantity captures the degree of agreement between  a  kernel  and a  given  learning task,  and has  very  natural interpre(cid:173) tations  in  machine  learning,  leading also  to  simple  algorithms for  model selection and learning.  We analyse its theoretical properties,  proving that it  is  sharply concentrated around its expected value,  and  we  discuss  its  relation  with  other  standard  measures  of  per(cid:173) formance.  Finally we  describe  some  of the algorithms that  can be  obtained within this framework,  giving experimental results show(cid:173) ing that adapting the kernel  to improve  alignment  on the labelled  data  significantly  increases  the  alignment  on  the  test  set,  giving  improved  classification  accuracy.  Hence,  the  approach  provides  a  principled method of performing transduction. 
Keywords:  Kernels,  alignment, eigenvectors, eigenvalues, transduction
Algorithmic Luckiness
Authors: Ralf Herbrich, Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: In  contrast  to  standard  statistical  learning  theory  which  studies  uniform bounds on the expected error we present a framework that  exploits  the  specific  learning  algorithm  used.  Motivated  by  the  luckiness framework  [8]  we  are also  able to exploit the serendipity  of the training sample.  The main difference to previous approaches  lies  in  the  complexity  measure;  rather  than covering  all  hypothe(cid:173) ses  in  a  given  hypothesis  space  it  is  only  necessary  to  cover  the  functions  which  could  have  been  learned  using  the  fixed  learning  algorithm.  We  show  how  the  resulting  framework  relates  to  the  VC, luckiness and compression frameworks.  Finally,  we  present an  application  of this  framework  to  the  maximum  margin  algorithm  for  linear classifiers which results in a  bound that exploits both the  margin and the distribution of the data in feature  space.
Optimising Synchronisation Times for Mobile Devices
Authors: Neil D. Lawrence, Antony I. T. Rowstron, Christopher M. Bishop, Michael J. Taylor
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: With the increasing number of users of mobile computing devices  (e.g.  personal digital assistants) and the advent of third generation  mobile phones, wireless communications are becoming increasingly  important.  Many  applications  rely  on  the  device  maintaining  a  replica  of a  data-structure which  is  stored on  a  server,  for  exam(cid:173) ple news databases, calendars and e-mail.  ill this paper we explore  the question of the optimal strategy for synchronising such replicas.  We utilise probabilistic models to represent how the data-structures  evolve  and to model  user behaviour.  We  then formulate objective  functions which can be minimised with respect to the synchronisa(cid:173) tion timings.  We demonstrate, using two real world data-sets, that  a user can obtain more up-to-date information using our approach.
Spike timing and the coding of naturalistic sounds in a central auditory area of songbirds
Authors: B. D. Wright, Kamal Sen, William Bialek, A. J. Doupe
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: In nature, animals encounter high dimensional sensory stimuli that have complex statistical and dynamical structure. Attempts to study the neu- ral coding of these natural signals face challenges both in the selection of the signal ensemble and in the analysis of the resulting neural responses. For zebra ﬁnches, naturalistic stimuli can be deﬁned as sounds that they encounter in a colony of conspeciﬁc birds. We assembled an ensemble of these sounds by recording groups of 10-40 zebra ﬁnches, and then ana- lyzed the response of single neurons in the songbird central auditory area (ﬁeld L) to continuous playback of long segments from this ensemble. Following methods developed in the ﬂy visual system, we measured the information that spike trains provide about the acoustic stimulus with- out any assumptions about which features of the stimulus are relevant. Preliminary results indicate that large amounts of information are carried by spike timing, with roughly half of the information accessible only at time resolutions better than 10 ms; additional information is still be- ing revealed as time resolution is improved to 2 ms. Information can be decomposed into that carried by the locking of individual spikes to the stimulus (or modulations of spike rate) vs. that carried by timing in spike patterns. Initial results show that in ﬁeld L, temporal patterns give at least
 % extra information. Thus, single central auditory neurons can pro-
vide an informative representation of naturalistic sounds, in which spike timing may play a signiﬁcant role.
A Variational Approach to Learning Curves
Authors: Dörthe Malzahn, Manfred Opper
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We combine the replica approach from statistical physics with a varia- tional approach to analyze learning curves analytically. We apply the method to Gaussian process regression. As a main result we derive ap- proximative relations between empirical error measures, the generaliza- tion error and the posterior variance.
Why Neuronal Dynamics Should Control Synaptic Learning Rules
Authors: Jesper Tegnér, Ádám Kepecs
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: Hebbian learning rules are generally formulated as static rules. Un(cid:173) der changing condition (e.g. neuromodulation, input statistics)  most rules are sensitive to parameters. In particular, recent work  has focused on two different formulations of spike-timing-dependent  plasticity rules. Additive STDP [1] is remarkably versatile but  also very fragile, whereas multiplicative STDP [2, 3] is more ro(cid:173) bust but lacks attractive features such as synaptic competition and  rate stabilization. Here we address the problem of robustness in  the additive STDP rule. We derive an adaptive control scheme,  where the learning function is under fast dynamic control by post(cid:173) synaptic activity to stabilize learning under a variety of conditions.  Such a control scheme can be implemented using known biophysical  mechanisms of synapses. We show that this adaptive rule makes  the addit ive STDP more robust. Finally, we give an example how  meta plasticity of the adaptive rule can be used to guide STDP  into different type of learning regimes.
Latent Dirichlet Allocation
Authors: David M. Blei, Andrew Y. Ng, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 14  (NIPS 2001) (Main Conference Track), 2001

Abstract: We propose a generative model for text and other collections of dis(cid:173) crete data that generalizes or improves on several previous models  including naive Bayes/unigram, mixture of unigrams  [6],  and Hof(cid:173) mann's  aspect  model,  also  known  as  probabilistic latent  semantic  indexing  (pLSI)  [3].  In  the  context  of text  modeling,  our  model  posits  that  each  document  is  generated  as  a  mixture  of  topics,  where  the  continuous-valued  mixture  proportions  are  distributed  as  a  latent  Dirichlet  random  variable.  Inference  and  learning  are  carried out efficiently  via variational  algorithms.  We  present  em(cid:173) pirical  results  on  applications  of this  model  to  problems  in  text  modeling,  collaborative filtering,  and text classification.