mtybilly/Instruct2DS huggingface dataset:
Dopamine Induced Bistability Enhances Signal Processing in Spiny Neurons
Authors: Aaron J. Gruber, Sara A. Solla, James C. Houk
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Single  unit  activity  in  the  striatum  of  awake  monkeys  shows  a  marked  dependence  on  the  expected  reward  that  a  behavior  will  elicit.  We  present  a  computational  model  of  spiny  neurons,  the  principal neurons of the striatum, to assess the hypothesis that di(cid:173) rect  neuromodulatory  effects  of dopamine  through  the  activation  of D 1  receptors  mediate  the  reward  dependency  of spiny  neuron  activity.  Dopamine  release  results  in  the  amplification  of key  ion  currents,  leading  to  the  emergence  of  bistability,  which  not  only  modulates the peak firing  rate but also introduces a  temporal and  state dependence  of the  model's  response,  thus improving the de(cid:173) tectability of temporally correlated inputs.
An Information Theoretic Approach to the Functional Classification of Neurons
Authors: Elad Schneidman, William Bialek, Michael Ii
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A population of neurons typically exhibits a broad diversity of responses to sensory inputs. The intuitive notion of functional classiﬁcation is that cells can be clustered so that most of the diversity is captured by the iden- tity of the clusters rather than by individuals within clusters. We show how this intuition can be made precise using information theory, with- out any need to introduce a metric on the space of stimuli or responses. Applied to the retinal ganglion cells of the salamander, this approach re- covers classical results, but also provides clear evidence for subclasses beyond those identiﬁed previously. Further, we ﬁnd that each of the gan- glion cells is functionally unique, and that even within the same subclass only a few spikes are needed to reliably distinguish between cells.
Shape Recipes: Scene Representations that Refer to the Image
Authors: William T. Freeman, Antonio Torralba
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The goal of low-level vision is to estimate an underlying scene, given an observed image. Real-world scenes (eg, albedos or shapes) can be very complex, conventionally requiring high dimensional representations which are hard to estimate and store. We propose a low-dimensional rep- resentation, called a scene recipe, that relies on the image itself to de- scribe the complex scene conﬁgurations. Shape recipes are an example: these are the regression coefﬁcients that predict the bandpassed shape from image data. We describe the beneﬁts of this representation, and show two uses illustrating their properties: (1) we improve stereo shape estimates by learning shape recipes at low resolution and applying them at full resolution; (2) Shape recipes implicitly contain information about lighting and materials and we use them for material segmentation.
Real-Time Particle Filters
Authors: Cody Kwok, Dieter Fox, Marina Meila
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Particle ﬁlters estimate the state of dynamical systems from sensor infor- mation. In many real time applications of particle ﬁlters, however, sensor information arrives at a signiﬁcantly higher rate than the update rate of the ﬁlter. The prevalent approach to dealing with such situations is to update the particle ﬁlter as often as possible and to discard sensor information that cannot be processed in time. In this paper we present real-time particle ﬁl- ters, which make use of all sensor information even when the ﬁlter update rate is below the update rate of the sensors. This is achieved by represent- ing posteriors as mixtures of sample sets, where each mixture component integrates one observation arriving during a ﬁlter update. The weights of the mixture components are set so as to minimize the approximation error introduced by the mixture representation. Thereby, our approach focuses computational resources (samples) on valuable sensor information. Exper- iments using data collected with a mobile robot show that our approach yields strong improvements over other approaches.
Critical Lines in Symmetry of Mixture Models and its Application to Component Splitting
Authors: Kenji Fukumizu, Shotaro Akaho, Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We show the existence of critical points as lines for the likelihood func- tion of mixture-type models. They are given by embedding of a critical point for models with less components. A sufﬁcient condition that the critical line gives local maxima or saddle points is also derived. Based on this fact, a component-split method is proposed for a mixture of Gaus- sian components, and its effectiveness is veriﬁed through experiments.
Manifold Parzen Windows
Authors: Pascal Vincent, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The similarity between objects is a fundamental element of many learn- ing algorithms. Most non-parametric methods take this similarity to be ﬁxed, but much recent work has shown the advantages of learning it, in particular to exploit the local invariances in the data or to capture the possibly non-linear manifold on which most of the data lies. We propose a new non-parametric kernel density estimation method which captures the local structure of an underlying manifold through the leading eigen- vectors of regularized local covariance matrices. Experiments in density estimation show signiﬁcant improvements with respect to Parzen density estimators. The density estimators can also be used within Bayes classi- ﬁers, yielding classiﬁcation rates similar to SVMs and much superior to the Parzen classiﬁer.
Parametric Mixture Models for Multi-Labeled Text
Authors: Naonori Ueda, Kazumi Saito
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We propose probabilistic generative models, called parametric mix- ture models (PMMs), for multiclass, multi-labeled text categoriza- tion problem. Conventionally, the binary classi(cid:12)cation approach has been employed, in which whether or not text belongs to a cat- egory is judged by the binary classi(cid:12)er for every category. In con- trast, our approach can simultaneously detect multiple categories of text using PMMs. We derive e(cid:14)cient learning and prediction algo- rithms for PMMs. We also empirically show that our method could signi(cid:12)cantly outperform the conventional binary methods when ap- plied to multi-labeled text categorization using real World Wide Web pages.
Transductive and Inductive Methods for Approximate Gaussian Process Regression
Authors: Anton Schwaighofer, Volker Tresp
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Gaussian process regression allows a simple analytical treatment of ex- act Bayesian inference and has been found to provide good performance, yet scales badly with the number of training data. In this paper we com- pare several approaches towards scaling Gaussian processes regression to large data sets: the subset of representers method, the reduced rank approximation, online Gaussian processes, and the Bayesian commit- tee machine. Furthermore we provide theoretical insight into some of our experimental results. We found that subset of representers methods can give good and particularly fast predictions for data sets with high and medium noise levels. On complex low noise data sets, the Bayesian committee machine achieves signiﬁcantly better accuracy, yet at a higher computational cost.
Adapting Codes and Embeddings for Polychotomies
Authors: Gunnar Rätsch, Sebastian Mika, Alex J. Smola
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper we consider formulations of multi-class problems based on a generalized notion of a margin and using output coding. This includes, but is not restricted to, standard multi-class SVM formulations. Differ- ently from many previous approaches we learn the code as well as the embedding function. We illustrate how this can lead to a formulation that allows for solving a wider range of problems with for instance many classes or even “missing classes”. To keep our optimization problems tractable we propose an algorithm capable of solving them using two- class classiﬁers, similar in spirit to Boosting.
Topographic Map Formation by Silicon Growth Cones
Authors: Brian Taba, Kwabena A. Boahen
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We  describe  a  self-configuring  neuromorphic  chip  that  uses  a  model of activity-dependent axon remodeling to  automatically wire  topographic  maps  based  solely  on  input  correlations.  Axons  are  guided by growth cones, which are modeled in analog VLSI for the  first  time.  Growth  cones  migrate  up  neurotropin  gradients,  which  are  represented  by  charge  diffusing  in  transistor  channels.  Virtual  axons  move  by  rerouting  address-events.  We  refined  an  initially  gross topographic projection by simulating retinal wave input. 
1  Neuromorphic  Systems 
Neuromorphic  engineers  are  attempting  to  match  the  computational  efficiency  of  biological  systems  by  morphing  neurocircuitry  into  silicon  circuits  [1].  One  of the  most  detailed  implementations  to  date  is  the  silicon  retina  described  in  [2] .  This  chip  comprises  thirteen different cell types,  each  of which must be individually and  painstakingly  wired.  While  this  circuit-level  approach  has  been  very  successful  in  sensory  systems,  it  is  less  helpful  when  modeling  largely  unelucidated  and  exceedingly plastic higher processing centers in cortex. 
Instead  of  an  explicit  blueprint  for  every  cortical  area,  what  is  needed  is  a  developmental rule that can wire complex circuits from  minimal  specifications.  One  candidate  is  the  famous  "cells  that  fire  together  wire  together"  rule,  which  strengthens  excitatory  connections  between  coactive  presynaptic  and  postsynaptic  cells.  We  implemented a self-rewiring scheme  of this  type in  silicon,  taking our cue  from axon remodeling during development.
Analysis of Information in Speech Based on MANOVA
Authors: Sachin S. Kajarekar, Hynek Hermansky
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We  propose  analysis  of information  in  speech  using  three  sources  - language  (phone),  speaker and channeL  Information in speech is  measured as mutual information between the source and the set of  features  extracted  from  speech  signaL  We  assume  that  distribu(cid:173) tion of features  can be modeled  using  Gaussian distribution.  The  mutual  information  is  computed  using  the  results  of  analysis  of  variability in speech.  We  observe similarity in the results of phone  variability and phone information, and show that the results of the  proposed  analysis  have  more  meaningful  interpretations  than  the  analysis of variability.
Discriminative Binaural Sound Localization
Authors: Ehud Ben-reuven, Yoram Singer
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Time difference of arrival (TDOA) is commonly used to estimate the az- imuth of a source in a microphone array. The most common methods to estimate TDOA are based on ﬁnding extrema in generalized cross- correlation waveforms. In this paper we apply microphone array tech- niques to a manikin head. By considering the entire cross-correlation waveform we achieve azimuth prediction accuracy that exceeds extrema locating methods. We do so by quantizing the azimuthal angle and treating the prediction problem as a multiclass categorization task. We demonstrate the merits of our approach by evaluating the various ap- proaches on Sony’s AIBO robot.
Fractional Belief Propagation
Authors: Wim Wiegerinck, Tom Heskes
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider loopy belief propagation for approximate inference in prob- abilistic graphical models. A limitation of the standard algorithm is that clique marginals are computed as if there were no loops in the graph. To overcome this limitation, we introduce fractional belief propagation. Fractional belief propagation is formulated in terms of a family of ap- proximate free energies, which includes the Bethe free energy and the naive mean-ﬁeld free as special cases. Using the linear response correc- tion of the clique marginals, the scale parameters can be tuned. Simula- tion results illustrate the potential merits of the approach.
Stability-Based Model Selection
Authors: Tilman Lange, Mikio L. Braun, Volker Roth, Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Model selection is linked to model assessment, which is the problem of comparing different models, or model parameters, for a speciﬁc learning task. For supervised learning, the standard practical technique is cross- validation, which is not applicable for semi-supervised and unsupervised settings. In this paper, a new model assessment scheme is introduced which is based on a notion of stability. The stability measure yields an upper bound to cross-validation in the supervised case, but extends to semi-supervised and unsupervised problems. In the experimental part, the performance of the stability measure is studied for model order se- lection in comparison to standard techniques in this area.
Hidden Markov Model of Cortical Synaptic Plasticity: Derivation of the Learning Rule
Authors: Michael Eisele, Kenneth D. Miller
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Cortical synaptic plasticity depends on the relative timing of pre- and postsynaptic spikes and also on the temporal pattern of presynaptic spikes and of postsynaptic spikes. We study the hypothesis that cortical synap- tic plasticity does not associate individual spikes, but rather whole ﬁr- ing episodes, and depends only on when these episodes start and how long they last, but as little as possible on the timing of individual spikes. Here we present the mathematical background for such a study. Stan- dard methods from hidden Markov models are used to deﬁne what “ﬁr- ing episodes” are. Estimating the probability of being in such an episode requires not only the knowledge of past spikes, but also of future spikes. We show how to construct a causal learning rule, which depends only on past spikes, but associates pre- and postsynaptic ﬁring episodes as if it also knew future spikes. We also show that this learning rule agrees with some features of synaptic plasticity in superﬁcial layers of rat visual cortex (Froemke and Dan, Nature 416:433, 2002).
Field-Programmable Learning Arrays
Authors: Seth Bridges, Miguel Figueroa, Chris Diorio, David Hsu
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper introduces the Field-Programmable Learning Array, a new paradigm for rapid prototyping of learning primitives and machine- learning algorithms in silicon. The FPLA is a mixed-signal counterpart to the all-digital Field-Programmable Gate Array in that it enables rapid prototyping of algorithms in hardware. Unlike the FPGA, the FPLA is targeted directly for machine learning by providing local, parallel, on- line analog learning using ﬂoating-gate MOS synapse transistors. We present a prototype FPLA chip comprising an array of reconﬁgurable computational blocks and local interconnect. We demonstrate the via- bility of this architecture by mapping several learning circuits onto the prototype chip.
Boosting Density Estimation
Authors: Saharon Rosset, Eran Segal
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Several authors have suggested viewing boosting as a gradient descent search for a good ﬁt in function space. We apply gradient-based boosting methodology to the unsupervised learning problem of density estimation. We show convergence properties of the algorithm and prove that a strength of weak learnability prop- erty applies to this problem as well. We illustrate the potential of this approach through experiments with boosting Bayesian networks to learn density models.
Support Vector Machines for Multiple-Instance Learning
Authors: Stuart Andrews, Ioannis Tsochantaridis, Thomas Hofmann
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This  paper  presents  two  new  formulations  of  multiple-instance  learning as a  maximum margin problem.  The proposed extensions  of the  Support  Vector  Machine  (SVM)  learning  approach lead  to  mixed integer quadratic programs that can be solved  heuristically.  Our generalization of SVMs  makes  a  state-of-the-art classification  technique,  including  non-linear  classification via kernels,  available  to  an  area that  up  to now  has  been  largely  dominated by  special  purpose methods.  We  present  experimental  results  on  a  pharma(cid:173) ceutical data set and on applications in automated image indexing  and document categorization.
Rate Distortion Function in the Spin Glass State: A Toy Model
Authors: Tatsuto Murayama, Masato Okada
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We applied statistical mechanics to an inverse problem of linear mapping to investigate the physics of optimal lossy compressions. We used the replica symmetry breaking technique with a toy model to demonstrate Shannon’s result. The rate distortion function, which is widely known as the theoretical limit of the compression with a ﬁdelity criterion, is derived. Numerical study shows that sparse constructions of the model provide suboptimal compressions.
Real Time Voice Processing with Audiovisual Feedback: Toward Autonomous Agents with Perfect Pitch
Authors: Lawrence K. Saul, Daniel D. Lee, Charles L. Isbell, Yann L. Cun
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We have implemented a real time front end for detecting voiced speech and estimating its fundamental frequency. The front end performs the signal processing for voice-driven agents that attend to the pitch contours of human speech and provide continuous audiovisual feedback. The al- gorithm we use for pitch tracking has several distinguishing features: it makes no use of FFTs or autocorrelation at the pitch period; it updates the pitch incrementally on a sample-by-sample basis; it avoids peak picking and does not require interpolation in time or frequency to obtain high res- olution estimates; and it works reliably over a four octave range, in real time, without the need for postprocessing to produce smooth contours. The algorithm is based on two simple ideas in neural computation: the introduction of a purposeful nonlinearity, and the error signal of a least squares ﬁt. The pitch tracker is used in two real time multimedia applica- tions: a voice-to-MIDI player that synthesizes electronic music from vo- calized melodies, and an audiovisual Karaoke machine with multimodal feedback. Both applications run on a laptop and display the user’s pitch scrolling across the screen as he or she sings into the computer.
Visual Development Aids the Acquisition of Motion Velocity Sensitivities
Authors: Robert A. Jacobs, Melissa Dominguez
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider the hypothesis that systems learning aspects of visual per- ception may beneﬁt from the use of suitably designed developmental pro- gressions during training. Four models were trained to estimate motion velocities in sequences of visual images. Three of the models were “de- velopmental models” in the sense that the nature of their input changed during the course of training. They received a relatively impoverished visual input early in training, and the quality of this input improved as training progressed. One model used a coarse-to-multiscale develop- mental progression (i.e. it received coarse-scale motion features early in training and ﬁner-scale features were added to its input as training progressed), another model used a ﬁne-to-multiscale progression, and the third model used a random progression. The ﬁnal model was non- developmental in the sense that the nature of its input remained the same throughout the training period. The simulation results show that the coarse-to-multiscale model performed best. Hypotheses are offered to account for this model’s superior performance. We conclude that suit- ably designed developmental sequences can be useful to systems learn- ing to estimate motion velocities. The idea that visual development can aid visual learning is a viable hypothesis in need of further study.
On the Complexity of Learning the Kernel Matrix
Authors: Olivier Bousquet, Daniel Herrmann
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate data based procedures for selecting the kernel when learn- ing with Support Vector Machines. We provide generalization error bounds by estimating the Rademacher complexities of the corresponding function classes. In particular we obtain a complexity bound for function classes induced by kernels with given eigenvectors, i.e., we allow to vary the spectrum and keep the eigenvectors ﬁx. This bound is only a loga- rithmic factor bigger than the complexity of the function class induced by a single kernel. However, optimizing the margin over such classes leads to overﬁtting. We thus propose a suitable way of constraining the class. We use an efﬁcient algorithm to solve the resulting optimization problem, present preliminary experimental results, and compare them to an alignment-based approach.
Clustering with the Fisher Score
Authors: Koji Tsuda, Motoaki Kawanabe, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recently the Fisher score (or the Fisher kernel) is increasingly used as a feature extractor for classiﬁcation problems. The Fisher score is a vector of parameter derivatives of loglikelihood of a probabilistic model. This paper gives a theoretical analysis about how class information is pre- served in the space of the Fisher score, which turns out that the Fisher score consists of a few important dimensions with class information and many nuisance dimensions. When we perform clustering with the Fisher score, K-Means type methods are obviously inappropriate because they make use of all dimensions. So we will develop a novel but simple clus- tering algorithm specialized for the Fisher score, which can exploit im- portant dimensions. This algorithm is successfully tested in experiments with artiﬁcial data and real data (amino acid sequences).
as follows: Let us assume that a probabilistic model     
parameter estimate 
Artefactual Structure from Least-Squares Multidimensional Scaling
Authors: Nicholas P. Hughes, David Lowe
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider the problem of illusory or artefactual structure from the vi- sualisation of high-dimensional structureless data. In particular we ex- amine the role of the distance metric in the use of topographic mappings based on the statistical ﬁeld of multidimensional scaling. We show that the use of a squared Euclidean metric (i.e. the SS TRESS measure) gives rise to an annular structure when the input data is drawn from a high- dimensional isotropic distribution, and we provide a theoretical justiﬁca- tion for this observation.
Adaptation and Unsupervised Learning
Authors: Peter Dayan, Maneesh Sahani, Gregoire Deback
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Adaptation is a ubiquitous neural and psychological phenomenon, with a wealth of instantiations and implications. Although a basic form of plasticity, it has, bar some notable exceptions, attracted computational theory of only one main variety. In this paper, we study adaptation from the perspective of factor analysis, a paradigmatic technique of unsuper- vised learning. We use factor analysis to re-interpret a standard view of adaptation, and apply our new model to some recent data on adaptation in the domain of face discrimination.
Learning in Zero-Sum Team Markov Games Using Factored Value Functions
Authors: Michail G. Lagoudakis, Ronald Parr
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a new method for learning good strategies in zero-sum Markov games in which each side is composed of multiple agents col- laborating against an opposing team of agents. Our method requires full observability and communication during learning, but the learned poli- cies can be executed in a distributed manner. The value function is rep- resented as a factored linear architecture and its structure determines the necessary computational resources and communication bandwidth. This approach permits a tradeoff between simple representations with little or no communication between agents and complex, computationally inten- sive representations with extensive coordination between agents. Thus, we provide a principled means of using approximation to combat the exponential blowup in the joint action space of the participants. The ap- proach is demonstrated with an example that shows the efﬁciency gains over naive enumeration.
Learning to Take Concurrent Actions
Authors: Khashayar Rohanimanesh, Sridhar Mahadevan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate a general semi-Markov Decision Process (SMDP) framework for modeling concurrent decision making, where agents learn optimal plans over concurrent temporally extended actions. We introduce three types of parallel termination schemes { all, any and continue { and theoretically and experimentally compare them.
Handling Missing Data with Variational Bayesian Learning of ICA
Authors: Kwokleung Chan, Te-Won Lee, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Missing data is common in real-world datasets and is a problem for many estimation techniques. We have developed a variational Bayesian method to perform Independent Component Analysis (ICA) on high-dimensional data containing missing entries. Missing data are handled naturally in the Bayesian framework by integrating the generative density model. Mod- eling the distributions of the independent sources with mixture of Gaus- sians allows sources to be estimated with different kurtosis and skewness. The variational Bayesian method automatically determines the dimen- sionality of the data and yields an accurate density model for the ob- served data without overﬁtting problems. This allows direct probability estimation of missing values in the high dimensional space and avoids dimension reduction preprocessing which is not feasible with missing data.
Ranking with Large Margin Principle: Two Approaches
Authors: Amnon Shashua, Anat Levin
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We discuss the problem of ranking k instances with the use of a "large  margin" principle. We introduce two main approaches: the first is the  "fixed margin" policy in which the margin of the closest neighboring  classes is being maximized - which turns out to be a direct generaliza(cid:173) tion of SVM to ranking learning. The second approach allows for k - 1  different margins where the sum of margins is maximized. This approach  is shown to reduce to lI-SVM when the number of classes k = 2. Both  approaches are optimal in size of 21 where I is the total number of training  examples. Experiments performed on visual classification and "collab(cid:173) orative filtering" show that both approaches outperform existing ordinal  regression algorithms applied for ranking and multi-class SVM applied  to general multi-class classification.
A Bilinear Model for Sparse Coding
Authors: David B. Grimes, Rajesh P. N. Rao
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recent algorithms for sparse coding and independent component analy- sis (ICA) have demonstrated how localized features can be learned from natural images. However, these approaches do not take image transfor- mations into account. As a result, they produce image codes that are redundant because the same feature is learned at multiple locations. We describe an algorithm for sparse coding based on a bilinear generative model of images. By explicitly modeling the interaction between im- age features and their transformations, the bilinear approach helps reduce redundancy in the image code and provides a basis for transformation- invariant vision. We present results demonstrating bilinear sparse coding of natural images. We also explore an extension of the model that can capture spatial relationships between the independent features of an ob- ject, thereby providing a new framework for parts-based object recogni- tion.
Information Diffusion Kernels
Authors: Guy Lebanon, John D. Lafferty
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A new family of kernels for statistical learning is introduced that ex- ploits the geometric structure of statistical models. Based on the heat equation on the Riemannian manifold deﬁned by the Fisher informa- tion metric, information diffusion kernels generalize the Gaussian kernel of Euclidean space, and provide a natural way of combining generative statistical modeling with non-parametric discriminative learning. As a special case, the kernels give a new approach to applying kernel-based learning algorithms to discrete data. Bounds on covering numbers for the new kernels are proved using spectral theory in differential geometry, and experimental results are presented for text classiﬁcation.
Boosted Dyadic Kernel Discriminants
Authors: Baback Moghaddam, Gregory Shakhnarovich
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We introduce a novel learning algorithm for binary classi(cid:12)cation with hyperplane discriminants based on pairs of training points from opposite classes (dyadic hypercuts). This algorithm is further extended to nonlinear discriminants using kernel functions satisfy- ing Mercer’s conditions. An ensemble of simple dyadic hypercuts is learned incrementally by means of a con(cid:12)dence-rated version of Ad- aBoost, which provides a sound strategy for searching through the (cid:12)nite set of hypercut hypotheses. In experiments with real-world datasets from the UCI repository, the generalization performance of the hypercut classi(cid:12)ers was found to be comparable to that of SVMs and k-NN classi(cid:12)ers. Furthermore, the computational cost of classi(cid:12)cation (at run time) was found to be similar to, or bet- ter than, that of SVM. Similarly to SVMs, boosted dyadic kernel discriminants tend to maximize the margin (via AdaBoost). In contrast to SVMs, however, we o(cid:11)er an on-line and incremental learning machine for building kernel discriminants whose complex- ity (number of kernel evaluations) can be directly controlled (traded o(cid:11) for accuracy).
Derivative Observations in Gaussian Process Models of Dynamic Systems
Authors: E. Solak, R. Murray-smith, W. E. Leithead, D. J. Leith, Carl E. Rasmussen
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Gaussian processes provide an approach to nonparametric modelling which allows a straightforward combination of function and derivative observations in an empirical model. This is of particular importance in identiﬁcation of nonlinear dynamic systems from experimental data. 1) It allows us to combine derivative information, and associated uncertainty with normal function observations into the learning and inference pro- cess. This derivative information can be in the form of priors speciﬁed by an expert or identiﬁed from perturbation data close to equilibrium. 2) It allows a seamless fusion of multiple local linear models in a consis- tent manner, inferring consistent models and ensuring that integrability constraints are met. 3) It improves dramatically the computational ef- ﬁciency of Gaussian process models for dynamic system identiﬁcation, by summarising large quantities of near-equilibrium data by a handful of linearisations, reducing the training set size – traditionally a problem for Gaussian process models.
Global Versus Local Methods in Nonlinear Dimensionality Reduction
Authors: Vin D. Silva, Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recently proposed algorithms for nonlinear dimensionality reduction fall broadly into two categories which have different advantages and disad- vantages: global (Isomap [1]), and local (Locally Linear Embedding [2], Laplacian Eigenmaps [3]). We present two variants of Isomap which combine the advantages of the global approach with what have previ- ously been exclusive advantages of local methods: computational spar- sity and the ability to invert conformal maps.
Learning Graphical Models with Mercer Kernels
Authors: Francis R. Bach, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a class of algorithms for learning the structure of graphical models from data. The algorithms are based on a measure known as the kernel generalized variance (KGV), which essentially allows us to treat all variables on an equal footing as Gaussians in a feature space obtained from Mercer kernels. Thus we are able to learn hybrid graphs involving discrete and continuous variables of arbitrary type. We explore the computational properties of our approach, showing how to use the kernel trick to compute the relevant statistics in linear time. We illustrate our framework with experiments involving discrete and continuous data.
Stochastic Neighbor Embedding
Authors: Geoffrey E. Hinton, Sam T. Roweis
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We describe a probabilistic approach to the task of placing objects, de- scribed by high-dimensional vectors or by pairwise dissimilarities, in a low-dimensional space in a way that preserves neighbor identities. A Gaussian is centered on each object in the high-dimensional space and the densities under this Gaussian (or the given dissimilarities) are used to deﬁne a probability distribution over all the potential neighbors of the object. The aim of the embedding is to approximate this distribu- tion as well as possible when the same operation is performed on the low-dimensional “images” of the objects. A natural cost function is a sum of Kullback-Leibler divergences, one per object, which leads to a simple gradient for adjusting the positions of the low-dimensional im- ages. Unlike other dimensionality reduction methods, this probabilistic framework makes it easy to represent each object by a mixture of widely separated low-dimensional images. This allows ambiguous objects, like the document count vector for the word “bank”, to have versions close to the images of both “river” and “ﬁnance” without forcing the images of outdoor concepts to be located close to those of corporate concepts.
A Model for Real-Time Computation in Generic Neural Microcircuits
Authors: Wolfgang Maass, Thomas Natschläger, Henry Markram
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Henry Markram Brain Mind Institute
EPFL, Lausanne, Switzerland
henry.markram@epﬂ.ch
A key challenge for neural modeling is to explain how a continuous stream of multi-modal input from a rapidly changing environment can be processed by stereotypical recurrent circuits of integrate-and-ﬁre neurons in real-time. We propose a new computational model that is based on principles of high dimensional dynamical systems in combination with statistical learning theory. It can be implemented on generic evolved or found recurrent circuitry.
Categorization Under Complexity: A Unified MDL Account of Human Learning of Regular and Irregular Categories
Authors: David Fass, Jacob Feldman
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present an account of human concept learning-that is, learning of categories from examples-based on the principle of minimum descrip(cid:173) tion length (MDL). In support of this theory, we tested a wide range of two-dimensional concept types, including both regular (simple) and highly irregular (complex) structures, and found the MDL theory to give a good account of subjects' performance. This suggests that the intrin(cid:173) sic complexity of a concept (that is, its description -length) systematically influences its leamability.
1- The Structure of Categories
A number of different principles have been advanced to explain the manner in which hu(cid:173) mans learn to categorize objects. It has been variously suggested that the underlying prin(cid:173) ciple might be the similarity structure of objects [1], the manipulability of decision bound~ aries [2], or Bayesian inference [3][4]. While many of these theories are mathematically well-grounded and have been successful in explaining a range of experimental findings, they have commonly only been tested on a narrow collection of concept types similar to the simple unimodal categories of Figure 1(a-e).
Learning with Multiple Labels
Authors: Rong Jin, Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper,  we  study a  special kind of learning problem in which  each  training  instance  is  given  a  set  of  (or  distribution  over)  candidate  class  labels  and  only  one  of the  candidate  labels  is  the  correct  one.  Such  a  problem  can  occur,  e.g.,  in  an  information  retrieval  setting  where  a  set  of words  is  associated  with  an  image,  or  if  classes  labels  are  organized  hierarchically.  We  propose  a  novel  discriminative  approach  for  handling  the  ambiguity  of class  labels  in the  training  examples.  The  experiments with the  proposed  approach over five  different UCI datasets  show that our approach is  able  to  find  the  correct label  among the  set of candidate  labels  and  actually  achieve  performance  close  to  the  case  when  each  training  instance  is  given  a  single  correct  label.  In  contrast,  naIve  methods  degrade rapidly as  more ambiguity is introduced into the labels.
Branching Law for Axons
Authors: Dmitri B. Chklovskii, Armen Stepanyants
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: What  determines  the  caliber  of  axonal  branches?  We  pursue  the  hypothesis  that  the  axonal  caliber  has  evolved  to  minimize  signal  propagation delays, while keeping arbor volume to  a  minimum.  We  show  that  for  a  general  cost  function  the  optimal  diameters  of  mother  (do)  and  daughter  (d], d2 )  branches at a bifurcation obey  h  a  ranc  mg  aw:  e  envatIOn  re les  on  t  e  fact  that the conduction  speed  scales  with  the  axon  diameter to  the  power  V  (v = 1  for  myelinated  axons  and  V  = 0.5  myelinated  axons).  We  test  the  branching  law  on  the  available  experimental data and find  a reasonable agreement. 
d v+2  d v+2  d v+2  Th  d "  0  = ]   +  2
Dyadic Classification Trees via Structural Risk Minimization
Authors: Clayton Scott, Robert Nowak
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Classiﬁcation trees are one of the most popular types of classiﬁers, with ease of implementation and interpretation being among their attractive features. Despite the widespread use of classiﬁcation trees, theoretical analysis of their performance is scarce. In this paper, we show that a new family of classiﬁcation trees, called dyadic classiﬁcation trees (DCTs), are near optimal (in a minimax sense) for a very broad range of clas- siﬁcation problems. This demonstrates that other schemes (e.g., neural networks, support vector machines) cannot perform signiﬁcantly better than DCTs in many cases. We also show that this near optimal perfor- mance is attained with linear (in the number of training data) complexity growing and pruning algorithms. Moreover, the performance of DCTs on benchmark datasets compares favorably to that of standard CART, which is generally more computationally intensive and which does not possess similar near optimality properties. Our analysis stems from the- oretical results on structural risk minimization, on which the pruning rule for DCTs is based.
Dynamical Causal Learning
Authors: David Danks, Thomas L. Griffiths, Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: theories of human causal
focus primarily on long-run predictions:
learning and Current psychological two by judgment estimating parameters of a causal Bayes nets (though for different parameterizations), and a third through structural learning. This short-run behavior by examining paper dynamical versions of these three theories, and comparing their predictions to a real-world dataset.
focuses on people's
Fast Exact Inference with a Factored Model for Natural Language Parsing
Authors: Dan Klein, Christopher D. Manning
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a novel generative model for natural language tree structures in which semantic (lexical dependency) and syntactic (PCFG) structures are scored with separate models. This factorization provides concep- tual simplicity, straightforward opportunities for separately improving the component models, and a level of performance comparable to simi- lar, non-factored models. Most importantly, unlike other modern parsing models, the factored model admits an extremely effective A* parsing al- gorithm, which enables efﬁcient, exact inference.
Morton-Style Factorial Coding of Color in Primary Visual Cortex
Authors: Javier R. Movellan, Thomas Wachtler, Thomas D. Albright, Terrence Sejnowski
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We introduce the notion of Morton-style factorial coding and illustrate how it may help understand information integration and perceptual cod- ing in the brain. We show that by focusing on average responses one may miss the existence of factorial coding mechanisms that become only apparent when analyzing spike count histograms. We show evidence suggesting that the classical/non-classical receptive ﬁeld organization in the cortex effectively enforces the development of Morton-style factorial codes. This may provide some cues to help understand perceptual cod- ing in the brain and to develop new unsupervised learning algorithms. While methods like ICA (Bell & Sejnowski, 1997) develop independent codes, in Morton-style coding the goal is to make two or more external aspects of the world become independent when conditioning on internal representations.
In this paper we introduce the notion of Morton-style factorial coding and illustrate how it may help analyze information integration and perceptual organization in the brain. In the neurosciences factorial codes are often studied in the context of mean tuning curves. A tuning curve is called separable if it can be expressed as the product of terms selectively inﬂuenced by different stimulus dimensions. Separable tuning curves are taken as evi- dence of factorial coding mechanisms. In this paper we show that by focusing on average responses one may miss the existence of factorial coding mechanisms that become only apparent when analyzing spike count histograms.
Morton (1969) analyzed a wide variety of psychophysical experiments on word perception and showed that they could be explained using a model in which stimulus and context have separable effects on perception. More precisely, in Mortons’ model the joint effect of stimulus and context on a perceptual representation can be obtained by multiplying terms
selectively controlled by stimulus and by context, i.e.,             
One-Class LP Classifiers for Dissimilarity Representations
Authors: Elzbieta Pekalska, David M.J. Tax, Robert Duin
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Problems in which abnormal or novel situations should be detected can be approached by describing the domain of the class of typical exam- ples. These applications come from the areas of machine diagnostics, fault detection, illness identiﬁcation or, in principle, refer to any prob- lem where little knowledge is available outside the typical class. In this paper we explain why proximities are natural representations for domain descriptors and we propose a simple one-class classiﬁer for dissimilarity representations. By the use of linear programming an efﬁcient one-class description can be found, based on a small number of prototype objects. This classiﬁer can be made (1) more robust by transforming the dissimi- larities and (2) cheaper to compute by using a reduced representation set. Finally, a comparison to a comparable one-class classiﬁer by Campbell and Bennett is given.
Regularized Greedy Importance Sampling
Authors: Finnegan Southey, Dale Schuurmans, Ali Ghodsi
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Greedy importance sampling is an unbiased estimation technique that re- duces the variance of standard importance sampling by explicitly search- ing for modes in the estimation objective. Previous work has demon- strated the feasibility of implementing this method and proved that the technique is unbiased in both discrete and continuous domains. In this paper we present a reformulation of greedy importance sampling that eliminates the free parameters from the original estimator, and introduces a new regularization strategy that further reduces variance without com- promising unbiasedness. The resulting estimator is shown to be effective for difﬁcult estimation problems arising in Markov random ﬁeld infer- ence. In particular, improvements are achieved over standard MCMC estimators when the distribution has multiple peaked modes.