mtybilly/Instruct2DS huggingface dataset:
Hidden Markov Model of Cortical Synaptic Plasticity: Derivation of the Learning Rule
Authors: Kenneth D. Miller Michael Eisele
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Cortical synaptic plasticity depends on the relative timing of pre- and postsynaptic spikes and also on the temporal pattern of presynaptic spikes and of postsynaptic spikes. We study the hypothesis that cortical synap- tic plasticity does not associate individual spikes, but rather whole ﬁr- ing episodes, and depends only on when these episodes start and how long they last, but as little as possible on the timing of individual spikes. Here we present the mathematical background for such a study. Stan- dard methods from hidden Markov models are used to deﬁne what “ﬁr- ing episodes” are. Estimating the probability of being in such an episode requires not only the knowledge of past spikes, but also of future spikes. We show how to construct a causal learning rule, which depends only on past spikes, but associates pre- and postsynaptic ﬁring episodes as if it also knew future spikes. We also show that this learning rule agrees with some features of synaptic plasticity in superﬁcial layers of rat visual cortex (Froemke and Dan, Nature 416:433, 2002).
Field-Programmable Learning Arrays
Authors: Miguel Figueroa, Chris Diorio, David Hsu Seth Bridges
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper introduces the Field-Programmable Learning Array, a new paradigm for rapid prototyping of learning primitives and machine- learning algorithms in silicon. The FPLA is a mixed-signal counterpart to the all-digital Field-Programmable Gate Array in that it enables rapid prototyping of algorithms in hardware. Unlike the FPGA, the FPLA is targeted directly for machine learning by providing local, parallel, on- line analog learning using ﬂoating-gate MOS synapse transistors. We present a prototype FPLA chip comprising an array of reconﬁgurable computational blocks and local interconnect. We demonstrate the via- bility of this architecture by mapping several learning circuits onto the prototype chip.
Boosting Density Estimation
Authors: Eran Segal Saharon Rosset
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Several authors have suggested viewing boosting as a gradient descent search for a good ﬁt in function space. We apply gradient-based boosting methodology to the unsupervised learning problem of density estimation. We show convergence properties of the algorithm and prove that a strength of weak learnability prop- erty applies to this problem as well. We illustrate the potential of this approach through experiments with boosting Bayesian networks to learn density models.
Support Vector Machines for Multiple-Instance Learning
Authors: Ioannis Tsochantaridis, Thomas Hofmann Stuart Andrews
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This  paper  presents  two  new  formulations  of  multiple-instance  learning as a  maximum margin problem.  The proposed extensions  of the  Support  Vector  Machine  (SVM)  learning  approach lead  to  mixed integer quadratic programs that can be solved  heuristically.  Our generalization of SVMs  makes  a  state-of-the-art classification  technique,  including  non-linear  classification via kernels,  available  to  an  area that  up  to now  has  been  largely  dominated by  special  purpose methods.  We  present  experimental  results  on  a  pharma(cid:173) ceutical data set and on applications in automated image indexing  and document categorization.
Bias-Optimal Incremental Problem Solving
Authors: Jürgen Schmidhuber
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Given is a problem sequence and a probability distribution (the bias) on programs computing solution candidates. We present an optimally fast way of incrementally solving each task in the sequence. Bias shifts are computed by program preﬁxes that modify the distribution on their suf- ﬁxes by reusing successful code for previous tasks (stored in non-modiﬁ- able memory). No tested program gets more runtime than its probability times the total search time. In illustrative experiments, ours becomes the ﬁrst general system to learn a universal solver for arbitrary  disk Tow- ers of Hanoi tasks (minimal solution size  ). It demonstrates the advantages of incremental learning by proﬁting from previously solved, simpler tasks involving samples of a simple context free language.
1 Brief Introduction to Optimal Universal Search
Consider an asymptotically optimal method for tasks with quickly veriﬁable solutions:
     
Rate Distortion Function in the Spin Glass State: A Toy Model
Authors: Masato Okada Tatsuto Murayama
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We applied statistical mechanics to an inverse problem of linear mapping to investigate the physics of optimal lossy compressions. We used the replica symmetry breaking technique with a toy model to demonstrate Shannon’s result. The rate distortion function, which is widely known as the theoretical limit of the compression with a ﬁdelity criterion, is derived. Numerical study shows that sparse constructions of the model provide suboptimal compressions.
Adaptive Nonlinear System Identification with Echo State Networks
Authors: Herbert Jaeger
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Echo state networks  (ESN)  are a  novel approach to recurrent neu(cid:173) ral  network training.  An  ESN  consists  of a  large,  fixed,  recurrent  "reservoir"  network, from  which the desired output is  obtained by  training suitable output connection weights.  Determination of op(cid:173) timal  output  weights  becomes  a  linear,  uniquely  solvable  task  of  MSE  minimization.  This  article  reviews  the  basic  ideas  and  de(cid:173) scribes  an  online  adaptation scheme  based  on the  RLS  algorithm  known  from  adaptive  linear  systems.  As  an  example,  a  10-th or(cid:173) der  NARMA  system  is  adaptively identified.  The known  benefits  of the RLS  algorithms carryover from  linear systems to nonlinear  ones;  specifically,  the  convergence rate and  misadjustment  can be  determined at design  time.
Real Time Voice Processing with Audiovisual Feedback: Toward Autonomous Agents with Perfect Pitch
Authors: Daniel D. Lee, Charles L. Isbell, Yann L. Cun Lawrence K. Saul
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We have implemented a real time front end for detecting voiced speech and estimating its fundamental frequency. The front end performs the signal processing for voice-driven agents that attend to the pitch contours of human speech and provide continuous audiovisual feedback. The al- gorithm we use for pitch tracking has several distinguishing features: it makes no use of FFTs or autocorrelation at the pitch period; it updates the pitch incrementally on a sample-by-sample basis; it avoids peak picking and does not require interpolation in time or frequency to obtain high res- olution estimates; and it works reliably over a four octave range, in real time, without the need for postprocessing to produce smooth contours. The algorithm is based on two simple ideas in neural computation: the introduction of a purposeful nonlinearity, and the error signal of a least squares ﬁt. The pitch tracker is used in two real time multimedia applica- tions: a voice-to-MIDI player that synthesizes electronic music from vo- calized melodies, and an audiovisual Karaoke machine with multimodal feedback. Both applications run on a laptop and display the user’s pitch scrolling across the screen as he or she sings into the computer.
An Impossibility Theorem for Clustering
Authors: Jon M. Kleinberg
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Although the study of clustering is centered around an intuitively compelling goal, it has been very di(cid:14)cult to develop a uni(cid:12)ed framework for reasoning about it at a technical level, and pro- foundly diverse approaches to clustering abound in the research community. Here we suggest a formal perspective on the di(cid:14)culty in (cid:12)nding such a uni(cid:12)cation, in the form of an impossibility theo- rem: for a set of three simple properties, we show that there is no clustering function satisfying all three. Relaxations of these prop- erties expose some of the interesting (and unavoidable) trade-o(cid:11)s at work in well-studied clustering techniques such as single-linkage, sum-of-pairs, k-means, and k-median.
Visual Development Aids the Acquisition of Motion Velocity Sensitivities
Authors: Melissa Dominguez Robert A. Jacobs
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider the hypothesis that systems learning aspects of visual per- ception may beneﬁt from the use of suitably designed developmental pro- gressions during training. Four models were trained to estimate motion velocities in sequences of visual images. Three of the models were “de- velopmental models” in the sense that the nature of their input changed during the course of training. They received a relatively impoverished visual input early in training, and the quality of this input improved as training progressed. One model used a coarse-to-multiscale develop- mental progression (i.e. it received coarse-scale motion features early in training and ﬁner-scale features were added to its input as training progressed), another model used a ﬁne-to-multiscale progression, and the third model used a random progression. The ﬁnal model was non- developmental in the sense that the nature of its input remained the same throughout the training period. The simulation results show that the coarse-to-multiscale model performed best. Hypotheses are offered to account for this model’s superior performance. We conclude that suit- ably designed developmental sequences can be useful to systems learn- ing to estimate motion velocities. The idea that visual development can aid visual learning is a viable hypothesis in need of further study.
On the Complexity of Learning the Kernel Matrix
Authors: Daniel Herrmann Olivier Bousquet
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate data based procedures for selecting the kernel when learn- ing with Support Vector Machines. We provide generalization error bounds by estimating the Rademacher complexities of the corresponding function classes. In particular we obtain a complexity bound for function classes induced by kernels with given eigenvectors, i.e., we allow to vary the spectrum and keep the eigenvectors ﬁx. This bound is only a loga- rithmic factor bigger than the complexity of the function class induced by a single kernel. However, optimizing the margin over such classes leads to overﬁtting. We thus propose a suitable way of constraining the class. We use an efﬁcient algorithm to solve the resulting optimization problem, present preliminary experimental results, and compare them to an alignment-based approach.
Clustering with the Fisher Score
Authors: Motoaki Kawanabe, Klaus-Robert Müller Koji Tsuda
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recently the Fisher score (or the Fisher kernel) is increasingly used as a feature extractor for classiﬁcation problems. The Fisher score is a vector of parameter derivatives of loglikelihood of a probabilistic model. This paper gives a theoretical analysis about how class information is pre- served in the space of the Fisher score, which turns out that the Fisher score consists of a few important dimensions with class information and many nuisance dimensions. When we perform clustering with the Fisher score, K-Means type methods are obviously inappropriate because they make use of all dimensions. So we will develop a novel but simple clus- tering algorithm specialized for the Fisher score, which can exploit im- portant dimensions. This algorithm is successfully tested in experiments with artiﬁcial data and real data (amino acid sequences).
as follows: Let us assume that a probabilistic model     
parameter estimate 
Artefactual Structure from Least-Squares Multidimensional Scaling
Authors: David Lowe Nicholas P. Hughes
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider the problem of illusory or artefactual structure from the vi- sualisation of high-dimensional structureless data. In particular we ex- amine the role of the distance metric in the use of topographic mappings based on the statistical ﬁeld of multidimensional scaling. We show that the use of a squared Euclidean metric (i.e. the SS TRESS measure) gives rise to an annular structure when the input data is drawn from a high- dimensional isotropic distribution, and we provide a theoretical justiﬁca- tion for this observation.
Adaptation and Unsupervised Learning
Authors: Maneesh Sahani, Gregoire Deback Peter Dayan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Adaptation is a ubiquitous neural and psychological phenomenon, with a wealth of instantiations and implications. Although a basic form of plasticity, it has, bar some notable exceptions, attracted computational theory of only one main variety. In this paper, we study adaptation from the perspective of factor analysis, a paradigmatic technique of unsuper- vised learning. We use factor analysis to re-interpret a standard view of adaptation, and apply our new model to some recent data on adaptation in the domain of face discrimination.
Learning in Zero-Sum Team Markov Games Using Factored Value Functions
Authors: Ronald Parr Michail G. Lagoudakis
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a new method for learning good strategies in zero-sum Markov games in which each side is composed of multiple agents col- laborating against an opposing team of agents. Our method requires full observability and communication during learning, but the learned poli- cies can be executed in a distributed manner. The value function is rep- resented as a factored linear architecture and its structure determines the necessary computational resources and communication bandwidth. This approach permits a tradeoff between simple representations with little or no communication between agents and complex, computationally inten- sive representations with extensive coordination between agents. Thus, we provide a principled means of using approximation to combat the exponential blowup in the joint action space of the participants. The ap- proach is demonstrated with an example that shows the efﬁciency gains over naive enumeration.
Convergence Properties of Some Spike-Triggered Analysis Techniques
Authors: Liam Paninski
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: vVe analyze the convergence properties of three spike-triggered data analysis techniques. All of our results are obtained in the set(cid:173) ting of a (possibly multidimensional) linear-nonlinear (LN) cascade model for stimulus-driven neural activity. We start by giving exact rate of convergence results for the common spike-triggered average (STA) technique. Next, we analyze a spike-triggered covariance method, variants of which have been recently exploited successfully by Bialek, Simoncelli, and colleagues. These first two methods suf(cid:173) fer from extraneous conditions on their convergence; therefore, we introduce an estimator for the LN model parameters which is de(cid:173) signed to be consistent under general conditions. We provide an algorithm for the computation of this estimator and derive its rate of convergence. We close with a brief discussion of the efficiency of these estimators and an application to data recorded from the primary motor cortex of awake, behaving primates.
Learning to Take Concurrent Actions
Authors: Sridhar Mahadevan Khashayar Rohanimanesh
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate a general semi-Markov Decision Process (SMDP) framework for modeling concurrent decision making, where agents learn optimal plans over concurrent temporally extended actions. We introduce three types of parallel termination schemes { all, any and continue { and theoretically and experimentally compare them.
Handling Missing Data with Variational Bayesian Learning of ICA
Authors: Te-Won Lee, Terrence J. Sejnowski Kwokleung Chan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Missing data is common in real-world datasets and is a problem for many estimation techniques. We have developed a variational Bayesian method to perform Independent Component Analysis (ICA) on high-dimensional data containing missing entries. Missing data are handled naturally in the Bayesian framework by integrating the generative density model. Mod- eling the distributions of the independent sources with mixture of Gaus- sians allows sources to be estimated with different kurtosis and skewness. The variational Bayesian method automatically determines the dimen- sionality of the data and yields an accurate density model for the ob- served data without overﬁtting problems. This allows direct probability estimation of missing values in the high dimensional space and avoids dimension reduction preprocessing which is not feasible with missing data.
Ranking with Large Margin Principle: Two Approaches
Authors: Anat Levin Amnon Shashua
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We discuss the problem of ranking k instances with the use of a "large  margin" principle. We introduce two main approaches: the first is the  "fixed margin" policy in which the margin of the closest neighboring  classes is being maximized - which turns out to be a direct generaliza(cid:173) tion of SVM to ranking learning. The second approach allows for k - 1  different margins where the sum of margins is maximized. This approach  is shown to reduce to lI-SVM when the number of classes k = 2. Both  approaches are optimal in size of 21 where I is the total number of training  examples. Experiments performed on visual classification and "collab(cid:173) orative filtering" show that both approaches outperform existing ordinal  regression algorithms applied for ranking and multi-class SVM applied  to general multi-class classification.
A Bilinear Model for Sparse Coding
Authors: Rajesh P. N. Rao David B. Grimes
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recent algorithms for sparse coding and independent component analy- sis (ICA) have demonstrated how localized features can be learned from natural images. However, these approaches do not take image transfor- mations into account. As a result, they produce image codes that are redundant because the same feature is learned at multiple locations. We describe an algorithm for sparse coding based on a bilinear generative model of images. By explicitly modeling the interaction between im- age features and their transformations, the bilinear approach helps reduce redundancy in the image code and provides a basis for transformation- invariant vision. We present results demonstrating bilinear sparse coding of natural images. We also explore an extension of the model that can capture spatial relationships between the independent features of an ob- ject, thereby providing a new framework for parts-based object recogni- tion.
Information Diffusion Kernels
Authors: John D. Lafferty Guy Lebanon
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A new family of kernels for statistical learning is introduced that ex- ploits the geometric structure of statistical models. Based on the heat equation on the Riemannian manifold deﬁned by the Fisher informa- tion metric, information diffusion kernels generalize the Gaussian kernel of Euclidean space, and provide a natural way of combining generative statistical modeling with non-parametric discriminative learning. As a special case, the kernels give a new approach to applying kernel-based learning algorithms to discrete data. Bounds on covering numbers for the new kernels are proved using spectral theory in differential geometry, and experimental results are presented for text classiﬁcation.
Boosted Dyadic Kernel Discriminants
Authors: Gregory Shakhnarovich Baback Moghaddam
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We introduce a novel learning algorithm for binary classi(cid:12)cation with hyperplane discriminants based on pairs of training points from opposite classes (dyadic hypercuts). This algorithm is further extended to nonlinear discriminants using kernel functions satisfy- ing Mercer’s conditions. An ensemble of simple dyadic hypercuts is learned incrementally by means of a con(cid:12)dence-rated version of Ad- aBoost, which provides a sound strategy for searching through the (cid:12)nite set of hypercut hypotheses. In experiments with real-world datasets from the UCI repository, the generalization performance of the hypercut classi(cid:12)ers was found to be comparable to that of SVMs and k-NN classi(cid:12)ers. Furthermore, the computational cost of classi(cid:12)cation (at run time) was found to be similar to, or bet- ter than, that of SVM. Similarly to SVMs, boosted dyadic kernel discriminants tend to maximize the margin (via AdaBoost). In contrast to SVMs, however, we o(cid:11)er an on-line and incremental learning machine for building kernel discriminants whose complex- ity (number of kernel evaluations) can be directly controlled (traded o(cid:11) for accuracy).
Derivative Observations in Gaussian Process Models of Dynamic Systems
Authors: R. Murray-smith, W. E. Leithead, D. J. Leith, Carl E. Rasmussen E. Solak
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Gaussian processes provide an approach to nonparametric modelling which allows a straightforward combination of function and derivative observations in an empirical model. This is of particular importance in identiﬁcation of nonlinear dynamic systems from experimental data. 1) It allows us to combine derivative information, and associated uncertainty with normal function observations into the learning and inference pro- cess. This derivative information can be in the form of priors speciﬁed by an expert or identiﬁed from perturbation data close to equilibrium. 2) It allows a seamless fusion of multiple local linear models in a consis- tent manner, inferring consistent models and ensuring that integrability constraints are met. 3) It improves dramatically the computational ef- ﬁciency of Gaussian process models for dynamic system identiﬁcation, by summarising large quantities of near-equilibrium data by a handful of linearisations, reducing the training set size – traditionally a problem for Gaussian process models.
Global Versus Local Methods in Nonlinear Dimensionality Reduction
Authors: Joshua B. Tenenbaum Vin D. Silva
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recently proposed algorithms for nonlinear dimensionality reduction fall broadly into two categories which have different advantages and disad- vantages: global (Isomap [1]), and local (Locally Linear Embedding [2], Laplacian Eigenmaps [3]). We present two variants of Isomap which combine the advantages of the global approach with what have previ- ously been exclusive advantages of local methods: computational spar- sity and the ability to invert conformal maps.
Learning Graphical Models with Mercer Kernels
Authors: Michael I. Jordan Francis R. Bach
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a class of algorithms for learning the structure of graphical models from data. The algorithms are based on a measure known as the kernel generalized variance (KGV), which essentially allows us to treat all variables on an equal footing as Gaussians in a feature space obtained from Mercer kernels. Thus we are able to learn hybrid graphs involving discrete and continuous variables of arbitrary type. We explore the computational properties of our approach, showing how to use the kernel trick to compute the relevant statistics in linear time. We illustrate our framework with experiments involving discrete and continuous data.
Stochastic Neighbor Embedding
Authors: Sam T. Roweis Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We describe a probabilistic approach to the task of placing objects, de- scribed by high-dimensional vectors or by pairwise dissimilarities, in a low-dimensional space in a way that preserves neighbor identities. A Gaussian is centered on each object in the high-dimensional space and the densities under this Gaussian (or the given dissimilarities) are used to deﬁne a probability distribution over all the potential neighbors of the object. The aim of the embedding is to approximate this distribu- tion as well as possible when the same operation is performed on the low-dimensional “images” of the objects. A natural cost function is a sum of Kullback-Leibler divergences, one per object, which leads to a simple gradient for adjusting the positions of the low-dimensional im- ages. Unlike other dimensionality reduction methods, this probabilistic framework makes it easy to represent each object by a mixture of widely separated low-dimensional images. This allows ambiguous objects, like the document count vector for the word “bank”, to have versions close to the images of both “river” and “ﬁnance” without forcing the images of outdoor concepts to be located close to those of corporate concepts.
Learning in Spiking Neural Assemblies
Authors: David Barber
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider a statistical framework for learning in a class of net- works of spiking neurons. Our aim is to show how optimal local learning rules can be readily derived once the neural dynamics and desired functionality of the neural assembly have been speciﬂed, in contrast to other models which assume (sub-optimal) learning rules. Within this framework we derive local rules for learning tem- poral sequences in a model of spiking neurons and demonstrate its superior performance to correlation (Hebbian) based approaches. We further show how to include mechanisms such as synaptic de- pression and outline how the framework is readily extensible to learning in networks of highly complex spiking neurons. A stochas- tic quantal vesicle release mechanism is considered and implications on the complexity of learning discussed.
A Model for Real-Time Computation in Generic Neural Microcircuits
Authors: Thomas Natschläger, Henry Markram Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Henry Markram Brain Mind Institute
EPFL, Lausanne, Switzerland
henry.markram@epﬂ.ch
A key challenge for neural modeling is to explain how a continuous stream of multi-modal input from a rapidly changing environment can be processed by stereotypical recurrent circuits of integrate-and-ﬁre neurons in real-time. We propose a new computational model that is based on principles of high dimensional dynamical systems in combination with statistical learning theory. It can be implemented on generic evolved or found recurrent circuitry.
Categorization Under Complexity: A Unified MDL Account of Human Learning of Regular and Irregular Categories
Authors: Jacob Feldman David Fass
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present an account of human concept learning-that is, learning of categories from examples-based on the principle of minimum descrip(cid:173) tion length (MDL). In support of this theory, we tested a wide range of two-dimensional concept types, including both regular (simple) and highly irregular (complex) structures, and found the MDL theory to give a good account of subjects' performance. This suggests that the intrin(cid:173) sic complexity of a concept (that is, its description -length) systematically influences its leamability.
1- The Structure of Categories
A number of different principles have been advanced to explain the manner in which hu(cid:173) mans learn to categorize objects. It has been variously suggested that the underlying prin(cid:173) ciple might be the similarity structure of objects [1], the manipulability of decision bound~ aries [2], or Bayesian inference [3][4]. While many of these theories are mathematically well-grounded and have been successful in explaining a range of experimental findings, they have commonly only been tested on a narrow collection of concept types similar to the simple unimodal categories of Figure 1(a-e).
Learning with Multiple Labels
Authors: Zoubin Ghahramani Rong Jin
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper,  we  study a  special kind of learning problem in which  each  training  instance  is  given  a  set  of  (or  distribution  over)  candidate  class  labels  and  only  one  of the  candidate  labels  is  the  correct  one.  Such  a  problem  can  occur,  e.g.,  in  an  information  retrieval  setting  where  a  set  of words  is  associated  with  an  image,  or  if  classes  labels  are  organized  hierarchically.  We  propose  a  novel  discriminative  approach  for  handling  the  ambiguity  of class  labels  in the  training  examples.  The  experiments with the  proposed  approach over five  different UCI datasets  show that our approach is  able  to  find  the  correct label  among the  set of candidate  labels  and  actually  achieve  performance  close  to  the  case  when  each  training  instance  is  given  a  single  correct  label.  In  contrast,  naIve  methods  degrade rapidly as  more ambiguity is introduced into the labels.
Branching Law for Axons
Authors: Armen Stepanyants Dmitri B. Chklovskii
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: What  determines  the  caliber  of  axonal  branches?  We  pursue  the  hypothesis  that  the  axonal  caliber  has  evolved  to  minimize  signal  propagation delays, while keeping arbor volume to  a  minimum.  We  show  that  for  a  general  cost  function  the  optimal  diameters  of  mother  (do)  and  daughter  (d], d2 )  branches at a bifurcation obey  h  a  ranc  mg  aw:  e  envatIOn  re les  on  t  e  fact  that the conduction  speed  scales  with  the  axon  diameter to  the  power  V  (v = 1  for  myelinated  axons  and  V  = 0.5  myelinated  axons).  We  test  the  branching  law  on  the  available  experimental data and find  a reasonable agreement. 
d v+2  d v+2  d v+2  Th  d "  0  = ]   +  2
Dyadic Classification Trees via Structural Risk Minimization
Authors: Robert Nowak Clayton Scott
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Classiﬁcation trees are one of the most popular types of classiﬁers, with ease of implementation and interpretation being among their attractive features. Despite the widespread use of classiﬁcation trees, theoretical analysis of their performance is scarce. In this paper, we show that a new family of classiﬁcation trees, called dyadic classiﬁcation trees (DCTs), are near optimal (in a minimax sense) for a very broad range of clas- siﬁcation problems. This demonstrates that other schemes (e.g., neural networks, support vector machines) cannot perform signiﬁcantly better than DCTs in many cases. We also show that this near optimal perfor- mance is attained with linear (in the number of training data) complexity growing and pruning algorithms. Moreover, the performance of DCTs on benchmark datasets compares favorably to that of standard CART, which is generally more computationally intensive and which does not possess similar near optimality properties. Our analysis stems from the- oretical results on structural risk minimization, on which the pruning rule for DCTs is based.
Dynamical Causal Learning
Authors: Thomas L. Griffiths, Joshua B. Tenenbaum David Danks
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: theories of human causal
focus primarily on long-run predictions:
learning and Current psychological two by judgment estimating parameters of a causal Bayes nets (though for different parameterizations), and a third through structural learning. This short-run behavior by examining paper dynamical versions of these three theories, and comparing their predictions to a real-world dataset.
focuses on people's
Fast Exact Inference with a Factored Model for Natural Language Parsing
Authors: Christopher D. Manning Dan Klein
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a novel generative model for natural language tree structures in which semantic (lexical dependency) and syntactic (PCFG) structures are scored with separate models. This factorization provides concep- tual simplicity, straightforward opportunities for separately improving the component models, and a level of performance comparable to simi- lar, non-factored models. Most importantly, unlike other modern parsing models, the factored model admits an extremely effective A* parsing al- gorithm, which enables efﬁcient, exact inference.
Morton-Style Factorial Coding of Color in Primary Visual Cortex
Authors: Thomas Wachtler, Thomas D. Albright, Terrence Sejnowski Javier R. Movellan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We introduce the notion of Morton-style factorial coding and illustrate how it may help understand information integration and perceptual cod- ing in the brain. We show that by focusing on average responses one may miss the existence of factorial coding mechanisms that become only apparent when analyzing spike count histograms. We show evidence suggesting that the classical/non-classical receptive ﬁeld organization in the cortex effectively enforces the development of Morton-style factorial codes. This may provide some cues to help understand perceptual cod- ing in the brain and to develop new unsupervised learning algorithms. While methods like ICA (Bell & Sejnowski, 1997) develop independent codes, in Morton-style coding the goal is to make two or more external aspects of the world become independent when conditioning on internal representations.
In this paper we introduce the notion of Morton-style factorial coding and illustrate how it may help analyze information integration and perceptual organization in the brain. In the neurosciences factorial codes are often studied in the context of mean tuning curves. A tuning curve is called separable if it can be expressed as the product of terms selectively inﬂuenced by different stimulus dimensions. Separable tuning curves are taken as evi- dence of factorial coding mechanisms. In this paper we show that by focusing on average responses one may miss the existence of factorial coding mechanisms that become only apparent when analyzing spike count histograms.
Morton (1969) analyzed a wide variety of psychophysical experiments on word perception and showed that they could be explained using a model in which stimulus and context have separable effects on perception. More precisely, in Mortons’ model the joint effect of stimulus and context on a perceptual representation can be obtained by multiplying terms
selectively controlled by stimulus and by context, i.e.,             
One-Class LP Classifiers for Dissimilarity Representations
Authors: David M.J. Tax, Robert Duin Elzbieta Pekalska
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Problems in which abnormal or novel situations should be detected can be approached by describing the domain of the class of typical exam- ples. These applications come from the areas of machine diagnostics, fault detection, illness identiﬁcation or, in principle, refer to any prob- lem where little knowledge is available outside the typical class. In this paper we explain why proximities are natural representations for domain descriptors and we propose a simple one-class classiﬁer for dissimilarity representations. By the use of linear programming an efﬁcient one-class description can be found, based on a small number of prototype objects. This classiﬁer can be made (1) more robust by transforming the dissimi- larities and (2) cheaper to compute by using a reduced representation set. Finally, a comparison to a comparable one-class classiﬁer by Campbell and Bennett is given.
Regularized Greedy Importance Sampling
Authors: Dale Schuurmans, Ali Ghodsi Finnegan Southey
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Greedy importance sampling is an unbiased estimation technique that re- duces the variance of standard importance sampling by explicitly search- ing for modes in the estimation objective. Previous work has demon- strated the feasibility of implementing this method and proved that the technique is unbiased in both discrete and continuous domains. In this paper we present a reformulation of greedy importance sampling that eliminates the free parameters from the original estimator, and introduces a new regularization strategy that further reduces variance without com- promising unbiasedness. The resulting estimator is shown to be effective for difﬁcult estimation problems arising in Markov random ﬁeld infer- ence. In particular, improvements are achieved over standard MCMC estimators when the distribution has multiple peaked modes.
Fast Kernels for String and Tree Matching
Authors: S.v.n. Vishwanathan Alex J. Smola
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper we present a new algorithm suitable for matching discrete  objects such as strings and trees in linear time, thus obviating dynarrtic  programming with quadratic time complexity. Furthermore, prediction  cost in many cases can be reduced to linear cost in the length of the se(cid:173) quence to be classified, regardless of the number of support vectors. This  improvement on the currently available algorithms makes string kernels  a viable alternative for the practitioner.
Expected and Unexpected Uncertainty: ACh and NE in the Neocortex
Authors: Angela J. Yu Peter Dayan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Inference and adaptation in noisy and changing, rich sensory environ- ments are rife with a variety of speciﬁc sorts of variability. Experimental and theoretical studies suggest that these different forms of variability play different behavioral, neural and computational roles, and may be reported by different (notably neuromodulatory) systems. Here, we re- ﬁne our previous theory of acetylcholine’s role in cortical inference in the (oxymoronic) terms of expected uncertainty, and advocate a theory for norepinephrine in terms of unexpected uncertainty. We suggest that norepinephrine reports the radical divergence of bottom-up inputs from prevailing top-down interpretations, to inﬂuence inference and plasticity. We illustrate this proposal using an adaptive factor analysis model.
Feature Selection and Classification on Matrix Data: From Large Margins to Small Covering Numbers
Authors: Klaus Obermayer Sepp Hochreiter
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate the problem of learning a classiﬂcation task for datasets which are described by matrices. Rows and columns of these matrices correspond to objects, where row and column ob- jects may belong to diﬁerent sets, and the entries in the matrix express the relationships between them. We interpret the matrix el- ements as being produced by an unknown kernel which operates on object pairs and we show that - under mild assumptions - these ker- nels correspond to dot products in some (unknown) feature space. Minimizing a bound for the generalization error of a linear classi- ﬂer which has been obtained using covering numbers we derive an objective function for model selection according to the principle of structural risk minimization. The new objective function has the advantage that it allows the analysis of matrices which are not pos- itive deﬂnite, and not even symmetric or square. We then consider the case that row objects are interpreted as features. We suggest an additional constraint, which imposes sparseness on the row objects and show, that the method can then be used for feature selection. Finally, we apply this method to data obtained from DNA microar- rays, where \column" objects correspond to samples, \row" objects correspond to genes and matrix elements correspond to expression levels. Benchmarks are conducted using standard one-gene classiﬂ- cation and support vector machines and K-nearest neighbors after standard feature selection. Our new method extracts a sparse set of genes and provides superior classiﬂcation results.
Learning Semantic Similarity
Authors: Nello Cristianini, John S. Shawe-taylor Jaz Kandola
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The standard representation of text documents as bags of words  suffers from well known limitations, mostly due to its inability to  exploit semantic similarity between terms. Attempts to incorpo(cid:173) rate some notion of term similarity include latent semantic index(cid:173) ing [8], the use of semantic networks [9], and probabilistic methods  [5]. In this paper we propose two methods for inferring such sim(cid:173) ilarity from a corpus. The first one defines word-similarity based  on document-similarity and viceversa, giving rise to a system of  equations whose equilibrium point we use to obtain a semantic  similarity measure. The second method models semantic relations  by means of a diffusion process on a graph defined by lexicon and  co-occurrence information. Both approaches produce valid kernel  functions parametrised by a real number. The paper shows how  the alignment measure can be used to successfully perform model  selection over this parameter. Combined with the use of support  vector machines we obtain positive results.
How Linear are Auditory Cortical Responses?
Authors: Jennifer F. Linden Maneesh Sahani
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: By comparison to some other sensory cortices, the functional proper- ties of cells in the primary auditory cortex are not yet well understood. Recent attempts to obtain a generalized description of auditory cortical responses have often relied upon characterization of the spectrotempo- ral receptive ﬁeld (STRF), which amounts to a model of the stimulus- response function (SRF) that is linear in the spectrogram of the stimulus. How well can such a model account for neural responses at the very ﬁrst stages of auditory cortical processing? To answer this question, we de- velop a novel methodology for evaluating the fraction of stimulus-related response power in a population that can be captured by a given type of SRF model. We use this technique to show that, in the thalamo-recipient layers of primary auditory cortex, STRF models account for no more than 40% of the stimulus-related power in neural responses.
Incremental Gaussian Processes
Authors: Ole Winther Joaquin Quiñonero-candela
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper, we consider Tipping’s relevance vector machine (RVM) [1] and formalize an incremental training strategy as a variant of the expectation-maximization (EM) algorithm that we call Subspace EM (SSEM). Working with a subset of active basis functions, the sparsity of the RVM solution will ensure that the number of basis functions and thereby the computational complexity is kept low. We also introduce a mean ﬁeld approach to the intractable classiﬁcation model that is ex- pected to give a very good approximation to exact Bayesian inference and contains the Laplace approximation as a special case. We test the algorithms on two large data sets with O(103 (cid:0) 104) examples. The re- sults indicate that Bayesian learning of large data sets, e.g. the MNIST database is realistic.
Dynamic Bayesian Networks with Deterministic Latent Tables
Authors: David Barber
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The application of latent/hidden variable Dynamic Bayesian Net- works is constrained by the complexity of marginalising over latent variables. For this reason either small latent dimensions or Gaus- sian latent conditional tables linearly dependent on past states are typically considered in order that inference is tractable. We suggest an alternative approach in which the latent variables are modelled using deterministic conditional probability tables. This specialisa- tion has the advantage of tractable inference even for highly com- plex non-linear/non-Gaussian visible conditional probability tables. This approach enables the consideration of highly complex latent dynamics whilst retaining the bene(cid:12)ts of a tractable probabilistic model.
Generalized² Linear² Models
Authors: Geoffrey J. Gordon
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We  introduce the Generalized2  Linear2  Model, a statistical estima(cid:173) tor which combines features of nonlinear regression and factor anal(cid:173) ysis.  A  (GL)2M  approximately  decomposes  a  rectangular  matrix  X  into  a  simpler  representation  j(g(A)h(B)).  Here  A  and  Bare  low-rank matrices,  while  j,  g,  and  h  are  link  functions.  (GL)2Ms  include  many  useful  models  as  special  cases,  including  principal  components analysis,  exponential-family peA, the infomax formu(cid:173) lation  of independent  components  analysis,  linear  regression,  and  generalized  linear  models.  They  also  include  new  and  interesting  special  cases,  one  of which  we  describe  below.  We  also  present  an  iterative  procedure  which  optimizes  the  parameters of a  (GL)2M.  This  procedure  reduces  to  well-known  algorithms  for  some  of the  special cases listed above;  for  other special cases,  it  is  new.