mtybilly/Instruct2DS huggingface dataset:
Fast Kernels for String and Tree Matching
Authors: Alex J. Smola, S.v.n. Vishwanathan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper we present a new algorithm suitable for matching discrete  objects such as strings and trees in linear time, thus obviating dynarrtic  programming with quadratic time complexity. Furthermore, prediction  cost in many cases can be reduced to linear cost in the length of the se(cid:173) quence to be classified, regardless of the number of support vectors. This  improvement on the currently available algorithms makes string kernels  a viable alternative for the practitioner.
Expected and Unexpected Uncertainty: ACh and NE in the Neocortex
Authors: Peter Dayan, Angela J. Yu
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Inference and adaptation in noisy and changing, rich sensory environ- ments are rife with a variety of speciﬁc sorts of variability. Experimental and theoretical studies suggest that these different forms of variability play different behavioral, neural and computational roles, and may be reported by different (notably neuromodulatory) systems. Here, we re- ﬁne our previous theory of acetylcholine’s role in cortical inference in the (oxymoronic) terms of expected uncertainty, and advocate a theory for norepinephrine in terms of unexpected uncertainty. We suggest that norepinephrine reports the radical divergence of bottom-up inputs from prevailing top-down interpretations, to inﬂuence inference and plasticity. We illustrate this proposal using an adaptive factor analysis model.
Feature Selection and Classification on Matrix Data: From Large Margins to Small Covering Numbers
Authors: Sepp Hochreiter, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We investigate the problem of learning a classiﬂcation task for datasets which are described by matrices. Rows and columns of these matrices correspond to objects, where row and column ob- jects may belong to diﬁerent sets, and the entries in the matrix express the relationships between them. We interpret the matrix el- ements as being produced by an unknown kernel which operates on object pairs and we show that - under mild assumptions - these ker- nels correspond to dot products in some (unknown) feature space. Minimizing a bound for the generalization error of a linear classi- ﬂer which has been obtained using covering numbers we derive an objective function for model selection according to the principle of structural risk minimization. The new objective function has the advantage that it allows the analysis of matrices which are not pos- itive deﬂnite, and not even symmetric or square. We then consider the case that row objects are interpreted as features. We suggest an additional constraint, which imposes sparseness on the row objects and show, that the method can then be used for feature selection. Finally, we apply this method to data obtained from DNA microar- rays, where \column" objects correspond to samples, \row" objects correspond to genes and matrix elements correspond to expression levels. Benchmarks are conducted using standard one-gene classiﬂ- cation and support vector machines and K-nearest neighbors after standard feature selection. Our new method extracts a sparse set of genes and provides superior classiﬂcation results.
Learning Semantic Similarity
Authors: Jaz Kandola, Nello Cristianini, John S. Shawe-taylor
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The standard representation of text documents as bags of words  suffers from well known limitations, mostly due to its inability to  exploit semantic similarity between terms. Attempts to incorpo(cid:173) rate some notion of term similarity include latent semantic index(cid:173) ing [8], the use of semantic networks [9], and probabilistic methods  [5]. In this paper we propose two methods for inferring such sim(cid:173) ilarity from a corpus. The first one defines word-similarity based  on document-similarity and viceversa, giving rise to a system of  equations whose equilibrium point we use to obtain a semantic  similarity measure. The second method models semantic relations  by means of a diffusion process on a graph defined by lexicon and  co-occurrence information. Both approaches produce valid kernel  functions parametrised by a real number. The paper shows how  the alignment measure can be used to successfully perform model  selection over this parameter. Combined with the use of support  vector machines we obtain positive results.
How Linear are Auditory Cortical Responses?
Authors: Maneesh Sahani, Jennifer F. Linden
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: By comparison to some other sensory cortices, the functional proper- ties of cells in the primary auditory cortex are not yet well understood. Recent attempts to obtain a generalized description of auditory cortical responses have often relied upon characterization of the spectrotempo- ral receptive ﬁeld (STRF), which amounts to a model of the stimulus- response function (SRF) that is linear in the spectrogram of the stimulus. How well can such a model account for neural responses at the very ﬁrst stages of auditory cortical processing? To answer this question, we de- velop a novel methodology for evaluating the fraction of stimulus-related response power in a population that can be captured by a given type of SRF model. We use this technique to show that, in the thalamo-recipient layers of primary auditory cortex, STRF models account for no more than 40% of the stimulus-related power in neural responses.
Incremental Gaussian Processes
Authors: Joaquin Quiñonero-candela, Ole Winther
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In this paper, we consider Tipping’s relevance vector machine (RVM) [1] and formalize an incremental training strategy as a variant of the expectation-maximization (EM) algorithm that we call Subspace EM (SSEM). Working with a subset of active basis functions, the sparsity of the RVM solution will ensure that the number of basis functions and thereby the computational complexity is kept low. We also introduce a mean ﬁeld approach to the intractable classiﬁcation model that is ex- pected to give a very good approximation to exact Bayesian inference and contains the Laplace approximation as a special case. We test the algorithms on two large data sets with O(103 (cid:0) 104) examples. The re- sults indicate that Bayesian learning of large data sets, e.g. the MNIST database is realistic.
Spectro-Temporal Receptive Fields of Subthreshold Responses in Auditory Cortex
Authors: Christian K. Machens, Michael Wehr, Anthony M. Zador
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: How do cortical neurons represent the acoustic environment? This ques- tion is often addressed by probing with simple stimuli such as clicks or tone pips. Such stimuli have the advantage of yielding easily interpreted answers, but have the disadvantage that they may fail to uncover complex or higher-order neuronal response properties. Here we adopt an alternative approach, probing neuronal responses with complex acoustic stimuli, including animal vocalizations and music. We have used in vivo whole cell methods in the rat auditory cortex to record subthreshold membrane potential ﬂuctuations elicited by these stimuli. Whole cell recording reveals the total synaptic input to a neuron from all the other neurons in the circuit, instead of just its output—a sparse bi- nary spike train—as in conventional single unit physiological recordings. Whole cell recording thus provides a much richer source of information about the neuron’s response. Many neurons responded robustly and reliably to the complex stimuli in our ensemble. Here we analyze the linear component—the spectro- temporal receptive ﬁeld (STRF)—of the transformation from the sound (as represented by its time-varying spectrogram) to the neuron’s mem- brane potential. We ﬁnd that the STRF has a rich dynamical structure, including excitatory regions positioned in general accord with the predic- tion of the simple tuning curve. We also ﬁnd that in many cases, much of the neuron’s response, although deterministically related to the stimulus, cannot be predicted by the linear component, indicating the presence of as-yet-uncharacterized nonlinear response properties.
Bayesian Models of Inductive Generalization
Authors: Neville E. Sanjana, Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We argue that human inductive generalization is best explained in a Bayesian framework, rather than by traditional models based on simi- larity computations. We go beyond previous work on Bayesian concept learning by introducing an unsupervised method for constructing ﬂex- ible hypothesis spaces, and we propose a version of the Bayesian Oc- cam’s razor that trades off priors and likelihoods to prevent under- or over-generalization in these ﬂexible spaces. We analyze two published data sets on inductive reasoning as well as the results of a new behavioral study that we have carried out.
Dynamical Constraints on Computing with Spike Timing in the Cortex
Authors: Arunava Banerjee, Alexandre Pouget
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: If the  cortex uses  spike timing to  compute,  the  timing of the  spikes  must  be  robust to  perturbations.  Based on  a  recent  framework  that  provides  a  simple  criterion  to  determine  whether  a  spike  sequence  produced by a generic network is  sensitive to  initial conditions, and  numerical  simulations  of  a  variety  of  network  architectures,  we  argue  within  the  limits  set  by  our  model  of the  neuron,  that  it  is  unlikely  that  precise  sequences  of  spike  timings  are  used  for  computation under conditions typically found  in the cortex.
Bayesian Estimation of Time-Frequency Coefficients for Audio Signal Enhancement
Authors: Patrick J. Wolfe, Simon J. Godsill
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The Bayesian paradigm provides a natural and effective means of exploit- ing prior knowledge concerning the time-frequency structure of sound signals such as speech and music—something which has often been over- looked in traditional audio signal processing approaches. Here, after con- structing a Bayesian model and prior distributions capable of taking into account the time-frequency characteristics of typical audio waveforms, we apply Markov chain Monte Carlo methods in order to sample from the resultant posterior distribution of interest. We present speech enhance- ment results which compare favourably in objective terms with standard time-varying ﬁltering techniques (and in several cases yield superior per- formance, both objectively and subjectively); moreover, in contrast to such methods, our results are obtained without an assumption of prior knowledge of the noise power.
Real-Time Monitoring of Complex Industrial Processes with Particle Filters
Authors: Rubén Morales-Menéndez, Nando de Freitas, David Poole
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper discusses the application of particle ﬁltering algorithms to fault diagnosis in complex industrial processes. We consider two ubiq- uitous processes: an industrial dryer and a level tank. For these appli- cations, we compared three particle ﬁltering variants: standard parti- cle ﬁltering, Rao-Blackwellised particle ﬁltering and a version of Rao- Blackwellised particle ﬁltering that does one-step look-ahead to select good sampling regions. We show that the overhead of the extra process- ing per particle of the more sophisticated methods is more than compen- sated by the decrease in error and variance.
Retinal Processing Emulation in a Programmable 2-Layer Analog Array Processor CMOS Chip
Authors: R. Carmona, F. Jiménez-garrido, R. Dominguez-castro, S. Espejo, A. Rodriguez-vázquez
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A bio-inspired model for an analog programmable array processor (APAP), based on studies on the vertebrate retina, has permitted the realization of complex programmable spatio-temporal dynam- ics in VLSI. This model mimics the way in which images are pro- cessed in the visual pathway, rendering a feasible alternative for the implementation of early vision applications in standard tech- nologies. A prototype chip has been designed and fabricated in a 0.5(cid:22)m standard CMOS process. Computing power per area and power consumption is amongst the highest reported for a single chip. Design challenges, trade-o(cid:11)s and some experimental results are presented in this paper.
Bayesian Image Super-Resolution
Authors: Michael E. Tipping, Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The  extraction  of a  single  high-quality  image  from  a  set  of  low(cid:173) resolution  images  is  an  important  problem  which  arises  in  fields  such as  remote  sensing,  surveillance,  medical  imaging and the  ex(cid:173) traction of still  images  from  video.  Typical  approaches are  based  on  the  use  of cross-correlation  to  register  the  images  followed  by  the  inversion  of the  transformation  from  the  unknown  high  reso(cid:173) lution image to the observed low  resolution images,  using regular(cid:173) ization to resolve  the  ill-posed  nature of the  inversion  process.  In  this paper we  develop a Bayesian treatment of the super-resolution  problem  in  which  the  likelihood  function  for  the  image  registra(cid:173) tion  parameters  is  based  on  a  marginalization  over  the  unknown  high-resolution  image.  This  approach  allows  us  to  estimate  the  unknown point spread function,  and is  rendered tractable through  the introduction of a  Gaussian  process  prior over images.  Results  indicate a  significant improvement over techniques  based on MAP  (maximum  a-posteriori)  point  optimization  of the  high  resolution  image and associated registration parameters.
A Minimal Intervention Principle for Coordinated Movement
Authors: Emanuel Todorov, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Behavioral goals are achieved reliably and repeatedly with movements rarely reproducible in their detail. Here we offer an explanation: we show that not only are variability and goal achievement compatible, but indeed that allowing variability in redundant dimensions is the optimal control strategy in the face of uncertainty. The optimal feedback control laws for typical motor tasks obey a “minimal intervention” principle: deviations from the average trajectory are only corrected when they interfere with the task goals. The resulting behavior exhibits task-constrained variabil- ity, as well as synergetic coupling among actuators—which is another unexplained empirical phenomenon.
A Probabilistic Approach to Single Channel Blind Signal Separation
Authors: Gil-jin Jang, Te-Won Lee
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We present a new technique for achieving source separation when given only a single channel recording. The main idea is based on exploiting the inherent time structure of sound sources by learning a priori sets of basis ﬁlters in time domain that encode the sources in a statistically efﬁcient manner. We derive a learning algorithm using a maximum likelihood approach given the observed single channel data and sets of basis ﬁlters. For each time point we infer the source signals and their contribution factors. This inference is possible due to the prior knowledge of the basis ﬁlters and the associated coefﬁcient densities. A ﬂexible model for density estimation allows accurate modeling of the observation and our experimental results exhibit a high level of separation performance for mixtures of two music signals as well as the separation of two voice signals.
A Digital Antennal Lobe for Pattern Equalization: Analysis and Design
Authors: Alex Holub, Gilles Laurent, Pietro Perona
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Re-mapping  patterns  in  order  to  equalize  their  distribution  may  greatly  simplify  both the  structure  and  the  training of classifiers.  Here,  the  properties  of one  such  map  obtained  by  running  a  few  steps of discrete-time dynamical system are explored.  The system  is  called  'Digital  Antennal  Lobe'  (DAL)  because  it  is  inspired  by  recent studies of the antennallobe, a structure in the olfactory sys(cid:173) tem  of the  grasshopper.  The  pattern-spreading  properties  of the  DAL  as  well  as  its  average behavior  as  a  function  of its  (few)  de(cid:173) sign  parameters are analyzed by extending previous results of Van  Vreeswijk and Sompolinsky.  Furthermore, a technique for  adapting  the  parameters of the  initial  design  in  order  to  obtain  opportune  noise-rejection behavior is suggested.  Our results are demonstrated  with a  number of simulations.
Knowledge-Based Support Vector Machine Classifiers
Authors: Glenn M. Fung, Olvi L. Mangasarian, Jude W. Shavlik
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Prior knowledge  in the  form  of multiple  polyhedral sets,  each  be(cid:173) longing to one of two categories, is introduced into a  reformulation  of a  linear support vector machine classifier.  The resulting formu(cid:173) lation leads to a linear program that can be solved efficiently.  Real  world examples, from DNA sequencing and breast cancer prognosis,  demonstrate the effectiveness  of the proposed method.  Numerical  results  show  improvement  in  test  set  accuracy  after  the  incorpo(cid:173) ration of prior knowledge into ordinary, data-based linear support  vector  machine  classifiers.  One  experiment  also  shows  that  a  lin(cid:173) ear classifier,  based solely on prior knowledge,  far  outperforms the  direct  application of prior knowledge  rules  to classify data.  Keywords:  use  and  refinement  of prior  knowledge,  sup(cid:173) port  vector machines,  linear programming
Conditional Models on the Ranking Poset
Authors: Guy Lebanon, John D. Lafferty
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A distance-based conditional model on the ranking poset is presented for use in classiﬁcation and ranking. The model is an extension of the Mallows  model, and generalizes the classiﬁer combination methods used by several ensemble learning algorithms, including error correcting output codes, discrete AdaBoost, logistic regression and cranking. The algebraic structure of the ranking poset leads to a simple Bayesian inter- pretation of the conditional model and its special cases. In addition to a unifying view, the framework suggests a probabilistic interpretation for error correcting output codes and an extension beyond the binary coding scheme.
A Formulation for Minimax Probability Machine Regression
Authors: Thomas Strohmann, Gregory Z. Grudic
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We formulate the regression problem as one of maximizing the mini- mum probability, symbolized by (cid:10), that future predicted outputs of the regression model will be within some (cid:6)" bound of the true regression function. Our formulation is unique in that we obtain a direct estimate of this lower probability bound (cid:10). The proposed framework, minimax probability machine regression (MPMR), is based on the recently de- scribed minimax probability machine classiﬁcation algorithm [Lanckriet et al.] and uses Mercer Kernels to obtain nonlinear regression models. MPMR is tested on both toy and real world data, verifying the accuracy of the (cid:10) bound, and the efﬁcacy of the regression models.
Multiclass Learning by Probabilistic Embeddings
Authors: Ofer Dekel, Yoram Singer
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We describe a new algorithmic framework for learning multiclass catego- rization problems. In this framework a multiclass predictor is composed of a pair of embeddings that map both instances and labels into a common space. In this space each instance is assigned the label it is nearest to. We outline and analyze an algorithm, termed Bunching, for learning the pair of embeddings from labeled data. A key construction in the analysis of the algorithm is the notion of probabilistic output codes, a generaliza- tion of error correcting output codes (ECOC). Furthermore, the method of multiclass categorization using ECOC is shown to be an instance of Bunching. We demonstrate the advantage of Bunching over ECOC by comparing their performance on numerous categorization problems.
VIBES: A Variational Inference Engine for Bayesian Networks
Authors: Christopher M. Bishop, David Spiegelhalter, John Winn
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: In recent years variational methods have become a popular tool for approximate inference and learning in a wide variety of proba- bilistic models. For each new application, however, it is currently necessary (cid:12)rst to derive the variational update equations, and then to implement them in application-speci(cid:12)c code. Each of these steps is both time consuming and error prone. In this paper we describe a general purpose inference engine called VIBES (‘Variational Infer- ence for Bayesian Networks’) which allows a wide variety of proba- bilistic models to be implemented and solved variationally without recourse to coding. New models are speci(cid:12)ed either through a simple script or via a graphical interface analogous to a drawing package. VIBES then automatically generates and solves the vari- ational equations. We illustrate the power and (cid:13)exibility of VIBES using examples from Bayesian mixture modelling.
A Convergent Form of Approximate Policy Iteration
Authors: Theodore J. Perkins, Doina Precup
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We study a new, model-free form of approximate policy iteration which uses Sarsa updates with linear state-action value function approximation for policy evaluation, and a “policy improvement operator” to generate a new policy based on the learned state-action values. We prove that if the policy improvement operator produces  -soft policies and is Lipschitz continuous in the action values, with a constant that is not too large, then the approximate policy iteration algorithm converges to a unique solu- tion from any initial policy. To our knowledge, this is the ﬁrst conver- gence result for any form of approximate policy iteration under similar computational-resource assumptions.
Mean Field Approach to a Probabilistic Model in Information Retrieval
Authors: Bin Wu, K. Wong, David Bodoff
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We study an explicit parametric model of documents, queries, and rel- evancy assessment for Information Retrieval (IR). Mean-ﬁeld methods are applied to analyze the model and derive efﬁcient practical algorithms to estimate the parameters in the problem. The hyperparameters are es- timated by a fast approximate leave-one-out cross-validation procedure based on the cavity method. The algorithm is further evaluated on several benchmark databases by comparing with standard algorithms in IR.
Exponential Family PCA for Belief Compression in POMDPs
Authors: Nicholas Roy, Geoffrey J. Gordon
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Geoffrey Gordon
Department of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213 ggordon@cs.cmu.edu
Standard value function approaches to ﬁnding policies for Partially Observable Markov Decision Processes (POMDPs) are intractable for large models. The in- tractability of these algorithms is due to a great extent to their generating an optimal policy over the entire belief space. However, in real POMDP problems most belief states are unlikely, and there is a structured, low-dimensional manifold of plausible beliefs embedded in the high-dimensional belief space. We introduce a new method for solving large-scale POMDPs by taking advantage of belief space sparsity. We reduce the dimensionality of the belief space by exponential family Principal Components Analysis [1], which allows us to turn the sparse, high- dimensional belief space into a compact, low-dimensional representation in terms of learned features of the belief state. We then plan directly on the low-dimensional belief features. By planning in a low-dimensional space, we can ﬁnd policies for POMDPs that are orders of magnitude larger than can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and also on a mobile robot navigation task.
Independent Components Analysis through Product Density Estimation
Authors: Trevor Hastie, Rob Tibshirani
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We  present a  simple direct approach for  solving the ICA problem,  using density estimation and maximum likelihood.  Given a  candi(cid:173) date  orthogonal frame,  we  model  each  of the  coordinates  using  a  semi-parametric density estimate based on cubic splines.  Since our  estimates have two continuous derivatives, we  can easily run a sec(cid:173) ond order search for  the frame  parameters.  Our method performs  very favorably  when compared to state-of-the-art techniques.
Discriminative Densities from Maximum Contrast Estimation
Authors: Peter Meinicke, Thorsten Twellmann, Helge Ritter
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We propose a framework for classiﬁer design based on discriminative densities for representation of the differences of the class-conditional dis- tributions in a way that is optimal for classiﬁcation. The densities are selected from a parametrized set by constrained maximization of some objective function which measures the average (bounded) difference, i.e. the contrast between discriminative densities. We show that maximiza- tion of the contrast is equivalent to minimization of an approximation of the Bayes risk. Therefore using suitable classes of probability den- sity functions, the resulting maximum contrast classiﬁers (MCCs) can approximate the Bayes rule for the general multiclass case. In particular for a certain parametrization of the density functions we obtain MCCs which have the same functional form as the well-known Support Vec- tor Machines (SVMs). We show that MCC-training in general requires some nonlinear optimization but under certain conditions the problem is concave and can be tackled by a single linear program. We indicate the close relation between SVM- and MCC-training and in particular we show that Linear Programming Machines can be viewed as an approxi- mate realization of MCCs. In the experiments on benchmark data sets, the MCC shows a competitive classiﬁcation performance.
Interpreting Neural Response Variability as Monte Carlo Sampling of the Posterior
Authors: Patrik O. Hoyer, Aapo Hyvärinen
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The responses of cortical sensory neurons are notoriously variable, with the number of spikes evoked by identical stimuli varying signiﬁcantly from trial to trial. This variability is most often interpreted as ‘noise’, purely detrimental to the sensory system. In this paper, we propose an al- ternative view in which the variability is related to the uncertainty, about world parameters, which is inherent in the sensory stimulus. Speciﬁ- cally, the responses of a population of neurons are interpreted as stochas- tic samples from the posterior distribution in a latent variable model. In addition to giving theoretical arguments supporting such a representa- tional scheme, we provide simulations suggesting how some aspects of response variability might be understood in this framework.
Combining Features for BCI
Authors: Guido Dornhege, Benjamin Blankertz, Gabriel Curio, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Recently, interest is growing to develop an effective communication in- terface connecting the human brain to a computer, the ’Brain-Computer Interface’ (BCI). One motivation of BCI research is to provide a new communication channel substituting normal motor output in patients with severe neuromuscular disabilities. In the last decade, various neuro- physiological cortical processes, such as slow potential shifts, movement related potentials (MRPs) or event-related desynchronization (ERD) of spontaneous EEG rhythms, were shown to be suitable for BCI, and, con- sequently, different independent approaches of extracting BCI-relevant EEG-features for single-trial analysis are under investigation. Here, we present and systematically compare several concepts for combining such EEG-features to improve the single-trial classiﬁcation. Feature combi- nations are evaluated on movement imagination experiments with 3 sub- jects where EEG-features are based on either MRPs or ERD, or both. Those combination methods that incorporate the assumption that the sin- gle EEG-features are physiologically mutually independent outperform the plain method of ’adding’ evidence where the single-feature vectors are simply concatenated. These results strengthen the hypothesis that MRP and ERD reﬂect at least partially independent aspects of cortical processes and open a new perspective to boost BCI effectiveness.
A Probabilistic Model for Learning Concatenative Morphology
Authors: Matthew G. Snover, Michael R. Brent
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper describes a system for the unsupervised learning of morpho- logical sufﬁxes and stems from word lists. The system is composed of a generative probability model and hill-climbing and directed search algo- rithms. By extracting and examining morphologically rich subsets of an input lexicon, the directed search identiﬁes highly productive paradigms. The hill-climbing algorithm then further maximizes the probability of the hypothesis. Quantitative results are shown by measuring the accuracy of the morphological relations identiﬁed. Experiments in English and Pol- ish, as well as comparisons with another recent unsupervised morphol- ogy learning algorithm demonstrate the effectiveness of this technique.
The Decision List Machine
Authors: Marina Sokolova, Mario Marchand, Nathalie Japkowicz, John S. Shawe-taylor
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We introduce a new learning algorithm for decision lists to allow features that are constructed from the data and to allow a trade- oﬁ between accuracy and complexity. We bound its generalization error in terms of the number of errors and the size of the classiﬂer it ﬂnds on the training data. We also compare its performance on some natural data sets with the set covering machine and the support vector machine.
How to Combine Color and Shape Information for 3D Object Recognition: Kernels do the Trick
Authors: B. Caputo, Gy. Dorkó
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper presents a  kernel method that allows to combine  color  and shape information for  appearance-based object recognition.  It  doesn't require to define a new common representation, but use the  power of kernels to combine different representations together in an  effective manner.  These results are achieved using results of statis(cid:173) tical mechanics of spin glasses combined with Markov random fields  via kernel functions.  Experiments show  an increase in recognition  rate up  to 5.92%  with respect to conventional strategies.
Selectivity and Metaplasticity in a Unified Calcium-Dependent Model
Authors: Luk Chong Yeung, Brian S. Blais, Leon N. Cooper, Harel Z. Shouval
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A uni(cid:12)ed, biophysically motivated Calcium-Dependent Learning model has been shown to account for various rate-based and spike time-dependent paradigms for inducing synaptic plasticity. Here, we investigate the properties of this model for a multi-synapse neuron that receives inputs with di(cid:11)erent spike-train statistics. In addition, we present a physiological form of metaplasticity, an activity-driven regulation mechanism, that is essential for the ro- bustness of the model. A neuron thus implemented develops stable and selective receptive (cid:12)elds, given various input statistics
Learning to Detect Natural Image Boundaries Using Brightness and Texture
Authors: David R. Martin, Charless C. Fowlkes, Jitendra Malik
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: The goal of this work is to accurately detect and localize boundaries in natural scenes using local image measurements. We formulate features that respond to characteristic changes in brightness and texture associated with natural boundaries. In order to combine the information from these features in an optimal way, a classiﬁer is trained using human labeled images as ground truth. We present precision-recall curves showing that the resulting detector outperforms existing approaches.
Concurrent Object Recognition and Segmentation by Graph Partitioning
Authors: Stella X. Yu, Ralph Gross, Jianbo Shi
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Segmentation and recognition have long been treated as two separate pro(cid:173) cesses.  We  propose a  mechanism  based  on  spectral  graph  partitioning  that readily combine the  two processes into one.  A part-based recogni(cid:173) tion system detects object patches, supplies their partial segmentations as  well as knowledge about the spatial configurations of the object.  The goal  of patch grouping is to find a set of patches that conform best to the object  configuration,  while the goal  of pixel grouping is  to  find  a set of pixels  that have  the  best low-level  feature  similarity.  Through pixel-patch  in(cid:173) teractions and between-patch competition encoded in the solution space,  these two processes are realized in  one joint optimization problem.  The  globally optimal partition is obtained by solving a constrained eigenvalue  problem.  We  demonstrate that the  resulting  object segmentation elimi(cid:173) nates false  positives for  the part detection, while  overcoming occlusion  and weak contours for the low-level edge detection.
Approximate Linear Programming for Average-Cost Dynamic Programming
Authors: Benjamin V. Roy, Daniela D. Farias
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper extends our earlier analysis on approximate linear program- ming as an approach to approximating the cost-to-go function in a discounted-cost dynamic program [6]. In this paper, we consider the average-cost criterion and a version of approximate linear programming that generates approximations to the optimal average cost and differential cost function. We demonstrate that a naive version of approximate linear programming prioritizes approximation of the optimal average cost and that this may not be well-aligned with the objective of deriving a policy with low average cost. For that, the algorithm should aim at producing a good approximation of the differential cost function. We propose a two- phase variant of approximate linear programming that allows for external control of the relative accuracy of the approximation of the differential cost function over different portions of the state space via state-relevance weights. Performance bounds suggest that the new algorithm is compat- ible with the objective of optimizing performance and provide guidance on appropriate choices for state-relevance weights.
Learning Sparse Topographic Representations with Products of Student-t Distributions
Authors: Max Welling, Simon Osindero, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We propose a model for natural images in which the probability of an im- age is proportional to the product of the probabilities of some ﬁlter out- puts. We encourage the system to ﬁnd sparse features by using a Student- t distribution to model each ﬁlter output. If the t-distribution is used to model the combined outputs of sets of neurally adjacent ﬁlters, the sys- tem learns a topographic map in which the orientation, spatial frequency and location of the ﬁlters change smoothly across the map. Even though maximum likelihood learning is intractable in our model, the product form allows a relatively efﬁcient learning procedure that works well even for highly overcomplete sets of ﬁlters. Once the model has been learned it can be used as a prior to derive the “iterated Wiener ﬁlter” for the pur- pose of denoising images.
Margin Analysis of the LVQ Algorithm
Authors: Koby Crammer, Ran Gilad-bachrach, Amir Navot, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Prototypes based algorithms are commonly used to reduce the computa- tional complexity of Nearest-Neighbour (NN) classiﬁers. In this paper we discuss theoretical and algorithmical aspects of such algorithms. On the theory side, we present margin based generalization bounds that sug- gest that these kinds of classiﬁers can be more accurate then the 1-NN rule. Furthermore, we derived a training algorithm that selects a good set of prototypes using large margin principles. We also show that the 20 years old Learning Vector Quantization (LVQ) algorithm emerges natu- rally from our framework.
A Neural Edge-Detection Model for Enhanced Auditory Sensitivity in Modulated Noise
Authors: Alon Fishbach, Bradford J. May
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Psychophysical data suggest that temporal modulations of stimulus  amplitude envelopes play a prominent role in the perceptual  segregation of concurrent sounds. In particular, the detection of an  unmodulated signal can be significantly improved by adding  amplitude modulation to the spectral envelope of a competing  masking noise. This perceptual phenomenon  is known as  “Comodulation Masking Release” (CMR). Despite the obvious  influence of temporal structure on the perception of complex  auditory scenes, the physiological mechanisms that contribute to  CMR and auditory streaming are not well known. A recent  physiological study by Nelken and colleagues has demonstrated an  enhanced cortical representation of auditory signals in modulated  noise. Our study evaluates these CMR-like response patterns from  the perspective of a hypothetical auditory edge-detection neuron. It  is shown that this simple neural model for the detection of  amplitude transients can reproduce not only the physiological data  of Nelken et al., but also, in light of previous results, a variety of  physiological and psychoacoustical phenomena that are related to  the perceptual segregation of concurrent sounds.
Concentration Inequalities for the Missing Mass and for Histogram Rule Error
Authors: Luis E. Ortiz, David A. McAllester
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: This paper gives distribution-free concentration inequalities for the miss- ing mass and the error rate of histogram rules. Negative association meth- ods can be used to reduce these concentration problems to concentration questions about independent sums. Although the sums are independent, they are highly heterogeneous. Such highly heterogeneous independent sums cannot be analyzed using standard concentration inequalities such as Hoeffding’s inequality, the Angluin-Valiant bound, Bernstein’s in- equality, Bennett’s inequality, or McDiarmid’s theorem.
Learning Sparse Multiscale Image Representations
Authors: Phil Sallee, Bruno A. Olshausen
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We describe a method for learning sparse multiscale image repre- sentations using a sparse prior distribution over the basis function coe(cid:14)cients. The prior consists of a mixture of a Gaussian and a Dirac delta function, and thus encourages coe(cid:14)cients to have exact zero values. Coe(cid:14)cients for an image are computed by sampling from the resulting posterior distribution with a Gibbs sampler. The learned basis is similar to the Steerable Pyramid basis, and yields slightly higher SNR for the same number of active coe(cid:14)cients. De- noising using the learned image model is demonstrated for some standard test images, with results that compare favorably with other denoising methods.
The Effect of Singularities in a Learning Machine when the True Parameters Do Not Lie on such Singularities
Authors: Sumio Watanabe, Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: A lot of learning machines with hidden variables used in infor- mation science have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, resulting that the learning theory of regular statistical models does not hold. Recently, it was proven that, if the true parameter is contained in singularities, then the coe(cid:14)cient of the Bayes gen- eralization error is equal to the pole of the zeta function of the Kullback information. In this paper, under the condition that the true parameter is almost but not contained in singularities, we show two results. (1) If the dimension of the parameter from in- puts to hidden units is not larger than three, then there exits a region of true parameters where the generalization error is larger than those of regular models, however, if otherwise, then for any true parameter, the generalization error is smaller than those of regular models. (2) The symmetry of the generalization error and the training error does not hold in singular models in general.
Learning About Multiple Objects in Images: Factorial Learning without Factorial Search
Authors: Christopher K. I. Williams, Michalis K. Titsias
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider data which are images containing views of multiple objects. Our task is to learn about each of the objects present in the images. This task can be approached as a factorial learning problem, where each image must be explained by instantiating a model for each of the objects present with the correct instantiation parameters. A major problem with learning a factorial model is that as the number of objects increases, there is a combinatorial explosion of the number of conﬁgurations that need to be considered. We develop a method to extract object models sequentially from the data by making use of a robust statistical method, thus avoid- ing the combinatorial explosion, and present results showing successful extraction of objects from real images.
Distance Metric Learning with Application to Clustering with Side-Information
Authors: Eric P. Xing, Michael I. Jordan, Stuart Russell, Andrew Y. Ng
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: @cs.berkeley.edu
Many algorithms rely critically on being given a good metric over their inputs. For instance, data can often be clustered in many “plausible” ways, and if a clustering algorithm such as K-means initially fails to ﬁnd one that is meaningful to a user, the only recourse may be for the user to manually tweak the metric until sufﬁciently good clusters are found. For these and other applications requiring good metrics, it is desirable that we provide a more systematic way for users to indicate what they con- sider “similar.” For instance, we may ask them to provide examples. In this paper, we present an algorithm that, given examples of similar (and, , learns a distance metric over if desired, dissimilar) pairs of points in that respects these relationships. Our method is based on posing met-  ric learning as a convex optimization problem, which allows us to give efﬁcient, local-optima-free algorithms. We also demonstrate empirically that the learned metrics can be used to signiﬁcantly improve clustering performance.

Location Estimation with a Differential Update Network
Authors: Ali Rahimi, Trevor Darrell
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: Given a set of hidden variables with an a-priori Markov structure, we derive an online algorithm which approximately updates the posterior as pairwise measurements between the hidden variables become available. The update is performed using Assumed Density Filtering: to incorporate each pairwise measurement, we compute the optimal Markov structure which represents the true posterior and use it as a prior for incorporating the next measurement. We demonstrate the resulting algorithm by cal- culating globally consistent trajectories of a robot as it navigates along a 2D trajectory. To update a trajectory of length t, the update takes O(t). When all conditional distributions are linear-Gaussian, the algorithm can be thought of as a Kalman Filter which simpliﬁes the state covariance matrix after incorporating each measurement.
Hyperkernels
Authors: Cheng S. Ong, Robert C. Williamson, Alex J. Smola
Published in: Advances in Neural Information Processing Systems 15  (NIPS 2002) (Main Conference Track), 2002

Abstract: We consider the problem of choosing a kernel suitable for estimation using a Gaussian Process estimator or a Support Vector Machine. A novel solution is presented which involves deﬁning a Reproducing Ker- nel Hilbert Space on the space of kernels itself. By utilizing an analog of the classical representer theorem, the problem of choosing a kernel from a parameterized family of kernels (e.g. of varying width) is reduced to a statistical estimation problem akin to the problem of minimizing a regularized risk functional. Various classical settings for model or kernel selection are special cases of our framework.