mtybilly/Instruct2DS huggingface dataset:
A Classification-based Cocktail-party Processor
Authors: Deliang Wang, Guy J. Brown Nicoleta Roman
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Guy J. Brown 
Department of Computer Science 
University of Sheffield  211 Portobello Street  Sheffield, S1 4DP, UK  g.brown@dcs.shef.ac.uk
Linear Dependent Dimensionality Reduction
Authors: Tommi S. Jaakkola Nathan Srebro
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We formulate linear dimensionality reduction as a semi-parametric esti- mation problem, enabling us to study its asymptotic behavior. We gen- eralize the problem beyond additive Gaussian noise to (unknown) non- Gaussian additive noise, and to unbiased non-additive models.
Clustering with the Connectivity Kernel
Authors: Volker Roth, Joachim M. Buhmann Bernd Fischer
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Clustering aims at extracting hidden structure in dataset. While the prob- lem of ﬁnding compact clusters has been widely studied in the litera- ture, extracting arbitrarily formed elongated structures is considered a much harder problem. In this paper we present a novel clustering algo- rithm which tackles the problem by a two step procedure: ﬁrst the data are transformed in such a way that elongated structures become compact ones. In a second step, these new objects are clustered by optimizing a compactness-based criterion. The advantages of the method over related approaches are threefold: (i) robustness properties of compactness-based criteria naturally transfer to the problem of extracting elongated struc- tures, leading to a model which is highly robust against outlier objects; (ii) the transformed distances induce a Mercer kernel which allows us to formulate a polynomial approximation scheme to the generally NP- hard clustering problem; (iii) the new method does not contain free kernel parameters in contrast to methods like spectral clustering or mean-shift clustering.
Attractive People: Assembling Loose-Limbed Models using Non-parametric Belief Propagation
Authors: Michael Isard, Benjamin H. Sigelman, Michael J. Black Leonid Sigal
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: The detection and pose estimation of people in images and video is made challenging by the variability of human appearance, the complexity of natural scenes, and the high dimensionality of articulated body mod- els. To cope with these problems we represent the 3D human body as a graphical model in which the relationships between the body parts are represented by conditional probability distributions. We formulate the pose estimation problem as one of probabilistic inference over a graphi- cal model where the random variables correspond to the individual limb parameters (position and orientation). Because the limbs are described by 6-dimensional vectors encoding pose in 3-space, discretization is im- practical and the random variables in our model must be continuous- valued. To approximate belief propagation in such a graph we exploit a recently introduced generalization of the particle ﬁlter. This framework facilitates the automatic initialization of the body-model from low level cues and is robust to occlusion of body parts and scene clutter.
Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering
Authors: Jean-françcois Paiement, Pascal Vincent, Olivier Delalleau, Nicolas L. Roux, Marie Ouimet Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Several unsupervised learning algorithms based on an eigendecompo- sition provide either an embedding or a clustering only for given train- ing points, with no straightforward extension for out-of-sample examples short of recomputing eigenvectors. This paper provides a uniﬁed frame- work for extending Local Linear Embedding (LLE), Isomap, Laplacian Eigenmaps, Multi-Dimensional Scaling (for dimensionality reduction) as well as for Spectral Clustering. This framework is based on seeing these algorithms as learning eigenfunctions of a data-dependent kernel. Numerical experiments show that the generalizations performed have a level of error comparable to the variability of the embedding algorithms due to the choice of training data.
Learning Spectral Clustering
Authors: Michael I. Jordan Francis R. Bach
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Spectral clustering refers to a class of techniques which rely on the eigen- structure of a similarity matrix to partition points into disjoint clusters with points in the same cluster having high similarity and points in dif- ferent clusters having low similarity. In this paper, we derive a new cost function for spectral clustering based on a measure of error between a given partition and a solution of the spectral relaxation of a minimum normalized cut problem. Minimizing this cost function with respect to the partition leads to a new spectral clustering algorithm. Minimizing with respect to the similarity matrix leads to an algorithm for learning the similarity matrix. We develop a tractable approximation of our cost function that is based on the power method of computing eigenvectors.
Maximum Likelihood Estimation of a Stochastic Integrate-and-Fire Neural Model
Authors: Eero P. Simoncelli, Jonathan W. Pillow Liam Paninski
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Recent work has examined the estimation of models of stimulus-driven neural activity in which some linear ﬁltering process is followed by a nonlinear, probabilistic spiking stage. We analyze the estimation of one such model for which this nonlinear step is implemented by a noisy, leaky, integrate-and-ﬁre mechanism with a spike-dependent after- current. This model is a biophysically plausible alternative to models with Poisson (memory-less) spiking, and has been shown to effectively reproduce various spiking statistics of neurons in vivo. However, the problem of estimating the model from extracellular spike train data has not been examined in depth. We formulate the problem in terms of max- imum likelihood estimation, and show that the computational problem of maximizing the likelihood is tractable. Our main contribution is an algorithm and a proof that this algorithm is guaranteed to ﬁnd the global optimum with reasonable speed. We demonstrate the effectiveness of our estimator with numerical simulations.
A central issue in computational neuroscience is the characterization of the functional re- lationship between sensory stimuli and neural spike trains. A common model for this re- lationship consists of linear ﬁltering of the stimulus, followed by a nonlinear, probabilistic spike generation process. The linear ﬁlter is typically interpreted as the neuron’s “receptive ﬁeld,” while the spiking mechanism accounts for simple nonlinearities like rectiﬁcation and response saturation. Given a set of stimuli and (extracellularly) recorded spike times, the characterization problem consists of estimating both the linear ﬁlter and the parameters governing the spiking mechanism.
One widely used model of this type is the Linear-Nonlinear-Poisson (LNP) cascade model, in which spikes are generated according to an inhomogeneous Poisson process, with rate determined by an instantaneous (“memoryless”) nonlinear function of the ﬁltered input. This model has a number of desirable features, including conceptual simplicity and com- putational tractability. Additionally, reverse correlation analysis provides a simple unbi- ased estimator for the linear ﬁlter [5], and the properties of estimators (for both the linear ﬁlter and static nonlinearity) have been thoroughly analyzed, even for the case of highly non-symmetric or “naturalistic” stimuli [12]. One important drawback of the LNP model,

JWP and LP contributed equally to this work. We thank E.J. Chichilnisky for helpful discussions.

l
Prediction on Spike Data Using Kernel Algorithms
Authors: Andreas Tolias, Alexander Zien, Malte Kuss, Jason Weston, Nikos Logothetis, Bernhard Schölkopf, Carl E. Rasmussen Jan Eichhorn
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We report and compare the performance of different learning algorithms based on data from cortical recordings. The task is to predict the orienta- tion of visual stimuli from the activity of a population of simultaneously recorded neurons. We compare several ways of improving the coding of the input (i.e., the spike data) as well as of the output (i.e., the orienta- tion), and report the results obtained using different kernel algorithms.
An MDP-Based Approach to Online Mechanism Design
Authors: Satinder P. Singh David C. Parkes
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Online mechanism design (MD) considers the problem of provid- ing incentives to implement desired system-wide outcomes in sys- tems with self-interested agents that arrive and depart dynami- cally. Agents can choose to misrepresent their arrival and depar- ture times, in addition to information about their value for di(cid:11)erent outcomes. We consider the problem of maximizing the total long- term value of the system despite the self-interest of agents. The online MD problem induces a Markov Decision Process (MDP), which when solved can be used to implement optimal policies in a truth-revealing Bayesian-Nash equilibrium.
Probabilistic Inference of Speech Signals from Phaseless Spectrograms
Authors: Sam T. Roweis, Brendan J. Frey Kannan Achan
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Many techniques for complex speech processing such as denoising and deconvolution, time/frequency warping, multiple speaker separation, and multiple microphone analysis operate on sequences of short-time power spectra (spectrograms), a representation which is often well-suited to these tasks. However, a signiﬁcant problem with algorithms that manipu- late spectrograms is that the output spectrogram does not include a phase component, which is needed to create a time-domain signal that has good perceptual quality. Here we describe a generative model of time-domain speech signals and their spectrograms, and show how an efﬁcient opti- mizer can be used to ﬁnd the maximum a posteriori speech signal, given the spectrogram. In contrast to techniques that alternate between esti- mating the phase and a spectrally-consistent signal, our technique di- rectly infers the speech signal, thus jointly optimizing the phase and a spectrally-consistent signal. We compare our technique with a standard method using signal-to-noise ratios, but we also provide audio ﬁles on the web for the purpose of demonstrating the improvement in perceptual quality that our technique offers.
Denoising and Untangling Graphs Using Degree Priors
Authors: Brendan J. Frey Quaid D. Morris
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper addresses the problem of untangling hidden graphs from a set of noisy detections of undirected edges. We present a model of the generation of the observed graph that includes degree-based structure priors on the hidden graphs. Exact inference in the model is intractable; we present an e–cient approximate inference algo- rithm to compute edge appearance posteriors. We evaluate our model and algorithm on a biological graph inference problem.
1
Introduction and motivation
The inference of hidden graphs from noisy edge appearance data is an important problem with obvious practical application. For example, biologists are currently building networks of all the physical protein-protein interactions (PPI) that occur in particular organisms. The importance of this enterprise is commensurate with its scale: a completed network would be as valuable as a completed genome sequence, and because each organism contains thousands of diﬁerent types of proteins, there are millions of possible types of interactions. However, scalable experimental meth- ods for detecting interactions are noisy, generating many false detections. Motivated by this application, we formulate the general problem of inferring hidden graphs as probabilistic inference in a graphical model, and we introduce an e–cient algorithm that approximates the posterior probability that an edge is present.
In our model, a set of hidden, constituent graphs are combined to generate the ob- served graph. Each hidden graph is independently sampled from a prior on graph structure. The combination mechanism acts independently on each edge but can be either stochastic or deterministic. Figure 1 shows an example of our generative model. Typically one of the hidden graphs represents the graph of interest (the true graph), the others represent diﬁerent types of observation noise. Independent edge noise may also be added by the combination mechanism. We use probabilistic in- ference to compute a likely decomposition of the observed graph into its constituent parts. This process is deemed \untangling". We use the term \denoising" to refer to the special case where the edge noise is independent. In denoising there is a single hidden graph, the true graph, and all edge noise in the observed graph is due
Learning a Distance Metric from Relative Comparisons
Authors: Thorsten Joachims Matthew Schultz
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents a method for learning a distance metric from rel- ative comparison such as “A is closer to B than A is to C”. Taking a Support Vector Machine (SVM) approach, we develop an algorithm that provides a ﬂexible way of describing qualitative training data as a set of constraints. We show that such constraints lead to a convex quadratic programming problem that can be solved by adapting standard meth- ods for SVM training. We empirically evaluate the performance and the modelling ﬂexibility of the algorithm on a collection of text documents.
Locality Preserving Projections
Authors: Partha Niyogi Xiaofei He
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Many problems in information processing involve some form of dimen- sionality reduction. In this paper, we introduce Locality Preserving Pro- jections (LPP). These are linear projective maps that arise by solving a variational problem that optimally preserves the neighborhood structure of the data set. LPP should be seen as an alternative to Principal Com- ponent Analysis (PCA) – a classical linear technique that projects the data along the directions of maximal variance. When the high dimen- sional data lies on a low dimensional manifold embedded in the ambient space, the Locality Preserving Projections are obtained by ﬁnding the optimal linear approximations to the eigenfunctions of the Laplace Bel- trami operator on the manifold. As a result, LPP shares many of the data representation properties of nonlinear techniques such as Laplacian Eigenmaps or Locally Linear Embedding. Yet LPP is linear and more crucially is deﬁned everywhere in ambient space rather than just on the training data points. This is borne out by illustrative examples on some high dimensional data sets.
Unsupervised Color Decomposition Of Histologically Stained Tissue Samples
Authors: Sameer Agarwal, Casey Laris, Jeffrey H. Price, Serge J. Belongie Andrew Rabinovich
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Accurate spectral decomposition is essential for the analysis and diagnosis of histologically stained tissue sections. In this paper we present the ﬁrst automated system for performing this decompo- sition. We compare the performance of our system with ground truth data and report favorable results.
Minimising Contrastive Divergence in Noisy, Mixed-mode VLSI Neurons
Authors: Patrice Fleury, Alan F. Murray Hsin Chen
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents VLSI circuits with continuous-valued proba- bilistic behaviour realized by injecting noise into each computing unit(neuron). Interconnecting the noisy neurons forms a Contin- uous Restricted Boltzmann Machine (CRBM), which has shown promising performance in modelling and classifying noisy biomed- ical data. The Minimising-Contrastive-Divergence learning algo- rithm for CRBM is also implemented in mixed-mode VLSI, to adapt the noisy neurons’ parameters on-chip.
Eye Movements for Reward Maximization
Authors: Dana Ballard Nathan Sprague
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: University of Rochester Rochester, NY 14627 dana@cs.rochester.edu
Recent eye tracking studies in natural tasks suggest that there is a tight link between eye movements and goal directed motor actions. However, most existing models of human eye movements provide a bottom up ac- count that relates visual attention to attributes of the visual scene. The purpose of this paper is to introduce a new model of human eye move- ments that directly ties eye movements to the ongoing demands of be- havior. The basic idea is that eye movements serve to reduce uncertainty about environmental variables that are task relevant. A value is assigned to an eye movement by estimating the expected cost of the uncertainty that will result if the movement is not made. If there are several candidate eye movements, the one with the highest expected value is chosen. The model is illustrated using a humanoid graphic ﬁgure that navigates on a sidewalk in a virtual urban environment. Simulations show our protocol is superior to a simple round robin scheduling mechanism.
Estimating Internal Variables and Paramters of a Learning Agent by a Particle Filter
Authors: Kenji Doya, Yasumasa Ueda, Minoru Kimura Kazuyuki Samejima
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: When we model a higher order functions, such as learning and memory, we face a difﬁculty of comparing neural activities with hidden variables that depend on the history of sensory and motor signals and the dynam- ics of the network. Here, we propose novel method for estimating hidden variables of a learning agent, such as connection weights from sequences of observable variables. Bayesian estimation is a method to estimate the posterior probability of hidden variables from observable data sequence using a dynamic model of hidden and observable variables. In this pa- per, we apply particle ﬁlter for estimating internal parameters and meta- parameters of a reinforcement learning model. We veriﬁed the effective- ness of the method using both artiﬁcial data and real animal behavioral data.
ICA-based Clustering of Genes from Microarray Expression Data
Authors: Serafim Batzoglou Su-in Lee
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We  propose  an  unsupervised  methodology  using  independent  component  analysis  (ICA)  to  cluster  genes  from  DNA  microarray  data.  Based  on  an  ICA  mixture  model  of  genomic  expression  patterns, linear and nonlinear ICA finds components that are specific  to  certain  biological  processes.  Genes  that  exhibit  significant  up-regulation  or  down-regulation  within  each  component  are  grouped  into  clusters.  We  test  the  statistical  significance  of  enrichment  of  gene  annotations  within  each  cluster.  ICA-based  clustering  outperformed  other  leading  methods  in  constructing  functionally  coherent  clusters  on  various  datasets.  This  result  supports our model of genomic expression data as composite effect  of  independent  biological  processes.  Comparison  of  clustering  performance  among  various  including  a  kernel-based  nonlinear  ICA  algorithm  shows  that  nonlinear  ICA  performed  the  best  for  small  datasets  and  natural-gradient  maximization-likelihood worked well for all the datasets.
On the Concentration of Expectation and Approximate Inference in Layered Networks
Authors: Michael I. Jordan Xuanlong Nguyen
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We present an analysis of concentration-of-expectation phenomena in layered Bayesian networks that use generalized linear models as the local conditional probabilities. This framework encompasses a wide variety of probability distributions, including both discrete and continuous random variables. We utilize ideas from large deviation analysis and the delta method to devise and evaluate a class of approximate inference algo- rithms for layered Bayesian networks that have superior asymptotic error bounds and very fast computation time.
Identifying Structure across Pre-partitioned Data
Authors: Ido Dagan, Eli Shamir Zvika Marx
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We propose an information-theoretic clustering approach that  incorporates a pre-known partition of the data, aiming to identify  common clusters that cut across the given partition. In the standard  clustering setting the formation of clusters is guided by a single  source of feature information. The newly utilized pre-partition  factor introduces an additional bias that counterbalances the impact  of the features whenever they become correlated with this known  partition. The resulting algorithmic framework was applied  successfully to synthetic data, as well as to identifying text-based  cross-religion correspondences.
Eigenvoice Speaker Adaptation via Composite Kernel Principal Component Analysis
Authors: Brian Mak, Simon Ho James T. Kwok
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Eigenvoice speaker adaptation has been shown to be effective when only a small amount of adaptation data is available. At the heart of the method is principal component analysis (PCA) employed to ﬁnd the most im- portant eigenvoices. In this paper, we postulate that nonlinear PCA, in particular kernel PCA, may be even more effective. One major challenge is to map the feature-space eigenvoices back to the observation space so that the state observation likelihoods can be computed during the estima- tion of eigenvoice weights and subsequent decoding. Our solution is to compute kernel PCA using composite kernels, and we will call our new method kernel eigenvoice speaker adaptation. On the TIDIGITS corpus, we found that compared with a speaker-independent model, our kernel eigenvoice adaptation method can reduce the word error rate by 28–33% while the standard eigenvoice approach can only match the performance of the speaker-independent model.
A Summating, Exponentially-Decaying CMOS Synapse for Spiking Neural Systems
Authors: Timothy K. Horiuchi Rock Z. Shi
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Synapses are a critical element of biologically-realistic, spike-based neu- ral computation, serving the role of communication, computation, and modiﬁcation. Many different circuit implementations of synapse func- tion exist with different computational goals in mind. In this paper we describe a new CMOS synapse design that separately controls quiescent leak current, synaptic gain, and time-constant of decay. This circuit im- plements part of a commonly-used kinetic model of synaptic conduc- tance. We show a theoretical analysis and experimental data for proto- types fabricated in a commercially-available 1.5µm CMOS process.
Extending Q-Learning to General Adaptive Multi-Agent Systems
Authors: Gerald Tesauro
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Recent multi-agent extensions of Q-Learning require knowledge of other agents’ payoffs and Q-functions, and assume game-theoretic play at all times by all other agents. This paper proposes a fundamentally different approach, dubbed “Hyper-Q” Learning, in which values of mixed strategies rather than base actions are learned, and in which other agents’ strategies are estimated from observed actions via Bayesian in- ference. Hyper-Q may be effective against many different types of adap- tive agents, even if they are persistently dynamic. Against certain broad categories of adaptation, it is argued that Hyper-Q may converge to ex- act optimal time-varying policies. In tests using Rock-Paper-Scissors, Hyper-Q learns to significantly exploit an Infinitesimal Gradient Ascent (IGA) player, as well as a Policy Hill Climber (PHC) player. Preliminary analysis of Hyper-Q against itself is also presented.
No Unbiased Estimator of the Variance of K-Fold Cross-Validation
Authors: Yves Grandvalet Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Most machine learning researchers perform quantitative experiments to estimate generalization error and compare algorithm performances. In order to draw statistically convincing conclusions, it is important to esti- mate the uncertainty of such estimates. This paper studies the estimation of uncertainty around the K-fold cross-validation estimator. The main theorem shows that there exists no universal unbiased estimator of the variance of K-fold cross-validation. An analysis based on the eigende- composition of the covariance matrix of errors helps to better understand the nature of the problem and shows that naive estimators may grossly underestimate variance, as con£rmed by numerical experiments.
GPPS: A Gaussian Process Positioning System for Cellular Networks
Authors: Marian Grigoras, Volker Tresp, Clemens Hoffmann Anton Schwaighofer
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: In this article, we present a novel approach to solving the localization problem in cellular networks. The goal is to estimate a mobile user’s position, based on measurements of the signal strengths received from network base stations. Our solution works by building Gaussian process models for the distribution of signal strengths, as obtained in a series of calibration measurements. In the localization stage, the user’s posi- tion can be estimated by maximizing the likelihood of received signal strengths with respect to the position. We investigate the accuracy of the proposed approach on data obtained within a large indoor cellular network.
Reconstructing MEG Sources with Unknown Correlations
Authors: Srikantan S. Nagarajan Maneesh Sahani
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Existing source location and recovery algorithms used in magnetoen- cephalographic imaging generally assume that the source activity at dif- ferent brain locations is independent or that the correlation structure is known. However, electrophysiological recordings of local ﬁeld poten- tials show strong correlations in aggregate activity over signiﬁcant dis- tances. Indeed, it seems very likely that stimulus-evoked activity would follow strongly correlated time-courses in different brain areas. Here, we present, and validate through simulations, a new approach to source reconstruction in which the correlation between sources is modelled and estimated explicitly by variational Bayesian methods, facilitating accu- rate recovery of source locations and the time-courses of their activation.
Increase Information Transfer Rates in BCI by CSP Extension to Multi-class
Authors: Benjamin Blankertz, Gabriel Curio, Klaus-Robert Müller Guido Dornhege
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Brain-Computer Interfaces (BCI) are an interesting emerging technology that is driven by the motivation to develop an effective communication in- terface translating human intentions into a control signal for devices like computers or neuroprostheses. If this can be done bypassing the usual hu- man output pathways like peripheral nerves and muscles it can ultimately become a valuable tool for paralyzed patients. Most activity in BCI re- search is devoted to ﬁnding suitable features and algorithms to increase information transfer rates (ITRs). The present paper studies the implica- tions of using more classes, e.g., left vs. right hand vs. foot, for operating a BCI. We contribute by (1) a theoretical study showing under some mild assumptions that it is practically not useful to employ more than three or four classes, (2) two extensions of the common spatial pattern (CSP) algorithm, one interestingly based on simultaneous diagonalization, and (3) controlled EEG experiments that underline our theoretical ﬁndings and show excellent improved ITRs.
Dopamine Modulation in a Basal Ganglio-Cortical Network of Working Memory
Authors: Peter Dayan, Boris S. Gutkin, Sara A. Solla Aaron J. Gruber
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Dopamine exerts two classes of effect on the sustained neural activity in prefrontal cortex that underlies working memory. Direct release in the cortex increases the contrast of prefrontal neurons, enhancing the ro- bustness of storage. Release of dopamine in the striatum is associated with salient stimuli and makes medium spiny neurons bistable; this mod- ulation of the output of spiny neurons affects prefrontal cortex so as to indirectly gate access to working memory and additionally damp sensi- tivity to noise. Existing models have treated dopamine in one or other structure, or have addressed basal ganglia gating of working memory ex- clusive of dopamine effects. In this paper we combine these mechanisms and explore their joint effect. We model a memory-guided saccade task to illustrate how dopamine’s actions lead to working memory that is se- lective for salient input and has increased robustness to distraction.
ARA*: Anytime A* with Provable Bounds on Sub-Optimality
Authors: Geoffrey J. Gordon, Sebastian Thrun Maxim Likhachev
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: In real world planning problems, time for deliberation is often limited. Anytime planners are well suited for these problems: they ﬁnd a feasi- ble solution quickly and then continually work on improving it until time runs out. In this paper we propose an anytime heuristic search, ARA, which tunes its performance bound based on available search time. It starts by ﬁnding a suboptimal solution quickly using a loose bound, then tightens the bound progressively as time allows. Given enough time it ﬁnds a provably optimal solution. While improving its bound, ARA reuses previous search efforts and, as a result, is signiﬁcantly more efﬁ- cient than other anytime search methods. In addition to our theoretical analysis, we demonstrate the practical utility of ARA* with experiments on a simulated robot kinematic arm and a dynamic path planning prob- lem for an outdoor rover.
Salient Boundary Detection using Ratio Contour
Authors: Toshiro Kubota, Jeffrey M. Siskind Song Wang
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents a novel graph-theoretic approach, named ratio con- tour, to extract perceptually salient boundaries from a set of noisy bound- ary fragments detected in real images. The boundary saliency is deﬁned using the Gestalt laws of closure, proximity, and continuity. This pa- per ﬁrst constructs an undirected graph with two different sets of edges: solid edges and dashed edges. The weights of solid and dashed edges measure the local saliency in and between boundary fragments, respec- tively. Then the most salient boundary is detected by searching for an optimal cycle in this graph with minimum average weight. The proposed approach guarantees the global optimality without introducing any biases related to region area or boundary length. We collect a variety of images for testing the proposed approach with encouraging results.
Semi-Definite Programming by Perceptron Learning
Authors: Ralf Herbrich, Andriy Kharechko, John S. Shawe-taylor Thore Graepel
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We present a modiﬁed version of the perceptron learning algorithm (PLA) which solves semideﬁnite programs (SDPs) in polynomial time. The algorithm is based on the following three observations: (i) Semideﬁnite programs are linear programs with inﬁnitely many (linear) constraints; (ii) every linear program can be solved by a sequence of constraint satisfaction problems with linear constraints; (iii) in general, the perceptron learning algorithm solves a constraint satisfaction problem with linear constraints in ﬁnitely many updates. Combining the PLA with a probabilistic rescaling algorithm (which, on average, increases the size of the feasable region) results in a prob- abilistic algorithm for solving SDPs that runs in polynomial time. We present preliminary results which demonstrate that the algo- rithm works, but is not competitive with state-of-the-art interior point methods.
An Iterative Improvement Procedure for Hierarchical Clustering
Authors: Sanjoy Dasgupta David Kauchak
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We describe a procedure which ﬁnds a hierarchical clustering by hill- climbing. The cost function we use is a hierarchical extension of the k-means cost; our local moves are tree restructurings and node reorder- ings. We show these can be accomplished efﬁciently, by exploiting spe- cial properties of squared Euclidean distances and by using techniques from scheduling algorithms.
Kernels for Structured Natural Language Data
Authors: Yutaka Sasaki, Eisaku Maeda Jun Suzuki
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper devises a novel kernel function for structured natural language data. In the ﬁeld of Natural Language Processing, feature extraction consists of the following two steps: (1) syntactically and semantically analyzing raw data, i.e., character strings, then representing the results as discrete structures, such as parse trees and dependency graphs with part-of-speech tags; (2) creating (possibly high-dimensional) numerical feature vectors from the discrete structures. The new kernels, called Hier- archical Directed Acyclic Graph (HDAG) kernels, directly accept DAGs whose nodes can contain DAGs. HDAG data structures are needed to fully reﬂect the syntactic and semantic structures that natural language data inherently have. In this paper, we deﬁne the kernel function and show how it permits efﬁcient calculation. Experiments demonstrate that the proposed kernels are superior to existing kernel functions, e.g., se- quence kernels, tree kernels, and bag-of-words kernels.
Near-Minimax Optimal Classification with Dyadic Classification Trees
Authors: Robert Nowak Clayton Scott
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper reports on a family of computationally practical classiﬁers that converge to the Bayes error at near-minimax optimal rates for a va- riety of distributions. The classiﬁers are based on dyadic classiﬁcation trees (DCTs), which involve adaptively pruned partitions of the feature space. A key aspect of DCTs is their spatial adaptivity, which enables lo- cal (rather than global) ﬁtting of the decision boundary. Our risk analysis involves a spatial decomposition of the usual concentration inequalities, leading to a spatially adaptive, data-dependent pruning criterion. For any distribution on (X, Y ) whose Bayes decision boundary behaves locally like a Lipschitz smooth function, we show that the DCT error converges to the Bayes error at a rate within a logarithmic factor of the minimax optimal rate. We also study DCTs equipped with polynomial classiﬁca- tion rules at each leaf, and show that as the smoothness of the boundary increases their errors converge to the Bayes error at a rate approaching n−1/2, the parametric rate. We are not aware of any other practical classi- ﬁers that provide similar rate of convergence guarantees. Fast algorithms for tree pruning are discussed.
Bounded Invariance and the Formation of Place Fields
Authors: Paul F. Verschure Reto Wyss
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: One current explanation of the view independent representation of space by the place-cells of the hippocampus is that they arise out of the summation of view dependent Gaussians. This proposal as- sumes that visual representations show bounded invariance. Here we investigate whether a recently proposed visual encoding scheme called the temporal population code can provide such representa- tions. Our analysis is based on the behavior of a simulated robot in a virtual environment containing speci(cid:12)c visual cues. Our re- sults show that the temporal population code provides a represen- tational substrate that can naturally account for the formation of place (cid:12)elds.
Learning Bounds for a Generalized Family of Bayesian Posterior Distributions
Authors: Tong Zhang
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: In this paper we obtain convergence bounds for the concentration of Bayesian posterior distributions (around the true distribution) using a novel method that simpliﬁes and enhances previous results. Based on the analysis, we also introduce a generalized family of Bayesian posteriors, and show that the convergence behavior of these generalized posteriors is completely determined by the local prior structure around the true distri- bution. This important and surprising robustness property does not hold for the standard Bayesian posterior in that it may not concentrate when there exist “bad” prior structures even at places far away from the true distribution.
Online Learning of Non-stationary Sequences
Authors: Tommi S. Jaakkola Claire Monteleoni
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We consider an online learning scenario in which the learner can make predictions on the basis of a ﬁxed set of experts. We derive upper and lower relative loss bounds for a class of universal learning algorithms in- volving a switching dynamics over the choice of the experts. On the basis of the performance bounds we provide the optimal a priori discretiza- tion for learning the parameter that governs the switching dynamics. We demonstrate the new algorithm in the context of wireless networks.
Modeling Time Varying Systems Using Hidden Control Neural Architecture
Authors: Esther Levin
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Multi-layered  neural  networks  have  recently  been  proposed  for  non(cid:173) linear  prediction  and  system  modeling.  Although  proven  successful  for  modeling  time  invariant nonlinear systems,  the  inability  of neural  networks  to  characterize  temporal  variability  has  so  far  been  an  obstacle  in  applying  them  to  complicated  non stationary  signals,  such  as  speech.  In  this  paper  we  present  a  network  architecture,  called  "Hidden  Control  Neural  Network"  (HCNN),  for  modeling  signals  generated  by  nonlinear  dynamical  systems  with  restricted  time  variability.  The approach  taken  here  is  to  allow  the  mapping  that  is  implemented  by  a  multi  layered  neural  network  to  change  with  time  as  a  function  of an  additional  control  input  signal.  This  network  is  trained  using  an  algorithm  that  is  based  on  "back-propagation"  and  segmentation  algorithms  for  estimating  the  unknown  control  together  with  the  network's  parameters.  The HCNN  approach  was  applied  to  several  tasks  including  modeling  of  time-varying  nonlinear  systems  and  speaker-independent  recognition  of  connected  digits,  yielding  a  word accuracy of 99.1 %.
Using Genetic Algorithms to Improve Pattern Classification Performance
Authors: Richard P Lippmann Eric I. Chang
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Genetic  algorithms  were  used  to select  and  create  features  and  to select  reference  exemplar  patterns for  machine vision  and speech  pattern  classi(cid:173) fication  tasks.  For a  complex speech  recognition  task,  genetic  algorithms  required no more computation time than traditional approaches to feature  selection  but reduced  the number of input features  required  by a factor  of  five  (from 153 to 33 features).  On a difficult artificial machine-vision task,  genetic  algorithms were  able  to create  new features  (polynomial functions  of the original features)  which reduced  classification error rates from  19%  to  almost  0%.  Neural  net  and  k  nearest  neighbor  (KNN)  classifiers  were  unable to provide such low error rates using only the original features.  Ge(cid:173) netic algorithms were also used to reduce the number of reference exemplar  patterns for  a  KNN classifier.  On a  338 training pattern vowel-recognition  problem with  10  classes,  genetic  algorithms reduced  the number of stored  exemplars from 338 to 43 without significantly increasing classification er(cid:173) ror  rate.  In  all  applications,  genetic  algorithms  were  easy  to  apply  and  found  good  solutions  in  many fewer  trials  than would  be  required  by ex(cid:173) haustive search.  Run times were long, but not unreasonable.  These results  suggest  that genetic  algorithms  are  becoming practical for  pattern  classi(cid:173) fication  problems  as faster  serial  and  parallel computers  are  developed.
Generalization Dynamics in LMS Trained Linear Networks
Authors: Yves Chauvin
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: For a simple linear case, a mathematical analysis of the training and gener(cid:173) alization (validation)  performance of networks trained by gradient descent  on a Least Mean Square cost function is provided as a function of the learn(cid:173) ing parameters and of the statistics of the training data base.  The analysis  predicts  that generalization error dynamics  are very dependent  on  a  pri(cid:173) ori initial weights.  In particular, the generalization error might sometimes  weave within a computable range during extended training.  In some cases,  the analysis provides bounds on the optimal number of training cycles  for  minimal  validation error.  For a  speech  labeling  task,  predicted  weaving  effects  were qualitatively tested  and  observed  by computer simulations in  networks trained by the linear and non-linear back-propagation algorithm.
Optimal Filtering in the Salamander Retina
Authors: W. Geoffrey Owen, William Bialek Fred Rieke
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The dark-adapted  visual  system can  count photons  wit h  a  reliability  lim(cid:173) ited by thermal noise  in  the rod photoreceptors - the processing circuitry  bet.ween  t.he  rod  cells  and the brain is  essentially noiseless  and  in fact  may  be  close  to  optimal.  Here  we  design  an  optimal  signal  processor  which  estimates  the  time-varying  light  intensit.y  at  the  retina  based  on  the  rod  signals.  \Ve show  that.  the first stage of optimal signal processing  involves  passing  the  rod  cell  out.put.  t.hrough  a  linear filter  with  characteristics de(cid:173) termined  entirely  by  the  rod  signal  and  noise  spectra.  This  filter  is  very  general;  in  fact  it.  is  the  first  st.age  in  any  visual  signal  processing  task  at.  10\'  photon  flux.  \Ve  iopntify  the  output  of this  first-st.age  filter  wit.h  the  intracellular  voltage  response of the  bipolar  celL  the  first  anatomical  st.age  in  retinal  signal  processing.  From  recent.  data on  tiger  salamander  phot.oreceptors  we  extract  t.he  relevant.  spect.ra  and  make  parameter-free,  quantit.ative predictions of the bipolar celll'esponse to a dim, diffuse flash.  Agreement  wit.h  experiment  is  essentially  perfect.  As  far  as  we  know  this  is  the  first  successful  predicti ve  t.heory  for  neural  dynamics. 
1
Oscillation Onset in Neural Delayed Feedback
Authors: André Longtin
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: This paper studies dynamical aspects of neural systems with delayed  neg(cid:173) ative  feedback  modelled  by  nonlinear  delay-differential  equations.  These  systems  undergo  a  Hopf bifurcation  from  a  stable  fixed  point  to  a  sta(cid:173) ble  limit  cycle  oscillation  as  certain  parameters  are  varied.  It is  shown  that  their  frequency  of oscillation  is  robust  to  parameter  variations  and  noisy  fluctuations,  a  property that  makes  these  systems  good  candidates  for  pacemakers.  The  onset  of oscillation  is  postponed  by  both  additive  and parametric noise in the sense that the state variable spends more time  near the fixed  point than it would  in  the absence of noise.  This is  also the  case  when  noise  affects  the  delayed  variable,  i.e.  when  the  system  has  a  faulty  memory.  Finally,  it  is  shown  that  a  distribution  of delays  (rather  than a  fixed  delay)  also  stabilizes  the fixed  point solution.
Relaxation Networks for Large Supervised Learning Problems
Authors: Robert B. Allen, Anthony Jayakumar, Torsten Zeppenfeld, Ronny Meir Joshua Alspector
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Feedback  connections  are  required  so  that  the  teacher  signal  on  the  output  neurons  can  modify  weights  during  supervised  learning.  Relaxation  methods  are  needed  for  learning  static  patterns  with  full-time  feedback  connections.  Feedback  network  learning  techniques  have  not  achieved  wide  popularity  because  of the  still greater  computational efficiency  of back-propagation.  We  show by simulation that relaxation networks of the kind we  are implementing in  VLSI  are  capable  of  learning  large  problems  just  like  back-propagation  networks.  A microchip incorporates deterministic mean-field theory learning as  well  as  stochastic  Boltzmann  learning.  A  multiple-chip  electronic  system  implementing  these  networks  will  make  high-speed  parallel  learning  in  them  feasible in the future.