mtybilly/Instruct2DS huggingface dataset:
Maximum Likelihood Estimation of a Stochastic Integrate-and-Fire Neural Model
Authors: Liam Paninski, Eero P. Simoncelli, Jonathan W. Pillow
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Recent work has examined the estimation of models of stimulus-driven neural activity in which some linear ﬁltering process is followed by a nonlinear, probabilistic spiking stage. We analyze the estimation of one such model for which this nonlinear step is implemented by a noisy, leaky, integrate-and-ﬁre mechanism with a spike-dependent after- current. This model is a biophysically plausible alternative to models with Poisson (memory-less) spiking, and has been shown to effectively reproduce various spiking statistics of neurons in vivo. However, the problem of estimating the model from extracellular spike train data has not been examined in depth. We formulate the problem in terms of max- imum likelihood estimation, and show that the computational problem of maximizing the likelihood is tractable. Our main contribution is an algorithm and a proof that this algorithm is guaranteed to ﬁnd the global optimum with reasonable speed. We demonstrate the effectiveness of our estimator with numerical simulations.
A central issue in computational neuroscience is the characterization of the functional re- lationship between sensory stimuli and neural spike trains. A common model for this re- lationship consists of linear ﬁltering of the stimulus, followed by a nonlinear, probabilistic spike generation process. The linear ﬁlter is typically interpreted as the neuron’s “receptive ﬁeld,” while the spiking mechanism accounts for simple nonlinearities like rectiﬁcation and response saturation. Given a set of stimuli and (extracellularly) recorded spike times, the characterization problem consists of estimating both the linear ﬁlter and the parameters governing the spiking mechanism.
One widely used model of this type is the Linear-Nonlinear-Poisson (LNP) cascade model, in which spikes are generated according to an inhomogeneous Poisson process, with rate determined by an instantaneous (“memoryless”) nonlinear function of the ﬁltered input. This model has a number of desirable features, including conceptual simplicity and com- putational tractability. Additionally, reverse correlation analysis provides a simple unbi- ased estimator for the linear ﬁlter [5], and the properties of estimators (for both the linear ﬁlter and static nonlinearity) have been thoroughly analyzed, even for the case of highly non-symmetric or “naturalistic” stimuli [12]. One important drawback of the LNP model,

JWP and LP contributed equally to this work. We thank E.J. Chichilnisky for helpful discussions.

l
Prediction on Spike Data Using Kernel Algorithms
Authors: Jan Eichhorn, Andreas Tolias, Alexander Zien, Malte Kuss, Jason Weston, Nikos Logothetis, Bernhard Schölkopf, Carl E. Rasmussen
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We report and compare the performance of different learning algorithms based on data from cortical recordings. The task is to predict the orienta- tion of visual stimuli from the activity of a population of simultaneously recorded neurons. We compare several ways of improving the coding of the input (i.e., the spike data) as well as of the output (i.e., the orienta- tion), and report the results obtained using different kernel algorithms.
An MDP-Based Approach to Online Mechanism Design
Authors: David C. Parkes, Satinder P. Singh
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Online mechanism design (MD) considers the problem of provid- ing incentives to implement desired system-wide outcomes in sys- tems with self-interested agents that arrive and depart dynami- cally. Agents can choose to misrepresent their arrival and depar- ture times, in addition to information about their value for di(cid:11)erent outcomes. We consider the problem of maximizing the total long- term value of the system despite the self-interest of agents. The online MD problem induces a Markov Decision Process (MDP), which when solved can be used to implement optimal policies in a truth-revealing Bayesian-Nash equilibrium.
Probabilistic Inference of Speech Signals from Phaseless Spectrograms
Authors: Kannan Achan, Sam T. Roweis, Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Many techniques for complex speech processing such as denoising and deconvolution, time/frequency warping, multiple speaker separation, and multiple microphone analysis operate on sequences of short-time power spectra (spectrograms), a representation which is often well-suited to these tasks. However, a signiﬁcant problem with algorithms that manipu- late spectrograms is that the output spectrogram does not include a phase component, which is needed to create a time-domain signal that has good perceptual quality. Here we describe a generative model of time-domain speech signals and their spectrograms, and show how an efﬁcient opti- mizer can be used to ﬁnd the maximum a posteriori speech signal, given the spectrogram. In contrast to techniques that alternate between esti- mating the phase and a spectrally-consistent signal, our technique di- rectly infers the speech signal, thus jointly optimizing the phase and a spectrally-consistent signal. We compare our technique with a standard method using signal-to-noise ratios, but we also provide audio ﬁles on the web for the purpose of demonstrating the improvement in perceptual quality that our technique offers.
Denoising and Untangling Graphs Using Degree Priors
Authors: Quaid D. Morris, Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper addresses the problem of untangling hidden graphs from a set of noisy detections of undirected edges. We present a model of the generation of the observed graph that includes degree-based structure priors on the hidden graphs. Exact inference in the model is intractable; we present an e–cient approximate inference algo- rithm to compute edge appearance posteriors. We evaluate our model and algorithm on a biological graph inference problem.
1
Introduction and motivation
The inference of hidden graphs from noisy edge appearance data is an important problem with obvious practical application. For example, biologists are currently building networks of all the physical protein-protein interactions (PPI) that occur in particular organisms. The importance of this enterprise is commensurate with its scale: a completed network would be as valuable as a completed genome sequence, and because each organism contains thousands of diﬁerent types of proteins, there are millions of possible types of interactions. However, scalable experimental meth- ods for detecting interactions are noisy, generating many false detections. Motivated by this application, we formulate the general problem of inferring hidden graphs as probabilistic inference in a graphical model, and we introduce an e–cient algorithm that approximates the posterior probability that an edge is present.
In our model, a set of hidden, constituent graphs are combined to generate the ob- served graph. Each hidden graph is independently sampled from a prior on graph structure. The combination mechanism acts independently on each edge but can be either stochastic or deterministic. Figure 1 shows an example of our generative model. Typically one of the hidden graphs represents the graph of interest (the true graph), the others represent diﬁerent types of observation noise. Independent edge noise may also be added by the combination mechanism. We use probabilistic in- ference to compute a likely decomposition of the observed graph into its constituent parts. This process is deemed \untangling". We use the term \denoising" to refer to the special case where the edge noise is independent. In denoising there is a single hidden graph, the true graph, and all edge noise in the observed graph is due
Learning a Distance Metric from Relative Comparisons
Authors: Matthew Schultz, Thorsten Joachims
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents a method for learning a distance metric from rel- ative comparison such as “A is closer to B than A is to C”. Taking a Support Vector Machine (SVM) approach, we develop an algorithm that provides a ﬂexible way of describing qualitative training data as a set of constraints. We show that such constraints lead to a convex quadratic programming problem that can be solved by adapting standard meth- ods for SVM training. We empirically evaluate the performance and the modelling ﬂexibility of the algorithm on a collection of text documents.
Locality Preserving Projections
Authors: Xiaofei He, Partha Niyogi
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Many problems in information processing involve some form of dimen- sionality reduction. In this paper, we introduce Locality Preserving Pro- jections (LPP). These are linear projective maps that arise by solving a variational problem that optimally preserves the neighborhood structure of the data set. LPP should be seen as an alternative to Principal Com- ponent Analysis (PCA) – a classical linear technique that projects the data along the directions of maximal variance. When the high dimen- sional data lies on a low dimensional manifold embedded in the ambient space, the Locality Preserving Projections are obtained by ﬁnding the optimal linear approximations to the eigenfunctions of the Laplace Bel- trami operator on the manifold. As a result, LPP shares many of the data representation properties of nonlinear techniques such as Laplacian Eigenmaps or Locally Linear Embedding. Yet LPP is linear and more crucially is deﬁned everywhere in ambient space rather than just on the training data points. This is borne out by illustrative examples on some high dimensional data sets.
Unsupervised Color Decomposition Of Histologically Stained Tissue Samples
Authors: Andrew Rabinovich, Sameer Agarwal, Casey Laris, Jeffrey H. Price, Serge J. Belongie
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Accurate spectral decomposition is essential for the analysis and diagnosis of histologically stained tissue sections. In this paper we present the ﬁrst automated system for performing this decompo- sition. We compare the performance of our system with ground truth data and report favorable results.
Minimising Contrastive Divergence in Noisy, Mixed-mode VLSI Neurons
Authors: Hsin Chen, Patrice Fleury, Alan F. Murray
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents VLSI circuits with continuous-valued proba- bilistic behaviour realized by injecting noise into each computing unit(neuron). Interconnecting the noisy neurons forms a Contin- uous Restricted Boltzmann Machine (CRBM), which has shown promising performance in modelling and classifying noisy biomed- ical data. The Minimising-Contrastive-Divergence learning algo- rithm for CRBM is also implemented in mixed-mode VLSI, to adapt the noisy neurons’ parameters on-chip.
Eye Movements for Reward Maximization
Authors: Nathan Sprague, Dana Ballard
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: University of Rochester Rochester, NY 14627 dana@cs.rochester.edu
Recent eye tracking studies in natural tasks suggest that there is a tight link between eye movements and goal directed motor actions. However, most existing models of human eye movements provide a bottom up ac- count that relates visual attention to attributes of the visual scene. The purpose of this paper is to introduce a new model of human eye move- ments that directly ties eye movements to the ongoing demands of be- havior. The basic idea is that eye movements serve to reduce uncertainty about environmental variables that are task relevant. A value is assigned to an eye movement by estimating the expected cost of the uncertainty that will result if the movement is not made. If there are several candidate eye movements, the one with the highest expected value is chosen. The model is illustrated using a humanoid graphic ﬁgure that navigates on a sidewalk in a virtual urban environment. Simulations show our protocol is superior to a simple round robin scheduling mechanism.
Estimating Internal Variables and Paramters of a Learning Agent by a Particle Filter
Authors: Kazuyuki Samejima, Kenji Doya, Yasumasa Ueda, Minoru Kimura
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: When we model a higher order functions, such as learning and memory, we face a difﬁculty of comparing neural activities with hidden variables that depend on the history of sensory and motor signals and the dynam- ics of the network. Here, we propose novel method for estimating hidden variables of a learning agent, such as connection weights from sequences of observable variables. Bayesian estimation is a method to estimate the posterior probability of hidden variables from observable data sequence using a dynamic model of hidden and observable variables. In this pa- per, we apply particle ﬁlter for estimating internal parameters and meta- parameters of a reinforcement learning model. We veriﬁed the effective- ness of the method using both artiﬁcial data and real animal behavioral data.
ICA-based Clustering of Genes from Microarray Expression Data
Authors: Su-in Lee, Serafim Batzoglou
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We  propose  an  unsupervised  methodology  using  independent  component  analysis  (ICA)  to  cluster  genes  from  DNA  microarray  data.  Based  on  an  ICA  mixture  model  of  genomic  expression  patterns, linear and nonlinear ICA finds components that are specific  to  certain  biological  processes.  Genes  that  exhibit  significant  up-regulation  or  down-regulation  within  each  component  are  grouped  into  clusters.  We  test  the  statistical  significance  of  enrichment  of  gene  annotations  within  each  cluster.  ICA-based  clustering  outperformed  other  leading  methods  in  constructing  functionally  coherent  clusters  on  various  datasets.  This  result  supports our model of genomic expression data as composite effect  of  independent  biological  processes.  Comparison  of  clustering  performance  among  various  including  a  kernel-based  nonlinear  ICA  algorithm  shows  that  nonlinear  ICA  performed  the  best  for  small  datasets  and  natural-gradient  maximization-likelihood worked well for all the datasets.
On the Concentration of Expectation and Approximate Inference in Layered Networks
Authors: Xuanlong Nguyen, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We present an analysis of concentration-of-expectation phenomena in layered Bayesian networks that use generalized linear models as the local conditional probabilities. This framework encompasses a wide variety of probability distributions, including both discrete and continuous random variables. We utilize ideas from large deviation analysis and the delta method to devise and evaluate a class of approximate inference algo- rithms for layered Bayesian networks that have superior asymptotic error bounds and very fast computation time.
Identifying Structure across Pre-partitioned Data
Authors: Zvika Marx, Ido Dagan, Eli Shamir
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We propose an information-theoretic clustering approach that  incorporates a pre-known partition of the data, aiming to identify  common clusters that cut across the given partition. In the standard  clustering setting the formation of clusters is guided by a single  source of feature information. The newly utilized pre-partition  factor introduces an additional bias that counterbalances the impact  of the features whenever they become correlated with this known  partition. The resulting algorithmic framework was applied  successfully to synthetic data, as well as to identifying text-based  cross-religion correspondences.
Eigenvoice Speaker Adaptation via Composite Kernel Principal Component Analysis
Authors: James T. Kwok, Brian Mak, Simon Ho
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Eigenvoice speaker adaptation has been shown to be effective when only a small amount of adaptation data is available. At the heart of the method is principal component analysis (PCA) employed to ﬁnd the most im- portant eigenvoices. In this paper, we postulate that nonlinear PCA, in particular kernel PCA, may be even more effective. One major challenge is to map the feature-space eigenvoices back to the observation space so that the state observation likelihoods can be computed during the estima- tion of eigenvoice weights and subsequent decoding. Our solution is to compute kernel PCA using composite kernels, and we will call our new method kernel eigenvoice speaker adaptation. On the TIDIGITS corpus, we found that compared with a speaker-independent model, our kernel eigenvoice adaptation method can reduce the word error rate by 28–33% while the standard eigenvoice approach can only match the performance of the speaker-independent model.
A Summating, Exponentially-Decaying CMOS Synapse for Spiking Neural Systems
Authors: Rock Z. Shi, Timothy K. Horiuchi
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Synapses are a critical element of biologically-realistic, spike-based neu- ral computation, serving the role of communication, computation, and modiﬁcation. Many different circuit implementations of synapse func- tion exist with different computational goals in mind. In this paper we describe a new CMOS synapse design that separately controls quiescent leak current, synaptic gain, and time-constant of decay. This circuit im- plements part of a commonly-used kinetic model of synaptic conduc- tance. We show a theoretical analysis and experimental data for proto- types fabricated in a commercially-available 1.5µm CMOS process.
No Unbiased Estimator of the Variance of K-Fold Cross-Validation
Authors: Yoshua Bengio, Yves Grandvalet
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Most machine learning researchers perform quantitative experiments to estimate generalization error and compare algorithm performances. In order to draw statistically convincing conclusions, it is important to esti- mate the uncertainty of such estimates. This paper studies the estimation of uncertainty around the K-fold cross-validation estimator. The main theorem shows that there exists no universal unbiased estimator of the variance of K-fold cross-validation. An analysis based on the eigende- composition of the covariance matrix of errors helps to better understand the nature of the problem and shows that naive estimators may grossly underestimate variance, as con£rmed by numerical experiments.
GPPS: A Gaussian Process Positioning System for Cellular Networks
Authors: Anton Schwaighofer, Marian Grigoras, Volker Tresp, Clemens Hoffmann
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: In this article, we present a novel approach to solving the localization problem in cellular networks. The goal is to estimate a mobile user’s position, based on measurements of the signal strengths received from network base stations. Our solution works by building Gaussian process models for the distribution of signal strengths, as obtained in a series of calibration measurements. In the localization stage, the user’s posi- tion can be estimated by maximizing the likelihood of received signal strengths with respect to the position. We investigate the accuracy of the proposed approach on data obtained within a large indoor cellular network.
Reconstructing MEG Sources with Unknown Correlations
Authors: Maneesh Sahani, Srikantan S. Nagarajan
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Existing source location and recovery algorithms used in magnetoen- cephalographic imaging generally assume that the source activity at dif- ferent brain locations is independent or that the correlation structure is known. However, electrophysiological recordings of local ﬁeld poten- tials show strong correlations in aggregate activity over signiﬁcant dis- tances. Indeed, it seems very likely that stimulus-evoked activity would follow strongly correlated time-courses in different brain areas. Here, we present, and validate through simulations, a new approach to source reconstruction in which the correlation between sources is modelled and estimated explicitly by variational Bayesian methods, facilitating accu- rate recovery of source locations and the time-courses of their activation.
Increase Information Transfer Rates in BCI by CSP Extension to Multi-class
Authors: Guido Dornhege, Benjamin Blankertz, Gabriel Curio, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Brain-Computer Interfaces (BCI) are an interesting emerging technology that is driven by the motivation to develop an effective communication in- terface translating human intentions into a control signal for devices like computers or neuroprostheses. If this can be done bypassing the usual hu- man output pathways like peripheral nerves and muscles it can ultimately become a valuable tool for paralyzed patients. Most activity in BCI re- search is devoted to ﬁnding suitable features and algorithms to increase information transfer rates (ITRs). The present paper studies the implica- tions of using more classes, e.g., left vs. right hand vs. foot, for operating a BCI. We contribute by (1) a theoretical study showing under some mild assumptions that it is practically not useful to employ more than three or four classes, (2) two extensions of the common spatial pattern (CSP) algorithm, one interestingly based on simultaneous diagonalization, and (3) controlled EEG experiments that underline our theoretical ﬁndings and show excellent improved ITRs.
Dopamine Modulation in a Basal Ganglio-Cortical Network of Working Memory
Authors: Aaron J. Gruber, Peter Dayan, Boris S. Gutkin, Sara A. Solla
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: Dopamine exerts two classes of effect on the sustained neural activity in prefrontal cortex that underlies working memory. Direct release in the cortex increases the contrast of prefrontal neurons, enhancing the ro- bustness of storage. Release of dopamine in the striatum is associated with salient stimuli and makes medium spiny neurons bistable; this mod- ulation of the output of spiny neurons affects prefrontal cortex so as to indirectly gate access to working memory and additionally damp sensi- tivity to noise. Existing models have treated dopamine in one or other structure, or have addressed basal ganglia gating of working memory ex- clusive of dopamine effects. In this paper we combine these mechanisms and explore their joint effect. We model a memory-guided saccade task to illustrate how dopamine’s actions lead to working memory that is se- lective for salient input and has increased robustness to distraction.
ARA*: Anytime A* with Provable Bounds on Sub-Optimality
Authors: Maxim Likhachev, Geoffrey J. Gordon, Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: In real world planning problems, time for deliberation is often limited. Anytime planners are well suited for these problems: they ﬁnd a feasi- ble solution quickly and then continually work on improving it until time runs out. In this paper we propose an anytime heuristic search, ARA, which tunes its performance bound based on available search time. It starts by ﬁnding a suboptimal solution quickly using a loose bound, then tightens the bound progressively as time allows. Given enough time it ﬁnds a provably optimal solution. While improving its bound, ARA reuses previous search efforts and, as a result, is signiﬁcantly more efﬁ- cient than other anytime search methods. In addition to our theoretical analysis, we demonstrate the practical utility of ARA* with experiments on a simulated robot kinematic arm and a dynamic path planning prob- lem for an outdoor rover.
Salient Boundary Detection using Ratio Contour
Authors: Song Wang, Toshiro Kubota, Jeffrey M. Siskind
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper presents a novel graph-theoretic approach, named ratio con- tour, to extract perceptually salient boundaries from a set of noisy bound- ary fragments detected in real images. The boundary saliency is deﬁned using the Gestalt laws of closure, proximity, and continuity. This pa- per ﬁrst constructs an undirected graph with two different sets of edges: solid edges and dashed edges. The weights of solid and dashed edges measure the local saliency in and between boundary fragments, respec- tively. Then the most salient boundary is detected by searching for an optimal cycle in this graph with minimum average weight. The proposed approach guarantees the global optimality without introducing any biases related to region area or boundary length. We collect a variety of images for testing the proposed approach with encouraging results.
Semi-Definite Programming by Perceptron Learning
Authors: Thore Graepel, Ralf Herbrich, Andriy Kharechko, John S. Shawe-taylor
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We present a modiﬁed version of the perceptron learning algorithm (PLA) which solves semideﬁnite programs (SDPs) in polynomial time. The algorithm is based on the following three observations: (i) Semideﬁnite programs are linear programs with inﬁnitely many (linear) constraints; (ii) every linear program can be solved by a sequence of constraint satisfaction problems with linear constraints; (iii) in general, the perceptron learning algorithm solves a constraint satisfaction problem with linear constraints in ﬁnitely many updates. Combining the PLA with a probabilistic rescaling algorithm (which, on average, increases the size of the feasable region) results in a prob- abilistic algorithm for solving SDPs that runs in polynomial time. We present preliminary results which demonstrate that the algo- rithm works, but is not competitive with state-of-the-art interior point methods.
An Iterative Improvement Procedure for Hierarchical Clustering
Authors: David Kauchak, Sanjoy Dasgupta
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We describe a procedure which ﬁnds a hierarchical clustering by hill- climbing. The cost function we use is a hierarchical extension of the k-means cost; our local moves are tree restructurings and node reorder- ings. We show these can be accomplished efﬁciently, by exploiting spe- cial properties of squared Euclidean distances and by using techniques from scheduling algorithms.
Kernels for Structured Natural Language Data
Authors: Jun Suzuki, Yutaka Sasaki, Eisaku Maeda
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper devises a novel kernel function for structured natural language data. In the ﬁeld of Natural Language Processing, feature extraction consists of the following two steps: (1) syntactically and semantically analyzing raw data, i.e., character strings, then representing the results as discrete structures, such as parse trees and dependency graphs with part-of-speech tags; (2) creating (possibly high-dimensional) numerical feature vectors from the discrete structures. The new kernels, called Hier- archical Directed Acyclic Graph (HDAG) kernels, directly accept DAGs whose nodes can contain DAGs. HDAG data structures are needed to fully reﬂect the syntactic and semantic structures that natural language data inherently have. In this paper, we deﬁne the kernel function and show how it permits efﬁcient calculation. Experiments demonstrate that the proposed kernels are superior to existing kernel functions, e.g., se- quence kernels, tree kernels, and bag-of-words kernels.
Near-Minimax Optimal Classification with Dyadic Classification Trees
Authors: Clayton Scott, Robert Nowak
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: This paper reports on a family of computationally practical classiﬁers that converge to the Bayes error at near-minimax optimal rates for a va- riety of distributions. The classiﬁers are based on dyadic classiﬁcation trees (DCTs), which involve adaptively pruned partitions of the feature space. A key aspect of DCTs is their spatial adaptivity, which enables lo- cal (rather than global) ﬁtting of the decision boundary. Our risk analysis involves a spatial decomposition of the usual concentration inequalities, leading to a spatially adaptive, data-dependent pruning criterion. For any distribution on (X, Y ) whose Bayes decision boundary behaves locally like a Lipschitz smooth function, we show that the DCT error converges to the Bayes error at a rate within a logarithmic factor of the minimax optimal rate. We also study DCTs equipped with polynomial classiﬁca- tion rules at each leaf, and show that as the smoothness of the boundary increases their errors converge to the Bayes error at a rate approaching n−1/2, the parametric rate. We are not aware of any other practical classi- ﬁers that provide similar rate of convergence guarantees. Fast algorithms for tree pruning are discussed.
Bounded Invariance and the Formation of Place Fields
Authors: Reto Wyss, Paul F. Verschure
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: One current explanation of the view independent representation of space by the place-cells of the hippocampus is that they arise out of the summation of view dependent Gaussians. This proposal as- sumes that visual representations show bounded invariance. Here we investigate whether a recently proposed visual encoding scheme called the temporal population code can provide such representa- tions. Our analysis is based on the behavior of a simulated robot in a virtual environment containing speci(cid:12)c visual cues. Our re- sults show that the temporal population code provides a represen- tational substrate that can naturally account for the formation of place (cid:12)elds.
Online Learning of Non-stationary Sequences
Authors: Claire Monteleoni, Tommi S. Jaakkola
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: We consider an online learning scenario in which the learner can make predictions on the basis of a ﬁxed set of experts. We derive upper and lower relative loss bounds for a class of universal learning algorithms in- volving a switching dynamics over the choice of the experts. On the basis of the performance bounds we provide the optimal a priori discretiza- tion for learning the parameter that governs the switching dynamics. We demonstrate the new algorithm in the context of wireless networks.
Using Genetic Algorithms to Improve Pattern Classification Performance
Authors: Eric I. Chang, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Genetic  algorithms  were  used  to select  and  create  features  and  to select  reference  exemplar  patterns for  machine vision  and speech  pattern  classi(cid:173) fication  tasks.  For a  complex speech  recognition  task,  genetic  algorithms  required no more computation time than traditional approaches to feature  selection  but reduced  the number of input features  required  by a factor  of  five  (from 153 to 33 features).  On a difficult artificial machine-vision task,  genetic  algorithms were  able  to create  new features  (polynomial functions  of the original features)  which reduced  classification error rates from  19%  to  almost  0%.  Neural  net  and  k  nearest  neighbor  (KNN)  classifiers  were  unable to provide such low error rates using only the original features.  Ge(cid:173) netic algorithms were also used to reduce the number of reference exemplar  patterns for  a  KNN classifier.  On a  338 training pattern vowel-recognition  problem with  10  classes,  genetic  algorithms reduced  the number of stored  exemplars from 338 to 43 without significantly increasing classification er(cid:173) ror  rate.  In  all  applications,  genetic  algorithms  were  easy  to  apply  and  found  good  solutions  in  many fewer  trials  than would  be  required  by ex(cid:173) haustive search.  Run times were long, but not unreasonable.  These results  suggest  that genetic  algorithms  are  becoming practical for  pattern  classi(cid:173) fication  problems  as faster  serial  and  parallel computers  are  developed.
Optimal Filtering in the Salamander Retina
Authors: Fred Rieke, W. Geoffrey Owen, William Bialek
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The dark-adapted  visual  system can  count photons  wit h  a  reliability  lim(cid:173) ited by thermal noise  in  the rod photoreceptors - the processing circuitry  bet.ween  t.he  rod  cells  and the brain is  essentially noiseless  and  in fact  may  be  close  to  optimal.  Here  we  design  an  optimal  signal  processor  which  estimates  the  time-varying  light  intensit.y  at  the  retina  based  on  the  rod  signals.  \Ve show  that.  the first stage of optimal signal processing  involves  passing  the  rod  cell  out.put.  t.hrough  a  linear filter  with  characteristics de(cid:173) termined  entirely  by  the  rod  signal  and  noise  spectra.  This  filter  is  very  general;  in  fact  it.  is  the  first  st.age  in  any  visual  signal  processing  task  at.  10\'  photon  flux.  \Ve  iopntify  the  output  of this  first-st.age  filter  wit.h  the  intracellular  voltage  response of the  bipolar  celL  the  first  anatomical  st.age  in  retinal  signal  processing.  From  recent.  data on  tiger  salamander  phot.oreceptors  we  extract  t.he  relevant.  spect.ra  and  make  parameter-free,  quantit.ative predictions of the bipolar celll'esponse to a dim, diffuse flash.  Agreement  wit.h  experiment  is  essentially  perfect.  As  far  as  we  know  this  is  the  first  successful  predicti ve  t.heory  for  neural  dynamics. 
1
Relaxation Networks for Large Supervised Learning Problems
Authors: Joshua Alspector, Robert B. Allen, Anthony Jayakumar, Torsten Zeppenfeld, Ronny Meir
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Feedback  connections  are  required  so  that  the  teacher  signal  on  the  output  neurons  can  modify  weights  during  supervised  learning.  Relaxation  methods  are  needed  for  learning  static  patterns  with  full-time  feedback  connections.  Feedback  network  learning  techniques  have  not  achieved  wide  popularity  because  of the  still greater  computational efficiency  of back-propagation.  We  show by simulation that relaxation networks of the kind we  are implementing in  VLSI  are  capable  of  learning  large  problems  just  like  back-propagation  networks.  A microchip incorporates deterministic mean-field theory learning as  well  as  stochastic  Boltzmann  learning.  A  multiple-chip  electronic  system  implementing  these  networks  will  make  high-speed  parallel  learning  in  them  feasible in the future.
Neural Networks Structured for Control Application to Aircraft Landing
Authors: Charles Schley, Yves Chauvin, Van Henkle, Richard Golden
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We present a generic neural network architecture capable of con(cid:173) trolling non-linear plants. The network is composed of dynamic.  parallel, linear maps gated by non-linear switches. Using a recur(cid:173) rent form of the back-propagation algorithm, control is achieved  by optimizing the control gains and task-adapted switch parame(cid:173) ters. A mean quadratic cost function computed across a nominal  plant trajectory is minimized along with performance constraint  penalties. The approach is demonstrated for a control task con(cid:173) sisting of landing a commercial aircraft in difficult wind conditions.  We show that the network yields excellent performance while re(cid:173) maining within acceptable damping response constraints.
Dynamics of Generalization in Linear Perceptrons
Authors: Anders Krogh, John A. Hertz
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We study the evolution of the generalization ability of a  simple linear per(cid:173) ceptron with N  inputs which learns to imitate a  "teacher perceptron".  The  system is  trained  on p  =  aN  binary  example  inputs  and  the  generaliza(cid:173) tion  ability measured by testing for  agreement with  the teacher on all 2N  possible  binary input  patterns.  The dynamics may  be solved  analytically  and  exhibits  a  phase  transition  from  imperfect  to  perfect  generalization  at  a  =  1.  Except  at  this  point  the  generalization  ability  approaches  its  asymptotic  value  exponentially,  with critical slowing  down near  the  tran(cid:173) sition;  the  relaxation  time  is  ex  (1  - y'a)-2.  Right  at  the  critical  point,  1  the  approach  to  perfect  generalization  follows  a  power  law  ex  t - '2.  In  the presence of noise,  the generalization ability is  degraded  by  an amount  ex  (va - 1)-1 just above a  =  1.
Translating Locative Prepositions
Authors: Paul W. Munro, Mary Tabasko
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: A network was trained by back propagation to map locative expressions  of the form "noun-preposition-noun" to a semantic representation, as in  Cosic  and  Munro  (1988).  The  network's  performance  was  analyzed  over  several  simulations  with  training  sets  in  both  English  and  German.  Translation  of prepositions  was  attempted  by  presenting  a  locative expression to a  network trained in one language to generate a  semantic representation; the semantic representation was  then presented  to the network trained in the other language to generate the appropriate  preposition.
An Analog VLSI Chip for Finding Edges from Zero-crossings
Authors: Wyeth Bair, Christof Koch
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We  have  designed  and  tested  a  one-dimensional  64  pixel,  analog  CMOS  VLSI chip which localizes intensity edges in real-time.  This device exploits  on-chip photoreceptors and the natural filtering properties of resistive net(cid:173) works to implement a  scheme similar  to  and  motivated  by  the Difference  of Gaussians (DOG) operator proposed by Marr and Hildreth (1980).  Our  chip computes the  zero-crossings associated  with  the difference of two ex(cid:173) ponential weighting functions.  If the  derivative  across  this  zero-crossing  is  above  a  threshold,  an edge  is  reported.  Simulations  indicate  that  this  technique will  extend well to two dimensions.
Comparison of three classification techniques: CART, C4.5 and Multi-Layer Perceptrons
Authors: A. C. Tsoi, R. A. Pearson
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: In this paper, after some introductory remarks into the classification prob(cid:173) lem as considered in various research communities, and some discussions  concerning some of the reasons for ascertaining the performances of the  three chosen algorithms, viz., CART (Classification and Regression Tree),  C4.5 (one of the more recent versions of a popular induction tree tech(cid:173) nique known as ID3), and a multi-layer perceptron (MLP), it is proposed  to compare the performances of these algorithms under two criteria: classi(cid:173) fication and generalisation. It is found that, in general, the MLP has better  classification and generalisation accuracies compared with the other two  algorithms.
Self-organization of Hebbian Synapses in Hippocampal Neurons
Authors: Thomas H. Brown, Zachary F. Mainen, Anthony M. Zador, Brenda J. Claiborne
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We are exploring the significance of biological complexity for neuronal  computation.  Here we demonstrate that Hebbian synapses in realistical(cid:173) ly-modeled  hippocampal  pyramidal  cells  may  give  rise  to  two  novel  forms of self -organization in response to structured synaptic input.  First,  on the basis of the electrotonic relationships between synaptic contacts,  a cell may become tuned to a small subset of its input space.  Second, the  same mechanisms may produce  clusters of potentiated synapses across  the space of the dendrites.  The latter type of self-organization may be  functionally  significant in  the presence of nonlinear dendritic  conduc(cid:173) tances.
Connection Topology and Dynamics in Lateral Inhibition Networks
Authors: C.M Marcus, F. R. Waugh, R. M. Westervelt
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: We show analytically how the stability of two-dimensional lateral  inhibition neural networks depends on the local connection topology.  For various network topologies, we calculate the critical time delay for  the onset of oscillation in continuous-time networks and present  analytic phase diagrams characterizing the dynamics of discrete-time  networks.
A Delay-Line Based Motion Detection Chip
Authors: Tim Horiuchi, John Lazzaro, Andrew Moore, Christof Koch
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Inspired by a visual motion detection model for the ra.bbit retina  and by a computational architecture used for early audition in the  barn owl, we have designed a chip that employs a correlation model  to report the one-dimensional field motion of a scene in real time.  Using subthreshold analog VLSI techniques, we have fabricated and  successfully tested a 8000 transistor chip using a standard MOSIS  process.
Applications of Neural Networks in Video Signal Processing
Authors: John C. Pearson, Clay D. Spence, Ronald Sverdlove
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Although color TV is an established technology, there are a number of  longstanding problems for which neural networks may be suited. Impulse  noise is such a problem, and a modular neural network approach is pre(cid:173) sented in this paper. The training and analysis was done on conventional  computers, while real-time simulations were performed on a massively par(cid:173) allel computer called the Princeton Engine. The network approach was  compared to a conventional alternative, a median filter. Real-time simula(cid:173) tions and quantitative analysis demonstrated the technical superiority of  the neural system. Ongoing work is investigating the complexity and cost  of implementing this system in hardware. 
1 THE POTENTIAL FOR NEURAL NETWORKS IN 
CONSUMER ELECTRONICS 
Neural networks are most often considered for application in emerging new tech(cid:173) nologies, such as speech recognition, machine vision, and robotics. The fundamental  ideas behind these technologies are still being developed, and it will be some time  before products containing neural networks are manufactured. As a result, research  in these areas will not drive the development of inexpensive neural network hard(cid:173) ware which could serve as a catalyst for the field of neural networks in general. 
In contrast, neural networks are rarely considered for application in mature tech(cid:173) nologies, such as consumer electronics. These technologies are based on established  principles of information processing and communication, and they are used in mil(cid:173) lions of products per year. The embedding of neural networks within such mass-
Grouping Contours by Iterated Pairing Network
Authors: Amnon Shashua, Shimon Ullman
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: Shimon Ullman 
We describe in this paper a network that performs grouping of image con(cid:173) tours. The input to the net are fragments of image contours, and the  output is the partitioning of the fragments into groups, together with a  saliency measure for each group. The grouping is based on a measure of  overall length and curvature. The network decomposes the overall opti(cid:173) mization problem into independent optimal pairing problems performed  at each node. The resulting computation maps into a uniform locally  connected network of simple computing elements. 
1 The Problenl: Contour Grouping 
A problem that often arises in visual information processing is the linking of con(cid:173) tour fragments into optimal groups. For example, certain subsets of contours spon(cid:173) taneously form perceptual groups, as illustrated in Fig. 1, and are often detected  immediately without scanning the image in a systematic manner. Grouping process  of this type are likely to play an important role in object recognition by segmenting  the image and selecting image structures that are likely to correspond to objects of  interest in the scene. 
'Ve propose that some form of autonomous grouping is performed at an early stage  based on geometrical characteristics, that are independent of the identity of objects  to be selected. The grouping process is governed by the notion of saliency in a way  that priority is given to forming salient groups at the expense of potentially less  salient ones. This general notion can again be illustrated by Fig. 1; it appears that  certain groups spontaneously emerge, while grouping decisions concerning the less  salient parts of the image may remain unresolved. As we shall see, the computation  below exhibits a similar behavior. 
We define a grouping of the image contours as the formation of a set of disjoint
Simulation of the Neocognitron on a CCD Parallel Processing Architecture
Authors: Michael L. Chuang, Alice M. Chiang
Published in: Advances in Neural Information Processing Systems 3  (NIPS 1990) (Main Conference Track), 1990

Abstract: The neocognitron  is  a  neural network for  pattern recognition  and feature  extraction.  An  analog  CCD  parallel  processing  architecture  developed  at Lincoln Laboratory is  particularly  well suited to the computational re(cid:173) quirements of shared-weight networks such as the neocognitron, and imple(cid:173) mentation of the neocognitron using the CCD architecture was simulated.  A  modification  to  the  neocognitron  training  procedure,  which  improves  network performance under the limited arithmetic  precision that would be  imposed  by the CCD  architecture,  is  presented.