mtybilly/Instruct2DS huggingface dataset:
Unsupervised Learning of Visual Sense Models for Polysemous Words
Authors: Kate Saenko, Trevor Darrell
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Polysemy is a problem for methods that exploit image search engines to build object category models. Existing unsupervised approaches do not take word sense into consideration. We propose a new method that uses a dictionary to learn models of visual word sense from a large collection of unlabeled web data. The use of LDA to discover a latent sense space makes the model robust despite the very limited nature of dictionary definitions. The definitions are used to learn a distribution in the latent space that best represents a sense. The algorithm then uses the text surrounding image links to retrieve images with high probability of a particular dictionary sense. An object classifier is trained on the resulting sense-specific images. We evaluate our method on a dataset obtained by searching the web for polysemous words. Category classification experiments show that our dictionary-based approach outperforms baseline methods.
Extended Grassmann Kernels for Subspace-Based Learning
Authors: Jihun Hamm, Daniel D. Lee
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Subspace-based learning problems involve data whose elements are linear subspaces of a vector space. To handle such data structures, Grassmann kernels have been proposed and used previously. In this paper, we analyze the relationship between Grassmann kernels and probabilistic similarity measures. Firstly, we show that the KL distance in the limit yields the Projection kernel on the Grassmann manifold, whereas the Bhattacharyya kernel becomes trivial in the limit and is suboptimal for subspace-based problems. Secondly, based on our analysis of the KL distance, we propose extensions of the Projection kernel which can be extended to the set of affine as well as scaled subspaces. We demonstrate the advantages of these extended kernels for classification and recognition tasks with Support Vector Machines and Kernel Discriminant Analysis using synthetic and real image databases.
Implicit Mixtures of Restricted Boltzmann Machines
Authors: Vinod Nair, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We present a mixture model whose components are Restricted Boltzmann Machines (RBMs). This possibility has not been considered before because computing the partition function of an RBM is intractable, which appears to make learning a mixture of RBMs intractable as well. Surprisingly, when formulated as a third-order Boltzmann machine, such a mixture model can be learned tractably using contrastive divergence. The energy function of the model captures three-way interactions among visible units, hidden units, and a single hidden multinomial unit that represents the cluster labels. The distinguishing feature of this model is that, unlike other mixture models, the mixing proportions are not explicitly parameterized. Instead, they are defined implicitly via the energy function and depend on all the parameters in the model. We present results for the MNIST and NORB datasets showing that the implicit mixture of RBMs learns clusters that reflect the class structure in the data.
Self-organization using synaptic plasticity
Authors: Vicençc Gómez, Andreas Kaltenbrunner, Vicente López, Hilbert J. Kappen
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Large networks of spiking neurons show abrupt changes in their collective dynamics resembling phase transitions studied in statistical physics. An example of this phenomenon is the transition from irregular, noise-driven dynamics to regular, self-sustained behavior observed in networks of integrate-and-fire neurons as the interaction strength between the neurons increases. In this work we show how a network of spiking neurons is able to self-organize towards a critical state for which the range of possible inter-spike-intervals (dynamic range) is maximized. Self-organization occurs via synaptic dynamics that we analytically derive. The resulting plasticity rule is defined locally so that global homeostasis near the critical state is achieved by local regulation of individual synapses.
Interpreting the neural code with Formal Concept Analysis
Authors: Dominik Endres, Peter Foldiak
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We propose a novel application of Formal Concept Analysis (FCA) to neural decoding: instead of just trying to figure out which stimulus was presented, we demonstrate how to explore the semantic relationships between the neural representation of large sets of stimuli. FCA provides a way of displaying and interpreting such relationships via concept lattices. We explore the effects of neural code sparsity on the lattice. We then analyze neurophysiological data from high-level visual cortical area STSa, using an exact Bayesian approach to construct the formal context needed by FCA. Prominent features of the resulting concept lattices are discussed, including indications for a product-of-experts code in real neurons.
Global Ranking Using Continuous Conditional Random Fields
Authors: Tao Qin, Tie-yan Liu, Xu-dong Zhang, De-sheng Wang, Hang Li
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: This paper studies global ranking problem by learning to rank methods. Conventional learning to rank methods are usually designed for `local ranking', in the sense that the ranking model is defined on a single object, for example, a document in information retrieval. For many applications, this is a very loose approximation. Relations always exist between objects and it is better to define the ranking model as a function on all the objects to be ranked (i.e., the relations are also included). This paper refers to the problem as global ranking and proposes employing a Continuous Conditional Random Fields (CRF) for conducting the learning task. The Continuous CRF model is defined as a conditional probability distribution over ranking scores of objects conditioned on the objects. It can naturally represent the content information of objects as well as the relation information between objects, necessary for global ranking. Taking two specific information retrieval tasks as examples, the paper shows how the Continuous CRF method can perform global ranking better than baselines.
Improving on Expectation Propagation
Authors: Manfred Opper, Ulrich Paquet, Ole Winther
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We develop as series of corrections to Expectation Propagation (EP), which is one of the most popular methods for approximate probabilistic inference. These corrections can lead to improvements of the inference approximation or serve as a sanity check, indicating when EP yields unrealiable results.
Learning a discriminative hidden part model for human action recognition
Authors: Yang Wang, Greg Mori
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We present a discriminative part-based approach for human action recognition from video sequences using motion features. Our model is based on the recently proposed hidden conditional random field~(hCRF) for object recognition. Similar to hCRF for object recognition, we model a human action by a flexible constellation of parts conditioned on image observations. Different from object recognition, our model combines both large-scale global features and local patch features to distinguish various actions. Our experimental results show that our model is comparable to other state-of-the-art approaches in action recognition. In particular, our experimental results demonstrate that combining large-scale global features and local patch features performs significantly better than directly applying hCRF on local patches alone.
Adaptive Template Matching with Shift-Invariant Semi-NMF
Authors: Jonathan L. Roux, Alain D. Cheveigné, Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: How does one extract unknown but stereotypical events that are linearly superimposed within a signal with variable latencies and variable amplitudes? One could think of using template matching or matching pursuit to find the arbitrarily shifted linear components. However, traditional matching approaches require that the templates be known a priori. To overcome this restriction we use instead semi Non-Negative Matrix Factorization (semi-NMF) that we extend to allow for time shifts when matching the templates to the signal. The algorithm estimates templates directly from the data along with their non-negative amplitudes. The resulting method can be thought of as an adaptive template matching procedure. We demonstrate the procedure on the task of extracting spikes from single channel extracellular recordings. On these data the algorithm essentially performs spike detection and unsupervised spike clustering. Results on simulated data and extracellular recordings indicate that the method performs well for signal-to-noise ratios of 6dB or higher and that spike templates are recovered accurately provided they are sufficiently different.
Extracting State Transition Dynamics from Multiple Spike Trains with Correlated Poisson HMM
Authors: Kentaro Katahira, Jun Nishikawa, Kazuo Okanoya, Masato Okada
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Neural activity is non-stationary and varies across time. Hidden Markov Models (HMMs) have been used to track the state transition among quasi-stationary discrete neural states. Within this context, independent Poisson models have been used for the output distribution of HMMs; hence, the model is incapable of tracking the change in correlation without modulating the firing rate. To achieve this, we applied a multivariate Poisson distribution with correlation terms for the output distribution of HMMs. We formulated a Variational Bayes (VB) inference for the model. The VB could automatically determine the appropriate number of hidden states and correlation types while avoiding the overlearning problem. We developed an efficient algorithm for computing posteriors using the recursive relationship of a multivariate Poisson distribution. We demonstrated the performance of our method on synthetic data and a real spike train recorded from a songbird.
Partially Observed Maximum Entropy Discrimination Markov Networks
Authors: Jun Zhu, Eric P. Xing, Bo Zhang
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Learning graphical models with hidden variables can offer semantic insights to complex data and lead to salient structured predictors without relying on expensive, sometime unattainable fully annotated training data. While likelihood-based methods have been extensively explored, to our knowledge, learning structured prediction models with latent variables based on the max-margin principle remains largely an open problem. In this paper, we present a partially observed Maximum Entropy Discrimination Markov Network (PoMEN) model that attempts to combine the advantages of Bayesian and margin based paradigms for learning Markov networks from partially labeled data. PoMEN leads to an averaging prediction rule that resembles a Bayes predictor that is more robust to overfitting, but is also built on the desirable discriminative laws resemble those of the M$^3$N. We develop an EM-style algorithm utilizing existing convex optimization algorithms for M$^3$N as a subroutine. We demonstrate competent performance of PoMEN over existing methods on a real-world web data extraction task.
Playing Pinball with non-invasive BCI
Authors: Matthias Krauledat, Konrad Grzeska, Max Sagebaum, Benjamin Blankertz, Carmen Vidaurre, Klaus-Robert Müller, Michael Schröder
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Compared to invasive Brain-Computer Interfaces (BCI), non-invasive BCI systems based on Electroencephalogram (EEG) signals have not been applied successfully for complex control tasks. In the present study, however, we demonstrate this is possible and report on the interaction of a human subject with a complex real device: a pinball machine. First results in this single subject study clearly show that fast and well-timed control well beyond chance level is possible, even though the environment is extremely rich and requires complex predictive behavior. Using machine learning methods for mental state decoding, BCI-based pinball control is possible within the first session without the necessity to employ lengthy subject training. While the current study is still of anecdotal nature, it clearly shows that very compelling control with excellent timing and dynamics is possible for a non-invasive BCI.
Logistic Normal Priors for Unsupervised Probabilistic Grammar Induction
Authors: Shay B. Cohen, Kevin Gimpel, Noah A. Smith
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We explore a new Bayesian model for probabilistic grammars, a family of distributions over discrete structures that includes hidden Markov models and probabilistic context-free grammars. Our model extends the correlated topic model framework to probabilistic grammars, exploiting the logistic normal distribution as a prior over the grammar parameters. We derive a variational EM algorithm for that model, and then experiment with the task of unsupervised grammar induction for natural language dependency parsing. We show that our model achieves superior results over previous models that use different priors.
Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text
Authors: Yi Zhang, Artur Dubrawski, Jeff G. Schneider
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: In this paper, we address the question of what kind of knowledge is generally transferable from unlabeled text. We suggest and analyze the semantic correlation of words as a generally transferable structure of the language and propose a new method to learn this structure using an appropriately chosen latent variable model. This semantic correlation contains structural information of the language space and can be used to control the joint shrinkage of model parameters for any specific task in the same space through regularization. In an empirical study, we construct 190 different text classification tasks from a real-world benchmark, and the unlabeled documents are a mixture from all these tasks. We test the ability of various algorithms to use the mixed unlabeled text to enhance all classification tasks. Empirical results show that the proposed approach is a reliable and scalable method for semi-supervised learning, regardless of the source of unlabeled data, the specific task to be enhanced, and the prediction model used.
Online Optimization in X-Armed Bandits
Authors: Sébastien Bubeck, Gilles Stoltz, Csaba Szepesvári, Rémi Munos
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We consider a generalization of stochastic bandit problems where the set of arms, X, is allowed to be a generic topological space. We constraint the mean-payoff function with a dissimilarity function over X in a way that is more general than Lipschitz. We construct an arm selection policy whose regret improves upon previous result for a large class of problems. In particular, our results imply that if X is the unit hypercube in a Euclidean space and the mean-payoff function has a finite number of global maxima around which the behavior of the function is locally Hölder with a known exponent, then the expected regret is bounded up to a logarithmic factor by $n$, i.e., the rate of the growth of the regret is independent of the dimension of the space. Moreover, we prove the minimax optimality of our algorithm for the class of mean-payoff functions we consider.
Variational Mixture of Gaussian Process Experts
Authors: Chao Yuan, Claus Neubauer
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Mixture of Gaussian processes models extended a single Gaussian process with ability of modeling multi-modal data and reduction of training complexity. Previous inference algorithms for these models are mostly based on Gibbs sampling, which can be very slow, particularly for large-scale data sets. We present a new generative mixture of experts model. Each expert is still a Gaussian process but is reformulated by a linear model. This breaks the dependency among training outputs and enables us to use a much faster variational Bayesian algorithm for training. Our gating network is more flexible than previous generative approaches as inputs for each expert are modeled by a Gaussian mixture model. The number of experts and number of Gaussian components for an expert are inferred automatically. A variety of tests show the advantages of our method.
On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost
Authors: Hamed Masnadi-shirazi, Nuno Vasconcelos
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: The machine learning problem of classifier design is studied from the perspective of probability elicitation, in statistics. This shows that the standard approach of proceeding from the specification of a loss, to the minimization of conditional risk is overly restrictive. It is shown that a better alternative is to start from the specification of a functional form for the minimum conditional risk, and derive the loss function. This has various consequences of practical interest, such as showing that 1) the widely adopted practice of relying on convex loss functions is unnecessary, and 2) many new losses can be derived for classification problems. These points are illustrated by the derivation of a new loss which is not convex, but does not compromise the computational tractability of classifier design, and is robust to the contamination of data with outliers. A new boosting algorithm, SavageBoost, is derived for the minimization of this loss. Experimental results show that it is indeed less sensitive to outliers than conventional methods, such as Ada, Real, or LogitBoost, and converges in fewer iterations.
Non-stationary dynamic Bayesian networks
Authors: Joshua W. Robinson, Alexander J. Hartemink
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: A principled mechanism for identifying conditional dependencies in time-series data is provided through structure learning of dynamic Bayesian networks (DBNs). An important assumption of DBN structure learning is that the data are generated by a stationary processâan assumption that is not true in many important settings. In this paper, we introduce a new class of graphical models called non-stationary dynamic Bayesian networks, in which the conditional dependence structure of the underlying data-generation process is permitted to change over time. Non-stationary dynamic Bayesian networks represent a new framework for studying problems in which the structure of a network is evolving over time. We define the non-stationary DBN model, present an MCMC sampling algorithm for learning the structure of the model from time-series data under different assumptions, and demonstrate the effectiveness of the algorithm on both simulated and biological data.
Nonlinear causal discovery with additive noise models
Authors: Patrik O. Hoyer, Dominik Janzing, Joris M. Mooij, Jonas Peters, Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: The discovery of causal relationships between a set of observed variables is a fundamental problem in science. For continuous-valued data linear acyclic causal models are often used because these models are well understood and there are well-known methods to fit them to data. In reality, of course, many causal relationships are more or less nonlinear, raising some doubts as to the applicability and usefulness of purely linear methods. In this contribution we show that in fact the basic linear framework can be generalized to nonlinear models with additive noise. In this extended framework, nonlinearities in the data-generating process are in fact a blessing rather than a curse, as they typically provide information on the underlying causal system and allow more aspects of the true data-generating mechanisms to be identified. In addition to theoretical results we show simulations and some simple real data experiments illustrating the identification power provided by nonlinearities.
Simple Local Models for Complex Dynamical Systems
Authors: Erik Talvitie, Satinder P. Singh
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We present a novel mathematical formalism for the idea of a local model,'' a model of a potentially complex dynamical system that makes only certain predictions in only certain situations. As a result of its restricted responsibilities, a local model may be far simpler than a complete model of the system. We then show how one might combine several local models to produce a more detailed model. We demonstrate our ability to learn a collection of local models on a large-scale example and do a preliminary empirical comparison of learning a collection of local models and some other model learning methods."
Fitted Q-iteration by Advantage Weighted Regression
Authors: Gerhard Neumann, Jan R. Peters
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Recently, fitted Q-iteration (FQI) based methods have become more popular due to their increased sample efficiency, a more stable learning process and the higher quality of the resulting policy. However, these methods remain hard to use for continuous action spaces which frequently occur in real-world tasks, e.g., in robotics and other technical applications. The greedy action selection commonly used for the policy improvement step is particularly problematic as it is expensive for continuous actions, can cause an unstable learning process, introduces an optimization bias and results in highly non-smooth policies unsuitable for real-world systems. In this paper, we show that by using a soft-greedy action selection the policy improvement step used in FQI can be simplified to an inexpensive advantage-weighted regression. With this result, we are able to derive a new, computationally efficient FQI algorithm which can even deal with high dimensional action spaces.
On the Generalization Ability of Online Strongly Convex Programming Algorithms
Authors: Sham M. Kakade, Ambuj Tewari
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: This paper examines the generalization properties of online convex programming algorithms when the loss function is Lipschitz and strongly convex. Our main result is a sharp bound, that holds with high probability, on the excess risk of the output of an online algorithm in terms of the average regret. This allows one to use recent algorithms with logarithmic cumulative regret guarantees to achieve fast convergence rates for the excess risk with high probability. The bound also solves an open problem regarding the convergence rate of {\pegasos}, a recently proposed method for solving the SVM optimization problem.
Multiscale Random Fields with Application to Contour Grouping
Authors: Longin J. Latecki, Chengen Lu, Marc Sobel, Xiang Bai
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We introduce a new interpretation of multiscale random fields (MSRFs) that admits efficient optimization in the framework of regular (single level) random fields (RFs). It is based on a new operator, called append, that combines sets of random variables (RVs) to single RVs. We assume that a MSRF can be decomposed into disjoint trees that link RVs at different pyramid levels. The append operator is then applied to map RVs in each tree structure to a single RV. We demonstrate the usefulness of the proposed approach on a challenging task involving grouping contours of target shapes in images. MSRFs provide a natural representation of multiscale contour models, which are needed in order to cope with unstable contour decompositions. The append operator allows us to find optimal image labels using the classical framework of relaxation labeling, Alternative methods like Markov Chain Monte Carlo (MCMC) could also be used.
Modeling Short-term Noise Dependence of Spike Counts in Macaque Prefrontal Cortex
Authors: Arno Onken, Steffen Grünewälder, Matthias Munk, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Correlations between spike counts are often used to analyze neural coding. The noise is typically assumed to be Gaussian. Yet, this assumption is often inappropriate, especially for low spike counts. In this study, we present copulas as an alternative approach. With copulas it is possible to use arbitrary marginal distributions such as Poisson or negative binomial that are better suited for modeling noise distributions of spike counts. Furthermore, copulas place a wide range of dependence structures at the disposal and can be used to analyze higher order interactions. We develop a framework to analyze spike count data by means of copulas. Methods for parameter inference based on maximum likelihood estimates and for computation of Shannon entropy are provided. We apply the method to our data recorded from macaque prefrontal cortex. The data analysis leads to three significant findings: (1) copula-based distributions provide better fits than discretized multivariate normal distributions; (2) negative binomial margins fit the data better than Poisson margins; and (3) a dependence model that includes only pairwise interactions overestimates the information entropy by at least 19% compared to the model with higher order interactions.
Predictive Indexing for Fast Search
Authors: Sharad Goel, John Langford, Alexander L. Strehl
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We tackle the computational problem of query-conditioned search. Given a machine-learned scoring rule and a query distribution, we build a predictive index by precomputing lists of potential results sorted based on an expected score of the result over future queries. The predictive index datastructure supports an anytime algorithm for approximate retrieval of the top elements. The general approach is applicable to webpage ranking, internet advertisement, and approximate nearest neighbor search. It is particularly effective in settings where standard techniques (e.g., inverted indices) are intractable. We experimentally find substantial improvement over existing methods for internet advertisement and approximate nearest neighbors.
Human Active Learning
Authors: Rui M. Castro, Charles Kalish, Robert Nowak, Ruichen Qian, Tim Rogers, Xiaojin Zhu
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We investigate a topic at the interface of machine learning and cognitive science. Human active learning, where learners can actively query the world for information, is contrasted with passive learning from random examples. Furthermore, we compare human active learning performance with predictions from statistical learning theory. We conduct a series of human category learning experiments inspired by a machine learning task for which active and passive learning error bounds are well understood, and dramatically distinct. Our results indicate that humans are capable of actively selecting informative queries, and in doing so learn better and faster than if they are given random training data, as predicted by learning theory. However, the improvement over passive learning is not as dramatic as that achieved by machine active learning algorithms. To the best of our knowledge, this is the first quantitative study comparing human category learning in active versus passive settings.
Clustered Multi-Task Learning: A Convex Formulation
Authors: Laurent Jacob, Jean-philippe Vert, Francis R. Bach
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: In multi-task learning several related tasks are considered simultaneously, with the hope that by an appropriate sharing of information across tasks, each task may benefit from the others. In the context of learning linear functions for supervised classification or regression, this can be achieved by including a priori information about the weight vectors associated with the tasks, and how they are expected to be related to each other. In this paper, we assume that tasks are clustered into groups, which are unknown beforehand, and that tasks within a group have similar weight vectors. We design a new spectral norm that encodes this a priori assumption, without the prior knowledge of the partition of tasks into groups, resulting in a new convex optimization formulation for multi-task learning. We show in simulations on synthetic examples and on the iedb MHC-I binding dataset, that our approach outperforms well-known convex methods for multi-task learning, as well as related non convex methods dedicated to the same problem.
Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation
Authors: Dotan D. Castro, Dmitry Volkinshtein, Ron Meir
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Actor-critic algorithms for reinforcement learning are achieving renewed popularity due to their good convergence properties in situations where other approaches often fail (e.g., when function approximation is involved). Interestingly, there is growing evidence that actor-critic approaches based on phasic dopamine signals play a key role in biological learning through the cortical and basal ganglia. We derive a temporal difference based actor critic learning algorithm, for which convergence can be proved without assuming separate time scales for the actor and the critic. The approach is demonstrated by applying it to networks of spiking neurons. The established relation between phasic dopamine and the temporal difference signal lends support to the biological relevance of such algorithms.
One sketch for all: Theory and Application of Conditional Random Sampling
Authors: Ping Li, Kenneth W. Church, Trevor J. Hastie
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: Conditional Random Sampling (CRS) was originally proposed for efficiently computing pairwise ($l_2$, $l_1$) distances, in static, large-scale, and sparse data sets such as text and Web data. It was previously presented using a heuristic argument. This study extends CRS to handle dynamic or streaming data, which much better reflect the real-world situation than assuming static data. Compared with other known sketching algorithms for dimension reductions such as stable random projections, CRS exhibits a significant advantage in that it is ``one-sketch-for-all.'' In particular, we demonstrate that CRS can be applied to efficiently compute the $l_p$ distance and the Hilbertian metrics, both are popular in machine learning. Although a fully rigorous analysis of CRS is difficult, we prove that, with a simple modification, CRS is rigorous at least for an important application of computing Hamming norms. A generic estimator and an approximate variance formula are provided and tested on various applications, for computing Hamming norms, Hamming distances, and $\chi^2$ distances.
The Mondrian Process
Authors: Daniel M. Roy, Yee W. Teh
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We describe a novel stochastic process that can be used to construct a multidimensional generalization of the stick-breaking process and which is related to the classic stick breaking process described by Sethuraman1994 in one dimension. We describe how the process can be applied to relational data modeling using the de Finetti representation for infinitely and partially exchangeable arrays.
Scalable Algorithms for String Kernels with Inexact Matching
Authors: Pavel P. Kuksa, Pai-hsi Huang, Vladimir Pavlovic
Published in: Advances in Neural Information Processing Systems 21  (NIPS 2008) (Main Conference Track), 2008

Abstract: We present a new family of linear time algorithms based on sufficient statistics for string comparison with mismatches under the string kernels framework. Our algorithms improve theoretical complexity bounds of existing approaches while scaling well with respect to the sequence alphabet size, the number of allowed mismatches and the size of the dataset. In particular, on large alphabets with loose mismatch constraints our algorithms are several orders of magnitude faster than the existing algorithms for string comparison under the mismatch similarity measure. We evaluate our algorithms on synthetic data and real applications in music genre classification, protein remote homology detection and protein fold prediction. The scalability of the algorithms allows us to consider complex sequence transformations, modeled using longer string features and larger numbers of mismatches, leading to a state-of-the-art performance with significantly reduced running times.
Beating a Defender in Robotic Soccer: Memory-Based Learning of a Continuous Function
Authors: Peter Stone, Manuela M. Veloso
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Learning how to adjust to an opponent's position is critical to  the success of having intelligent agents collaborating towards the  achievement of specific tasks in unfriendly environments. This pa(cid:173) per describes our work on a Memory-based technique for to choose  an action based on a continuous-valued state attribute indicating  the position of an opponent. We investigate the question of how an  agent performs in nondeterministic variations of the training situ(cid:173) ations. Our experiments indicate that when the random variations  fall within some bound of the initial training, the agent performs  better with some initial training rather than from a tabula-rasa.
A Neural Network Autoassociator for Induction Motor Failure Prediction
Authors: Thomas Petsche, Angelo Marcantonio, Christian Darken, Stephen Jose Hanson, Gary M. Kuhn, N. Iwan Santoso
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  present results  on  the  use  of neural  network based  autoassociators  which  act  as  novelty  or anomaly  detectors  to  detect  imminent  motor  failures.  The autoassociator is  trained  to  reconstruct  spectra obtained  from the healthy motor.  In laboratory tests, we have demonstrated that the  trained autoassociator has a small reconstruction error on measurements  recorded from healthy motors but a larger error on those recorded from a  motor with a fault.  We have designed and built a motor monitoring system  using an  autoassociator for anomaly detection and are in the process of  testing the system at three industrial and commercial sites.
The Gamma MLP for Speech Phoneme Recognition
Authors: Steve Lawrence, Ah Chung Tsoi, Andrew D. Back
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  define  a  Gamma  multi-layer  perceptron  (MLP)  as  an  MLP  with the usual synaptic weights replaced by gamma filters  (as pro(cid:173) posed by de Vries and Principe  (de Vries and Principe,  1992)) and  associated  gain  terms  throughout  all  layers.  We  derive  gradient  descent  update equations and apply  the  model  to  the recognition  of speech  phonemes.  We  find  that  both  the  inclusion  of gamma  filters  in  all  layers,  and  the  inclusion  of synaptic  gains,  improves  the  performance  of the  Gamma  MLP.  We  compare  the  Gamma  MLP  with  TDNN,  Back-Tsoi FIR MLP,  and Back-Tsoi I1R  MLP  architectures, and a  local approximation scheme.  We find  that the  Gamma MLP results in an substantial reduction  in error rates.
Laterally Interconnected Self-Organizing Maps in Hand-Written Digit Recognition
Authors: Yoonsuck Choe, Joseph Sirosh, Risto Miikkulainen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: An  application  of  laterally  interconnected  self-organizing  maps  (LISSOM)  to handwritten  digit recognition  is  presented.  The lat(cid:173) eral connections  learn the correlations of activity between units on  the  map.  The  resulting  excitatory  connections  focus  the  activity  into local patches and the inhibitory connections decorrelate redun(cid:173) dant activity on the map.  The map thus forms internal representa(cid:173) tions that are easy to recognize with e.g.  a perceptron network.  The  recognition rate on a subset of NIST database 3 is 4.0% higher with  LISSOM  than  with  a  regular  Self-Organizing  Map  (SOM)  as  the  front  end,  and 15.8% higher than recognition of raw input bitmaps  directly.  These results form a promising starting point for  building  pattern  recognition systems with a  LISSOM  map as  a  front  end.
Improved Silicon Cochlea using Compatible Lateral Bipolar Transistors
Authors: André van Schaik, Eric Fragnière, Eric A. Vittoz
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Analog  electronic  cochlear  models  need  exponentially  scaled  filters.  CMOS  Compatible  Lateral  Bipolar  Transistors  (CLBTs)  can  create  exponentially scaled currents when biased using a resistive line with a  voltage  difference  between both  ends  of the  line.  Since these  CLBTs  are  independent  of  the  CMOS  threshold  voltage,  current  sources  implemented  with  CLBTs  are  much  better  matched  than  current  sources  created  with  MOS  transistors  operated  in  weak  inversion.  Measurements  from  integrated  test  chips  are  shown  to  verify  the  improved matching.
Parallel Optimization of Motion Controllers via Policy Iteration
Authors: Jefferson A. Coelho Jr., R. Sitaraman, Roderic A. Grupen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This paper describes a  policy iteration algorithm for optimizing the  performance  of a  harmonic function-based  controller  with  respect  to a  user-defined  index.  Value functions  are represented  as poten(cid:173) tial  distributions  over  the  problem  domain,  being  control  policies  represented  as  gradient  fields  over  the  same domain.  All  interme(cid:173) diate  policies  are intrinsically  safe,  i.e.  collisions  are not  promoted  during  the adaptation process.  The algorithm  has  efficient  imple(cid:173) mentation in parallel  SIMD  architectures.  One  potential  applica(cid:173) tion - travel distance  minimization - illustrates  its usefulness.
Model Matching and SFMD Computation
Authors: Steven Rehfuss, Dan W. Hammerstrom
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  systems  that  process sensory  data there is  frequently  a  model  matching stage where class hypotheses are combined to recognize a  complex entity.  We introduce a new model of parallelism, the Single  Function  Multiple  Data  (SFMD) model, appropriate to this stage.  SFMD functionality can be added with small hardware expense to  certain existing SIMD architectures, and as an incremental addition  to the programming model.  Adding  SFMD  to an SIMD  machine  will  not  only  allow  faster  model  matching,  but  also  increase  its  flexibility as a  general purpose machine and its scope in performing  the initial stages of sensory processing.
Learning with ensembles: How overfitting can be useful
Authors: Peter Sollich, Anders Krogh
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  study  the  characteristics  of learning  with ensembles.  Solving  exactly  the  simple  model  of an  ensemble  of linear  students,  we  find  surprisingly  rich  behaviour.  For  learning  in  large  ensembles,  it  is  advantageous  to  use  under-regularized  students,  which  actu(cid:173) ally  over-fit  the  training data.  Globally optimal performance  can  be  obtained by  choosing  the training set  sizes  of the students ap(cid:173) propriately.  For  smaller ensembles,  optimization of the  ensemble  weights  can yield significant improvements in ensemble generaliza(cid:173) tion  performance,  in particular if the  individual students  are  sub(cid:173) ject to noise in the training process.  Choosing students with a wide  range of regularization parameters makes this improvement robust  against changes in the unknown level of noise  in the training data.
Analog VLSI Processor Implementing the Continuous Wavelet Transform
Authors: R. Timothy Edwards, Gert Cauwenberghs
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We present an integrated analog processor for real-time wavelet decom(cid:173) position and reconstruction of continuous temporal signals covering the  audio frequency range. The processor performs complex harmonic modu(cid:173) lation and Gaussian lowpass filtering in 16 parallel channels, each clocked  at a different rate, producing a multiresolution mapping on a logarithmic  frequency scale. Our implementation uses mixed-mode analog and dig(cid:173) ital circuits, oversampling techniques, and switched-capacitor filters to  achieve a wide linear dynamic range while maintaining compact circuit  size and low power consumption. We include experimental results on the  processor and characterize its components separately from measurements  on a single-channel test chip.
A Novel Channel Selection System in Cochlear Implants Using Artificial Neural Network
Authors: Marwan A. Jabri, Raymond J. Wang
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: State-of-the-art  speech  processors  in  cochlear  implants  perform  channel selection  using  a spectral maxima strategy.  This strategy  can  lead  to  confusions  when  high  frequency  features  are  needed  to discriminate between  sounds.  We  present  in this paper  a  novel  channel selection strategy based upon pattern recognition which al(cid:173) lows  "smart" channel selections to be made.  The proposed strategy  is  implemented using  multi-layer  perceptrons  trained  on  a  multi(cid:173) speaker labelled speech database.  The input to the network are the  energy coefficients of N  energy channels.  The output of the system  are  the indices  of the M  selected  channels.  We  compare  the  performance  of our  proposed  system  to  that  of  spectral maxima strategy, and show that our strategy can produce  significantly better results.
Rapid Quality Estimation of Neural Network Input Representations
Authors: Kevin J. Cherkauer, Jude W. Shavlik
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The choice of an input representation for a neural network can have  a  profound  impact  on  its  accuracy  in  classifying  novel  instances.  However,  neural networks are typically  computationally expensive  to  train,  making  it  difficult  to  test  large  numbers  of alternative  representations.  This  paper  introduces  fast  quality  measures  for  neural  network  representations,  allowing  one  to  quickly  and  ac(cid:173) curately estimate which  of a  collection  of possible  representations  for  a  problem is  the best.  We show  that our measures for  ranking  representations are more accurate than a previously published mea(cid:173) sure,  based on experiments with  three difficult,  real-world pattern  recognition  problems.
Generating Accurate and Diverse Members of a Neural-Network Ensemble
Authors: David W. Opitz, Jude W. Shavlik
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Neural-network  ensembles  have  been  shown  to  be  very  accurate  classification  techniques.  Previous  work  has  shown  that  an effec(cid:173) tive ensemble  should  consist  of networks  that are not  only  highly  correct,  but  ones  that  make  their  errors on  different  parts  of the  input  space  as  well.  Most  existing  techniques,  however,  only  in(cid:173) directly  address  the  problem  of  creating  such  a  set  of  networks.  In  this  paper  we  present  a  technique  called  ADDEMUP  that  uses  genetic  algorithms  to  directly  search  for  an  accurate  and  diverse  set  of trained  networks.  ADDEMUP  works  by  first  creating an ini(cid:173) tial  population,  then  uses  genetic  operators  to  continually  create  new  networks,  keeping the set of networks that are as  accurate as  possible while disagreeing with each other as much as possible.  Ex(cid:173) periments on three DNA problems show that  ADDEMUP  is able to  generate a  set of trained  networks that is  more accurate than sev(cid:173) eral existing  approaches.  Experiments  also  show  that  ADDEMUP  is  able  to  effectively  incorporate  prior  knowledge,  if  available,  to  improve the quality of its ensemble.
A Model of Spatial Representations in Parietal Cortex Explains Hemineglect
Authors: Alexandre Pouget, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We have recently developed a theory of spatial representations in  which the position of an object is not encoded in a particular frame  of reference but, instead, involves neurons computing basis func(cid:173) tions of their sensory inputs. This type of representation is able  to perform nonlinear sensorimotor transformations and is consis(cid:173) tent with the response properties of parietal neurons. We now ask  whether the same theory could account for the behavior of human  patients with parietal lesions. These lesions induce a deficit known  as hemineglect that is characterized by a lack of reaction to stimuli  located in the hemispace contralateral to the lesion. A simulated  lesion in a basis function representation was found to replicate three  of the most important aspects of hemineglect: i) The models failed  to cross the leftmost lines in line cancellation experiments, ii) the  deficit affected multiple frames of reference and, iii) it could be  object centered. These results strongly support the basis function  hypothesis for spatial representations and provide a computational  theory of hemineglect at the single cell level.
Tempering Backpropagation Networks: Not All Weights are Created Equal
Authors: Nicol N. Schraudolph, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Terrence J. Sejnowski 
Computational Neurobiology Lab  The Salk Institute for BioI. Studies  San Diego, CA 92186-5800, USA