bluuebunny/crossref_metadata_2025 huggingface dataset:
Reduction of Conductance-Based Models with Slow Synapses to Neural Nets
Authors: Ermentrout Bard
Published: April 2008

The method of averaging and a detailed bifurcation calculation are used to reduce a system of synaptically coupled neurons to a Hopfield type continuous time neural network. Due to some special properties of the bifurcation, explicit averaging is not required and the reduction becomes a simple algebraic problem. The resultant calculations show one how to derive a new type of “squashing function” whose properties are directly related to the detailed ionic mechanisms of the membrane. Frequency encoding as opposed to amplitude encoding emerges in a natural fashion from the theory. The full system and the reduced system are numerically compared.
Stability of Oja's PCA Subspace Rule
Authors: Karhunen Juha
Published: April 2008

This paper deals with stability of Oja's symmetric algorithm for estimating the principal component subspace of the input data. Exact conditions are derived for the gain parameter on which the discrete algorithm remains bounded. The result is extended for a nonlinear version of Oja's algorithm.
A Bayesian Analysis of Self-Organizing Maps
Authors: Luttrell Stephen P.
Published: April 2008

In this paper Bayesian methods are used to analyze some of the properties of a special type of Markov chain. The forward transitions through the chain are followed by inverse transitions (using Bayes' theorem) backward through a copy of the same chain; this will be called a folded Markov chain. If an appropriately defined Euclidean error (between the original input and its “reconstruction” via Bayes' theorem) is minimized with respect to the choice of Markov chain transition probabilities, then the familiar theories of both vector quantizers and self-organizing maps emerge. This approach is also used to derive the theory of self-supervision, in which the higher layers of a multilayer network supervise the lower layers, even though overall there is no external teacher.
Measuring the VC-Dimension of a Learning Machine
Authors: Vapnik Vladimir , Levin Esther , Cun Yann Le
Published: April 2008

A method for measuring the capacity of learning machines is described. The method is based on fitting a theoretically derived function to empirical measurements of the maximal difference between the error rates on two separate data sets of varying sizes. Experimental measurements of the capacity of various types of linear classifiers are presented.
On Langevin Updating in Multilayer Perceptrons
Authors: Rögnvaldsson Thorsteinn
Published: April 2008

The Langevin updating rule, in which noise is added to the weights during learning, is presented and shown to improve learning on problems with initially ill-conditioned Hessians. This is particularly important for multilayer perceptrons with many hidden layers, that often have ill-conditioned Hessians. In addition, Manhattan updating is shown to have a similar effect.
Learning in Neural Networks with Material Synapses
Authors: Amit Daniel J. , Fusi Stefano
Published: April 2008

We discuss the long term maintenance of acquired memory in synaptic connections of a perpetually learning electronic device. This is affected by ascribing each synapse a finite number of stable states in which it can maintain for indefinitely long periods. Learning uncorrelated stimuli is expressed as a stochastic process produced by the neural activities on the synapses. In several interesting cases the stochastic process can be analyzed in detail, leading to a clarification of the performance of the network, as an associative memory, during the process of uninterrupted learning. The stochastic nature of the process and the existence of an asymptotic distribution for the synaptic values in the network imply generically that the memory is a palimpsest but capacity is as low as log N for a network of N neurons. The only way we find for avoiding this tight constraint is to allow the parameters governing the learning process (the coding level of the stimuli; the transition probabilities for potentiation and depression and the number of stable synaptic levels) to depend on the number of neurons. It is shown that a network with synapses that have two stable states can dynamically learn with optimal storage efficiency, be a palimpsest, and maintain its (associative) memory for an indefinitely long time provided the coding level is low and depression is equilibrated against potentiation. We suggest that an option so easily implementable in material devices would not have been overlooked by biology. Finally we discuss the stochastic learning on synapses with variable number of stable synaptic states.
Information Processing in Dendritic Trees
Authors: Mel Bartlett W.
Published: April 2008

This review considers the input-output behavior of neurons with dendritic trees, with an emphasis on questions of information processing. The parts of this review are (1) a brief history of ideas about dendritic trees, (2) a review of the complex electrophysiology of dendritic neurons, (3) an overview of conceptual tools used in dendritic modeling studies, including the cable equation and compartmental modeling techniques, and (4) a review of modeling studies that have addressed various issues relevant to dendritic information processing.
Simulations of Intrinsically Bursting Neocortical Pyramidal Neurons
Authors: Rhodes Paul A. , Gray Charles M.
Published: April 2008

Neocortical layer 5 intrinsically bursting (IB) pyramidal neurons were simulated using compartment model methods. Morphological data as well as target neurophysiological responses were taken from a series of published studies on the same set of rat visual cortex pyramidal neurons (Mason, A. and Larkman, A. J., 1990. J. Neurosci. 9,1440-1447; Larkman, A. J. 1991. J. Comp. Neurol. 306, 307-319). A dendritic distribution of ion channels was found that reproduced the range of in vitro responses of layer 5 IB pyramidal neurons, including the transition from repetitive bursting to the burst/tonic spiking mode seen in these neurons as input magnitude increases. In light of available data, the simulation results suggest that in these neurons bursts are driven by an inward flow of current during a high threshold Ca2+ spike extending throughout both the basal and apical dendritic branches.
Control of Chaos in Networks with Delay: A Model for Synchronization of Cortical Tissue
Authors: Lourenço C. , Babloyantz A.
Published: April 2008

The unstable periodic orbits of chaotic dynamics in systems described by delay differential equations are considered. An orbit is stabilized successfully, using a method proposed by Pyragas. The system under investigation is a network of excitatory and inhibitory neurons of moderate size, describing cortical activity. The relevance of the results for synchronized cortical activity is discussed.
First-Order Recurrent Neural Networks and Deterministic Finite State Automata
Authors: Manolios Peter , Fanelli Robert
Published: April 2008

We examine the correspondence between first-order recurrent neural networks and deterministic finite state automata. We begin with the problem of inducing deterministic finite state automata from finite training sets, that include both positive and negative examples, an NP-hard problem (Angluin and Smith 1983). We use a neural network architecture with two recurrent layers, which we argue can approximate any discrete-time, time-invariant dynamic system, with computation of the full gradient during learning. The networks are trained to classify strings as belonging or not belonging to the grammar. The training sets used contain only short strings, and the sets are constructed in a way that does not require a priori knowledge of the grammar. After training, the networks are tested using various test sets with strings of length up to 1000, and are often able to correctly classify all the test strings. These results are comparable to those obtained with second-order networks (Giles et al. 1992; Watrous and Kuhn 1992a; Zeng et al. 1993). We observe that the networks emulate finite state automata, confirming the results of other authors, and we use a vector quantization algorithm to extract deterministic finite state automata after training and during testing of the networks, obtaining a table listing the start state, accept states, reject states, all transitions from the states, as well as some useful statistics. We examine the correspondence between finite state automata and neural networks in detail, showing two major stages in the learning process. To this end, we use a graphics module, which graphically depicts the states of the network during the learning and testing phases. We examine the networks' performance when tested on strings much longer than those in the training set, noting a measure based on clustering that is correlated to the stability of the networks. Finally, we observe that with sufficiently long training times, neural networks can become true finite state automata, due to the attractor structure of their dynamics.
Estimation of Network Parameters in Semiparametric Stochastic Perceptron
Authors: Kawanabe Motoaki , Amari Shun-ichi
Published: April 2008

It was reported (Kabashima and Shinomoto 1992) that estimators of a binary decision boundary show asymptotically strange behaviors when the probability model is ill-posed or semiparametric. We give a rigorous analysis of this phenomenon in a stochastic perceptron by using the estimating function method. A stochastic perceptron consists of a neuron that is excited depending on the weighted sum of inputs but its probability distribution form is unknown here. It is shown that there exists no √n-consistent estimator of the threshold value h, that is, no estimator h that converges to h in the order of 1/ √n as the number n of observations increases. Therefore, the accuracy of estimation is much worse in this semiparametric case with an unspecified probability function than in the ordinary case. On the other hand, it is shown that there is a √n-consistent estimator ŵ of the synaptic weight vector. These results elucidate strange behaviors of learning curves in a semiparametric statistical model.
Degree of Approximation Results for Feedforward Networks Approximating Unknown Mappings and Their Derivatives
Authors: Hornik Kurt , Stinchcombe Maxwell , White Halbert , Auer Peter
Published: April 2008

Recently Barron (1993) has given rates for hidden layer feedforward networks with sigmoid activation functions approximating a class of functions satisfying a certain smoothness condition. These rates do not depend on the dimension of the input space. We extend Barron's results to feedforward networks with possibly nonsigmoid activation functions approximating mappings and their derivatives simultaneously. Our conditions are similar but not identical to Barron's, but we obtain the same rates of approximation, showing that the approximation error decreases at rates as fast as n−1/2, where n is the number of hidden units. The dimension of the input space appears only in the constants of our bounds.
Boosting and Other Ensemble Methods
Authors: Drucker Harris , Cortes Corinna , Jackel L. D. , LeCun Yann , Vapnik Vladimir
Published: April 2008

We compare the performance of three types of neural network-based ensemble techniques to that of a single neural network. The ensemble algorithms are two versions of boosting and committees of neural networks trained independently. For each of the four algorithms, we experimentally determine the test and training error curves in an optical character recognition (OCR) problem as both a function of training set size and computational cost using three architectures. We show that a single machine is best for small training set size while for large training set size some version of boosting is best. However, for a given computational cost, boosting is always best. Furthermore, we show a surprising result for the original boosting algorithm: namely, that as the training set size increases, the training error decreases until it asymptotes to the test error rate. This has potential implications in the search for better training algorithms.
Training with Noise is Equivalent to Tikhonov Regularization
Authors: Bishop Chris M.
Published: April 2008

It is well known that the addition of noise to the input data of a neural network during training can, in some circumstances, lead to significant improvements in generalization performance. Previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function. However, the regularization term, which involves second derivatives of the error function, is not bounded below, and so can lead to difficulties if used directly in a learning algorithm based on error minimization. In this paper we show that for the purposes of network training, the regularization term can be reduced to a positive semi-definite form that involves only first derivatives of the network mapping. For a sum-of-squares error function, the regularization term belongs to the class of generalized Tikhonov regularizers. Direct minimization of the regularized error function provides a practical alternative to training with noise.
Empirical Risk Minimization versus Maximum-Likelihood Estimation: A Case Study
Authors: Meir Ronny
Published: April 2008

We study the interaction between input distributions, learning algorithms, and finite sample sizes in the case of learning classification tasks. Focusing on the case of normal input distributions, we use statistical mechanics techniques to calculate the empirical and expected (or generalization) errors for several well-known algorithms learning the weights of a single-layer perceptron. In the case of spherically symmetric distributions within each class we find that the simple Hebb rule, corresponding to maximum-likelihood parameter estimation, outperforms the other more complex algorithms, based on error minimization. Moreover, we show that in the regime where the overlap between the classes is large, algorithms with low empirical error do worse in terms of generalization, a phenomenon known as overtraining.
Learning a Decision Boundary from Stochastic Examples: Incremental Algorithms with and without Queries
Authors: Kabashima Yoshiyuki , Shinomoto Shigeru
Published: April 2008

Even if it is not possible to reproduce a target input-output relation, a learning machine should be able to minimize the probability of making errors. A practical learning algorithm should also be simple enough to go without memorizing example data, if possible. Incremental algorithms such as error backpropagation satisfy this requirement. We propose incremental algorithms that provide fast convergence of the machine parameter θ to its optimal choice θo with respect to the number of examples t. We will consider the binary choice model whose target relation has a blurred boundary and the machine whose parameter θ specifies a decision boundary to make the output prediction. The question we wish to address here is how fast θ can approach θo, depending upon whether in the learning stage the machine can specify inputs as queries to the target relation, or the inputs are drawn from a certain distribution. If queries are permitted, the machine can achieve the fastest convergence, (θ - θo)2 ∼ O(t−1). If not, O(t−1) convergence is generally not attainable. For learning without queries, we showed in a previous paper that the error minimum algorithm exhibits a slow convergence (θ - θo)2 ∼ O(t−2/3). We propose here a practical algorithm that provides a rather fast convergence, O(t−4/5). It is possible to further accelerate the convergence by using more elaborate algorithms. The fastest convergence turned out to be O[(lnt)2 t−1]. This scaling is considered optimal among possible algorithms, and is not due to the incremental nature of our algorithm.
Arithmetic Perceptrons
Authors: Cannas Sergio A.
Published: April 2008

A feedforward layered neural network (perceptron) with one hidden layer, which adds two N-bit binary numbers is constructed. The set of synaptic strengths and thresholds is obtained exactly for different architectures of the network and for arbitrary N. These structures can be easily generalized to perform more complicated arithmetic operations (like subtraction).
Compensatory Mechanisms in an Attractor Neural Network Model of Schizophrenia
Authors: Horn D. , Ruppin E.
Published: April 2008

We investigate the effect of synaptic compensation on the dynamic behavior of an attractor neural network receiving its input stimuli as external fields projecting on the network. It is shown how, in the face of weakened inputs, memory performance may be preserved by strengthening internal synaptic connections and increasing the noise level. Yet, these compensatory changes necessarily have adverse side effects, leading to spontaneous, stimulus-independent retrieval of stored patterns. These results can support Stevens' recent hypothesis that the onset of schizophrenia is associated with frontal synaptic regeneration, occurring subsequent to the degeneration of temporal neurons projecting on these areas.
A Type of Duality between Self-Organizing Maps and Minimal Wiring
Authors: Mitchison Graeme
Published: April 2008

I show here that two interpretations of neural maps are closely related. The first, due to Kohonen, sees these maps as forming by an adaptive process in response to stimuli. The second—the minimal wiring or dimension-reduction perspective—interprets the maps as the solution of a minimization problem, where the goal is to keep the “wiring” between neurons with similar receptive fields as short as possible. Recent work by Luttrell provides a bridging concept, by showing that Kohonen's algorithm can be regarded as an approximation to gradient descent on a certain functional. I show how this functional can be generalized in a way that allows it to be interpreted as a measure of wirelength.
Development of Oriented Ocular Dominance Bands as a Consequence of Areal Geometry
Authors: Bauer H.-U.
Published: April 2008

It has been hypothesized that the different appearance of ocular dominance bands in the cat and the monkey is a consequence of the different mapping geometries in these species (LeVay et al. 1985; Anderson et al. 1988). Here I investigate the impact of areal geometries on the preferred direction of ocular dominance bands in two adaptive map formation models, the self-organizing feature map and the elastic net algorithm. In the case of the self-organizing feature map, the occurrence of instabilities that correspond to ocular dominance bands can be analytically investigated. The instabilities automatically yield stripes of correct orientation. These analytic results are complemented by simulations. In the case of the elastic net algorithm, simulations reveal two different parameter regimes of the algorithm, only one of which leads to stripes of correct orientation. The results suggest that neighborhood preservation in visual maps is enforced in the backward direction, such that neighboring cells in the cortex have neighboring receptive fields, and not vice versa.
A Multiple Cause Mixture Model for Unsupervised Learning
Authors: Saund Eric
Published: April 2008

This paper presents a formulation for unsupervised learning of clusters reflecting multiple causal structure in binary data. Unlike the “hard” k-means clustering algorithm and the “soft” mixture model, each of which assumes that a single hidden event generates each data point, a multiple cause model accounts for observed data by combining assertions from many hidden causes, each of which can pertain to varying degree to any subset of the observable dimensions. We employ an objective function and iterative gradient descent learning algorithm resembling the conventional mixture model. A crucial issue is the mixing function for combining beliefs from different cluster centers in order to generate data predictions whose errors are minimized both during recognition and learning. The mixing function constitutes a prior assumption about underlying structural regularities of the data domain; we demonstrate a weakness inherent to the popular weighted sum followed by sigmoid squashing, and offer alternative forms of the nonlinearity for two types of data domain. Results are presented demonstrating the algorithm's ability successfully to discover coherent multiple causal representations in several experimental data sets.
Unsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer Networks
Authors: Deco G. , Finnoff W. , Zimmermann H. G.
Published: April 2008

Controlling the network complexity in order to prevent overfitting is one of the major problems encountered when using neural network models to extract the structure from small data sets. In this paper we present a network architecture designed for use with a cost function that includes a novel complexity penalty term. In this architecture the outputs of the hidden units are strictly positive and sum to one, and their outputs are defined as the probability that the actual input belongs to a certain class formed during learning. The penalty term expresses the mutual information between the inputs and the extracted classes. This measure effectively describes the network complexity with respect to the given data in an unsupervised fashion. The efficiency of this architecture/penalty-term when combined with backpropagation training, is demonstrated on a real world economic time series forecasting problem. The model was also applied to the benchmark sunspot data and to a synthetic data set from the statistics community.
A Counterexample to Temporal Differences Learning
Authors: Bertsekas Dimitri P.
Published: April 2008

Sutton's TD(λ) method aims to provide a representation of the cost function in an absorbing Markov chain with transition costs. A simple example is given where the representation obtained depends on λ. For λ = 1 the representation is optimal with respect to a least-squares error criterion, but as λ decreases toward 0 the representation becomes progressively worse and, in some cases, very poor. The example suggests a need to understand better the circumstances under which TD(0) and Q-learning obtain satisfactory neural network-based compact representations of the cost function. A variation of TD(0) is also given, which performs better on the example.
On the Ordering Conditions for Self-Organizing Maps
Authors: Budinich Marco , Taylor John G.
Published: April 2008

We present a geometric interpretation of ordering in self-organizing feature maps. This view provides simpler proofs of Kohonen ordering theorem and of convergence to an ordered state in the one-dimensional case. At the same time it explains intuitively the origin of the problems in higher dimensional cases. Furthermore it provides a geometric view of the known characteristics of learning in self-organizing nets.
Learning Linear Threshold Approximations Using Perceptrons
Authors: Bylander Tom
Published: April 2008

We demonstrate sufficient conditions for polynomial learnability of suboptimal linear threshold functions using perceptrons. The central result is as follows. Suppose there exists a vector w*, of n weights (including the threshold) with “accuracy” 1 − α, “average error” η, and “balancing separation” σ, i.e., with probability 1 − α, w* correctly classifies an example x; over examples incorrectly classified by w*, the expected value of |w* · x| is η (source of inaccuracy does not matter); and over a certain portion of correctly classified examples, the expected value of |w* · x| is σ. Then, with probability 1 − δ, the perceptron achieves accuracy at least 1 − [∊ + α(1 + η/σ)] after O[n∊−2σ−2(ln 1/δ)] examples.
An Algorithm for Building Regularized Piecewise Linear Discrimination Surfaces: The Perceptron Membrane
Authors: Deffuant Guillaume
Published: April 2008

The perceptron membrane is a new connectionist model that aims at solving discrimination (classification) problems with piecewise linear surfaces. The discrimination surfaces of perceptron membranes are defined by the union of convex polyhedrons. Starting from only one convex polyhedron, new facets and new polyhedrons are added during learning. Moreover, the positions and orientations of the facets are continuously adapted according to the training examples. Considering each facet as a perceptron cell, a geometric credit assignment provides a local training domain to each perceptron of the network. This enables one to apply statistical theorems on the probability of good generalization for each unit on its learning domain, and gives a reliable criterion for perceptron elimination (using Vapnik-Chervonenkis dimension). Furthermore, a regularization procedure is implemented. The model efficiency is demonstrated on well-known problems such as the 2-spirals or waveforms.
Models of Orientation and Ocular Dominance Columns in the Visual Cortex: A Critical Comparison
Authors: Erwin E. , Obermayer K. , Schulten K.
Published: April 2008

Orientation and ocular dominance maps in the primary visual cortex of mammals are among the most thoroughly investigated of the patterns in the cerebral cortex. A considerable amount of work has been dedicated to unraveling both their detailed structure and the neural mechanisms that underlie their formation and development. Many schemes have been proposed, some of which are in competition. Some models focus on development of receptive fields while others focus on the structure of cortical maps, i.e., the arrangement of receptive field properties across the cortex. Each model used different means to determine its success at reproducing experimental map patterns, often relying principally on visual comparison. Experimental data are becoming available that allow a more careful evaluation of models. In this contribution more than 10 of the most prominent models of cortical map formation and structure are critically evaluated and compared with the most recent experimental findings from macaque striate cortex. Comparisons are based on properties of the predicted or measured cortical map patterns. We introduce several new measures for comparing experimental and model map data that reveal important differences between models. We expect that the use of these measures will improve current models by helping determine parameters to match model maps to experimental data now becoming available from a variety of species. Our study reveals that (1) despite apparent differences, many models are based on similar principles and consequently make similar predictions, (2) several models produce orientation map patterns that are not consistent with the experimental data from macaques, regardless of the plausibility of the models' suggested physiological implementations, and (3) no models have yet fully accounted for both the local and the global relationships between orientation and ocular dominance map patterns.
How Precise is Neuronal Synchronization?
Authors: König Peter , Engel Andreas K. , Roelfsema Pieter R. , Singer Wolf
Published: April 2008

Recent work suggests that synchronization of neuronal activity could serve to define functionally relevant relationships between spatially distributed cortical neurons. At present, it is not known to what extent this hypothesis is compatible with the widely supported notion of coarse coding, which assumes that features of a stimulus are represented by the graded responses of a population of optimally and suboptimally activated cells. To resolve this issue we investigated the temporal relationship between responses of optimally and suboptimally stimulated neurons in area 17 of cat visual cortex. We find that optimally and suboptimally activated cells can synchronize their responses with a precision of a few milliseconds. However, there are consistent and systematic deviations of the phase relations from zero phase lag. Systematic variation of the orientation of visual stimuli shows that optimally driven neurons tend to lead over suboptimally activated cells. The observed phase lag depends linearly on the stimulus orientation and is, in addition, proportional to the difference between the preferred orientations of the recorded cells. Similar effects occur when testing the influence of the movement direction and the spatial frequency of visual stimuli. These results suggest that binding by synchrony can be used to define assemblies of neurons representing a coarse-coded stimulus. Furthermore, they allow a quantitative test of neuronal network models designed to reproduce physiological results on stimulus-specific synchronization.
Memory Recall by Quasi-Fixed-Point Attractors in Oscillator Neural Networks
Authors: Fukai Tomoki , Shiino Masatoshi
Published: April 2008

It is shown that approximate fixed-point attractors rather than synchronized oscillations can be employed by a wide class of neural networks of oscillators to achieve an associative memory recall. This computational ability of oscillator neural networks is ensured by the fact that reduced dynamic equations for phase variables in general involve two terms that can be respectively responsible for the emergence of synchronization and cessation of oscillations. Thus the cessation occurs in memory retrieval if the corresponding term dominates in the dynamic equations. A bottomless feature of the energy function for such a system makes the retrieval states quasi-fixed points, which admit continual rotating motion to a small portion of oscillators, when an extensive number of memory patterns are embedded. An approximate theory based on the self-consistent signal-to-noise analysis enables one to study the equilibrium properties of the neural network of phase variables with the quasi-fixed-point attractors. As far as the memory retrieval by the quasi-fixed points is concerned, the equilibrium properties including the storage capacity of oscillator neural networks are proved to be similar to those of the Hopfield type neural networks.
Learning Population Codes by Minimizing Description Length
Authors: Zemel Richard S. , Hinton Geoffrey E.
Published: April 2008

The minimum description length (MDL) principle can be used to train the hidden units of a neural network to extract a representation that is cheap to describe but nonetheless allows the input to be reconstructed accurately. We show how MDL can be used to develop highly redundant population codes. Each hidden unit has a location in a low-dimensional implicit space. If the hidden unit activities form a bump of a standard shape in this space, they can be cheaply encoded by the center of this bump. So the weights from the input units to the hidden units in an autoencoder are trained to make the activities form a standard bump. The coordinates of the hidden units in the implicit space are also learned, thus allowing flexibility, as the network develops a discontinuous topography when presented with different input classes.
Bayesian Self-Organization Driven by Prior Probability Distributions
Authors: Yuille Alan L. , Smirnakis Stelios M. , Xu Lei
Published: April 2008

Recent work by Becker and Hinton (1992) shows a promising mechanism, based on maximizing mutual information assuming spatial coherence, by which a system can self-organize to learn visual abilities such as binocular stereo. We introduce a more general criterion, based on Bayesian probability theory, and thereby demonstrate a connection to Bayesian theories of visual perception and to other organization principles for early vision (Atick and Redlich 1990). Methods for implementation using variants of stochastic learning are described.
Adaptive Voting Rules for k-Nearest Neighbors Classifiers
Authors: Rovatti R. , Ragazzoni R. , Kovàcs Zs. M. , Guerrieri R.
Published: April 2008

A simple form of cooperation between the k-nearest neighbors (NN) approach to classification and the neural-like property of adaptation is explored. A tunable, high level k-nearest neighbors decision rule is defined that comprehends most previous generalizations of the common majority rule. A learning procedure is developed that applies to this rule and exploits those statistical features that can be induced from the training set. The overall approach is tested on a problem of handwritten character recognition. Experiments show that adaptivity in the decision rule may improve the recognition and rejection capability of standard k-NN classifiers.
Generalization and Analysis of the Lisberger-Sejnowski VOR Model
Authors: Qian Ning
Published: April 2008

Lisberger and Sejnowski (1992) recently proposed a computational model for motor learning in the vestibular-ocular reflex (VOR) system. They showed that the steady-state gain of the system can be modified by changing the ratio of the two time constants along the feedforward and the feedback projections to the Purkinje cell unit in their model VOR network. Here we generalize their model by including two additional time constant variables and two synaptic weight variables, which were set to fixed values in their original model. We derive the stability conditions of the generalized system and thoroughly analyze its steady-state and transient behavior. It is found that the generalized system can display a continuum of behavior with the Lisberger-Sejnowski model and a static model proposed by Miles et al. (1980b) as special cases. Moreover, although mathematically the Lisberger-Sejnowski model requires two precise relationships among its parameters, the model is robust against small perturbations from the physiological point of view. Additional considerations on the gain of smooth pursuit eye movement, which is believed to share the positive feedback loop with the VOR network, suggest that the VOR network should operate in the parameter range favoring the behavior studied by Lisberger and Sejnowski. Under this condition, the steady-state gain of the VOR is found to depend on all four time constants in the network. The time constant of the Purkinje cell unit should be relatively small in order to achieve effective VOR learning through the modifications of the other time constants. Our analysis provides a thorough characterization of the system and could thus be useful for guiding further physiological tests of the model.
A Modular and Hybrid Connectionist System for Speaker Identification
Authors: Bennani Younès
Published: April 2008

This paper presents and evaluates a modular/hybrid connectionist system for speaker identification. Modularity has emerged as a powerful technique for reducing the complexity of connectionist systems, and allowing a priori knowledge to be incorporated into their design. Text-independent speaker identification is an inherently complex task where the amount of training data is often limited. It thus provides an ideal domain to test the validity of the modular/hybrid connectionist approach. To achieve such identification, we develop, in this paper, an architecture based upon the cooperation of several connectionist modules, and a Hidden Markov Model module. When tested on a population of 102 speakers extracted from the DARPA-TIMIT database, perfect identification was obtained.
Error Estimation by Series Association for Neural Network Systems
Authors: Kim Keehoon , Bartlett Eric B.
Published: April 2008

Estimation of confidence intervals for neural network outputs is important when the uncertainty of a neural network system must be addressed for safety or reliability. This paper presents a new approach for estimating confidence intervals, which can help users validate neural network outputs. The estimation of confidence intervals, called error estimation by series association, is performed by a supplementary neural network trained to predict the error of the main neural network using input features and the output of the main network. The accuracy of this approach is shown using a simple nonlinear mapping and more complicated, realistic nuclear power plant fault diagnosis problems. The results demonstrate that the approach performs confidence estimation successfully.
Methods For Combining Experts' Probability Assessments
Authors: Jacobs Robert A.
Published: April 2008

This article reviews statistical techniques for combining multiple probability distributions. The framework is that of a decision maker who consults several experts regarding some events. The experts express their opinions in the form of probability distributions. The decision maker must aggregate the experts' distributions into a single distribution that can be used for decision making. Two classes of aggregation methods are reviewed. When using a supra Bayesian procedure, the decision maker treats the expert opinions as data that may be combined with its own prior distribution via Bayes' rule. When using a linear opinion pool, the decision maker forms a linear combination of the expert opinions. The major feature that makes the aggregation of expert opinions difficult is the high correlation or dependence that typically occurs among these opinions. A theme of this paper is the need for training procedures that result in experts with relatively independent opinions or for aggregation methods that implicitly or explicitly model the dependence among the experts. Analyses are presented that show that m dependent experts are worth the same as k independent experts where k ≤ m. In some cases, an exact value for k can be given; in other cases, lower and upper bounds can be placed on k.
The Helmholtz Machine
Authors: Dayan Peter , Hinton Geoffrey E. , Neal Radford M. , Zemel Richard S.
Published: April 2008

Discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning. One fruitful approach is to build a parameterized stochastic generative model, independent draws from which are likely to produce the patterns. For all but the simplest generative models, each pattern can be generated in exponentially many ways. It is thus intractable to adjust the parameters to maximize the probability of the observed patterns. We describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations. Our method can be viewed as a form of hierarchical self-supervised learning that may relate to the function of bottom-up and top-down cortical processing pathways.
Spontaneous Excitations in the Visual Cortex: Stripes, Spirals, Rings, and Collective Bursts
Authors: Fohlmeister Corinna , Gerstner Wulfram , Ritz Raphael , Hemmen J. Leo van
Published: April 2008

As a simple model of the cortical sheet, we study a locally connected net of spiking neurons, Refractoriness, noise, axonal delays, and the time course of excitatory and inhibitory postsynaptic potentials are taken into account explicitly. In addition to a low-activity state and depending on the synaptic efficacy, four different scenarios evolve spontaneously, viz., stripes, spirals, rings, and collective bursts. Our results can be related to experimental observations of drug-induced epilepsy and hallucinations.
From Data Distributions to Regularization in Invariant Learning
Authors: Leen Todd K.
Published: April 2008

Ideally pattern recognition machines provide constant output when the inputs are transformed under a group G of desired invariances. These invariances can be achieved by enhancing the training data to include examples of inputs transformed by elements of G, while leaving the corresponding targets unchanged. Alternatively the cost function for training can include a regularization term that penalizes changes in the output when the input is transformed under the group. This paper relates the two approaches, showing precisely the sense in which the regularized cost function approximates the result of adding transformed examples to the training data. We introduce the notion of a probability distribution over the group transformations, and use this to rewrite the cost function for the enhanced training data. Under certain conditions, the new cost function is equivalent to the sum of the original cost function plus a regularizer. For unbiased models, the regularizer reduces to the intuitively obvious choice—a term that penalizes changes in the output when the inputs are transformed under the group. For infinitesimal transformations, the coefficient of the regularization term reduces to the variance of the distortions introduced into the training data. This correspondence provides a simple bridge between the two approaches.
On the Distribution and Convergence of Feature Space in Self-Organizing Maps
Authors: Yin Hujun , Allinson Nigel M.
Published: April 2008

In this paper an analysis of the statistical and the convergence properties of Kohonen's self-organizing map of any dimension is presented. Every feature in the map is considered as a sum of a number of random variables. We extend the Central Limit Theorem to a particular case, which is then applied to prove that the feature space during learning tends to multiple gaussian distributed stochastic processes, which will eventually converge in the mean-square sense to the probabilistic centers of input subsets to form a quantization mapping with a minimum mean squared distortion either globally or locally. The diminishing effect, as training progresses, of the initial states on the value of the feature map is also shown.
Introducing Asymmetry into Interneuron Learning
Authors: Fyfe Colin
Published: April 2008

A review is given of a new artificial neural network architecture in which the weights converge to the principal component subspace. The weights learn by only simple Hebbian learning yet require no clipping, normalization or weight decay. The net self-organizes using negative feedback of activation from a set of "interneurons" to the input neurons. By allowing this negative feedback from the interneurons to act on other interneurons we can introduce the necessary asymmetry to cause convergence to the actual principal components. Simulations and analysis confirm such convergence.
Regularized Neural Networks: Some Convergence Rate Results
Authors: Corradi Valentina , White Halbert
Published: April 2008

In a recent paper, Poggio and Girosi (1990) proposed a class of neural networks obtained from the theory of regularization. Regularized networks are capable of approximating arbitrarily well any continuous function on a compactum. In this paper we consider in detail the learning problem for the one-dimensional case. We show that in the case of output data observed with noise, regularized networks are capable of learning and approximating (on compacta) elements of certain classes of Sobolev spaces, known as reproducing kernel Hilbert spaces (RKHS), at a nonparametric rate that optimally exploits the smoothness properties of the unknown mapping. In particular we show that the total squared error, given by the sum of the squared bias and the variance, will approach zero at a rate of n(-2m)/(2m+1), where m denotes the order of differentiability of the true unknown function. On the other hand, if the unknown mapping is a continuous function but does not belong to an RKHS, then there still exists a unique regularized solution, but this is no longer guaranteed to converge in mean square to a well-defined limit. Further, even if such a solution converges, the total squared error is bounded away from zero for all n sufficiently large.
A Comparison of Some Error Estimates for Neural Network Models
Authors: Tibshirani Robert
Published: April 2008

We discuss a number of methods for estimating the standard error of predicted values from a multilayer perceptron. These methods include the delta method based on the Hessian, bootstrap estimators, and the “sandwich” estimator. The methods are described and compared in a number of examples. We find that the bootstrap methods perform best, partly because they capture variability due to the choice of starting weights.
Neural Networks for Optimal Approximation of Smooth and Analytic Functions
Authors: Mhaskar H. N.
Published: April 2008

We prove that neural networks with a single hidden layer are capable of providing an optimal order of approximation for functions assumed to possess a given number of derivatives, if the activation function evaluated by each principal element satisfies certain technical conditions. Under these conditions, it is also possible to construct networks that provide a geometric order of approximation for analytic target functions. The permissible activation functions include the squashing function (1 − e−x)−1 as well as a variety of radial basis functions. Our proofs are constructive. The weights and thresholds of our networks are chosen independently of the target function; we give explicit formulas for the coefficients as simple, continuous, linear functionals of the target function.
A Short Proof of the Posterior Probability Property of Classifier Neural Networks
Authors: Rojas Raúl
Published: April 2008

It is now well known that neural classifiers can learn to compute a posteriori probabilities of classes in input space. This note offers a shorter proof than the traditional ones. Only one class has to be considered and straightforward minimization of the error function provides the main result. The method can be extended to any kind of differentiable error function. We also present a simple visual proof of the same theorem, which stresses the fact that the network must be perfectly trained and have enough plasticity.
Encoding with Bursting, Subthreshold Oscillations, and Noise in Mammalian Cold Receptors
Authors: Longtin André , Hinzer Karin
Published: April 2008

Mammalian cold thermoreceptors encode steady-state temperatures into characteristic temporal patterns of action potentials. We propose a mechanism for the encoding process. It is based on Plant's ionic model of slow wave bursting, to which stochastic forcing is added. The model reproduces firing patterns from cat lingual cold receptors as the parameters most likely to underlie the thermosensitivity of these receptors varied over a 25°C range. The sequence of firing patterns goes from regular bursting, to simple periodic, to stochastically phase-locked firing or “skipping.” The skipping at higher temperatures is shown to necessitate an interaction between noise and a subthreshold endogenous oscillation in the receptor. The basic period of all patterns is robust to noise. Further, noise extends the range of encodable stimuli. An increase in firing irregularity with temperature also results from the loss of stability accompanying the approach by the slow dynamics of a reverse Hopf bifurcation. The results are not dependent on the precise details of the Plant model, but are generic features of models where an autonomous slow wave arises through a Hopf bifurcation. The model also addresses the variability of the firing patterns across fibers. An alternate model of slow-wave bursting (Chay and Fan 1993) in which skipping can occur without noise is also analyzed here in the context of cold thermoreception. Our study quantifies the possible origins and relative contribution of deterministic and stochastic dynamics to the coding scheme. Implications of our findings for sensory coding are discussed.
Binary-Oscillator Networks: Bridging a Gap between Experimental and Abstract Modeling of Neural Networks
Authors: Wang Wei-Ping
Published: April 2008

This paper proposes a simplified oscillator model, called binary-oscillator, and develops a class of neural network models having binary-oscillators as basic units. The binary-oscillator has a binary dynamic variable v = ±1 modeling the “membrane potential” of a neuron, and due to the presence of a “slow current” (as in a classical relaxation-oscillator) it can oscillate between two states. The purpose of the simplification is to enable abstract algorithmic study on the dynamics of oscillator networks. A binary-oscillator network is formally analogous to a system of stochastic binary spins (atomic magnets) in statistical mechanics.