bluuebunny/crossref_metadata_2025 huggingface dataset:
Bayesian Modeling and Classification of Neural Signals
Authors: Lewicki Michael S.
Published: April 2008

Identifying and classifying action potential shapes in extracellular neural waveforms have long been the subject of research, and although several algorithms for this purpose have been successfully applied, their use has been limited by some outstanding problems. The first is how to determine shapes of the action potentials in the waveform and, second, how to decide how many shapes are distinct. A harder problem is that action potentials frequently overlap making difficult both the determination of the shapes and the classification of the spikes. In this report, a solution to each of these problems is obtained by applying Bayesian probability theory. By defining a probabilistic model of the waveform, the probability of both the form and number of spike shapes can be quantified. In addition, this framework is used to obtain an efficient algorithm for the decomposition of arbitrarily complex overlap sequences. This algorithm can extract many times more information than previous methods and facilitates the extracellular investigation of neuronal classes and of interactions within neuronal circuits.
Neural Nets with Superlinear VC-Dimension
Authors: Maass Wolfgang
Published: April 2008

It has been known for quite a while that the Vapnik-Chervonenkis dimension (VC-dimension) of a feedforward neural net with linear threshold gates is at most O(w · log w), where w is the total number of weights in the neural net. We show in this paper that this bound is in fact asymptotically optimal. More precisely, we exhibit for any depth d ≥ 3 a large class of feedforward neural nets of depth d with w weights that have VC-dimension Ω(w · log w). This lower bound holds even if the inputs are restricted to Boolean values. The proof of this result relies on a new method that allows us to encode more “program-bits” in the weights of a neural net than previously thought possible.
A Novel Design Method for Multilayer Feedforward Neural Networks
Authors: Lee Jihong
Published: April 2008

A multilayer feedforward neural network and a design method that takes the distribution of given training patterns into consideration are proposed. The size of the network, initial values of interconnection weights, and parameters defining the nonlinearities of processing elements are determined from a specially selected portion of the given training patterns, which is called a set of feature points. With these initial settings, the performance of the network is further improved by a modified error backpropagation learning process. It is shown in several examples that the proposed model and the design method are capable of rapidly learning the training patterns compared to conventional multilayer feedforward neural networks with random initialization techniques.
An Internal Mechanism for Detecting Parasite Attractors in a Hopfield Network
Authors: Gascuel Jean-Dominique , Moobed Bahram , Weinfeld Michel
Published: April 2008

This paper presents a built-in mechanism for automatic detection of prototypes (as opposed to parasite attractors) in a Hopfield network. It has a good statistical performance and avoids host computer overhead for this purpose. This mechanism is based on an internal coding of the prototypes during learning, using cyclic redundancy codes, and leads to an efficient implementation in VLSI. The immediate separation of prototypes from parasite attractors can be used to enhance the autonomy of the networks, aiming at hierarchical multinetwork structures. As an example, the use of such an architecture for the classification of handwritten digits is described.
Probabilistic Winner-Take-All Learning Algorithm for Radial-Basis-Function Neural Classifiers
Authors: Osman Hossam , Fahmy Moustafa M.
Published: April 2008

This paper proposes a new adaptive competitive learning algorithm called “the probabilistic winner-take-all.” The algorithm is based on a learning scheme developed by Agrawala within the statistical pattern recognition literature (Agrawala 1970). Its name stems from the fact that for a given input pattern once each competitor computes the probability of being the one that generated this pattern, the computed probabilities are utilized to probabilistically choose a winner. Then, only this winner is permitted to learn. The learning rule of the algorithm is derived for three different cases. Its properties are discussed and compared to those of two other competitive learning algorithms, namely the standard winner-take-all and the maximum-likelihood soft competition. Experimental comparison is also given. When all three algorithms are used to train the hidden layer of radial-basis-function classifiers, experiments indicate that classifiers trained with the probabilistic winner-take-all outperform those trained with the other two algorithms.
Realization of the “Weak Rod” by a Double Layer Parallel Network
Authors: Matsumoto T. , Kondo K.
Published: April 2008

The weak rod can be realized by a very natural double layer network with feedback. While the second layer is linear, the first layer is significantly nonlinear. Elements associated with the first layer are called “synaptic fuses.” An MRF argument is used to eliminate the line variables. The network makes what is cut transparent when a crease or a step edge is present.
Model Based on Extracellular Potassium for Spontaneous Synchronous Activity in Developing Retinas
Authors: Burgi Pierre-Yves , Grzywacz Norberto M.
Published: April 2008

Waves of action-potential bursts propagate across the ganglion-cell surface of isolated developing retinas. It has been suggested that the rise of extracellular potassium concentration following a burst of action potentials in a cell may underlie these waves by depolarizing neighbor cells. This suggestion is sensible for developing tissues, since their glial system is immature. We tested whether this extracellular-potassium suggestion is feasible. For this purpose, we built a realistic biophysical model of the ganglion-cell layer of the developing retina. Simulations with this model show that increases of extracellular potassium are sufficiently high (about fourfold) to mediate the waves consistently with experimental physiological and pharmacological data. Even if another mechanism mediates the waves, these simulations indicate that extracellular potassium should significantly modulate the waves' properties.
Effects of Input Synchrony on the Firing Rate of a Three-Conductance Cortical Neuron Model
Authors: Murthy Venkatesh N. , Fetz Eberhard E.
Published: April 2008

For a model cortical neuron with three active conductances, we studied the dependence of the firing rate on the degree of synchrony in its synaptic inputs. The effect of synchrony was determined as a function of three parameters: number of inputs, average input frequency, and the synaptic strength (maximal unitary conductance change). Synchrony alone could increase the cell's firing rate when the product of these three parameters was below a critical value. But for higher values of the three parameters, synchrony could reduce firing rate. Instantaneous responses to time-varying input firing rates were close to predictions from steady-state responses when input synchrony was high, but fell below steady-state responses when input synchrony was low. Effectiveness of synaptic transmission, measured by the peak area of cross-correlations between input and output spikes, increased with increasing synchrony.
The Functional Role of Excitatory and Inhibitory Interactions in Chopper Cells of the Anteroventral Cochlear Nucleus
Authors: Lai Ying-Cheng , Winslow Raimond L. , Sachs Murray B.
Published: April 2008

Chopper cells in the anteroventral cochlear nucleus of the cat maintain a robust rate-place representation of vowel spectra over a broad range of stimulus levels. This representation resembles that of low threshold, high spontaneous rate primary auditory nerve fibers at low stimulus levels, and that of high threshold, low spontaneous rate auditory-nerve fibers at high stimulus levels. This has led to the hypothesis that chopper cells in the anteroventral cochlear nucleus selectively process inputs from different spontaneous rate populations of primary auditory-nerve fibers at different stimulus levels. We present a computational model, making use of shunting inhibition, for how this level dependent processing may be performed within the chopper cell dendritic tree. We show that this model (1) implements level-dependent selective processing, (2) reproduces detailed features of real chopper cell post-stimulus-time histograms, and (3) reproduces nonmonotonic rate versus level functions in response to single tones measured.
Learning in Boltzmann Trees
Authors: Saul Lawrence , Jordan Michael I.
Published: April 2008

We introduce a large family of Boltzmann machines that can be trained by standard gradient descent. The networks can have one or more layers of hidden units, with tree-like connectivity. We show how to implement the supervised learning algorithm for these Boltzmann machines exactly, without resort to simulated or mean-field annealing. The stochastic averages that yield the gradients in weight space are computed by the technique of decimation. We present results on the problems of N-bit parity and the detection of hidden symmetries.
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Authors: Jaakkola Tommi , Jordan Michael I. , Singh Satinder P.
Published: April 2008

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD(λ) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(λ) and Q-learning belong.
A Dynamic Neural Network Architecture by Sequential Partitioning of the Input Space
Authors: Shadafan R. S. , Niranjan M.
Published: April 2008

We present a sequential approach to training multilayer perceptrons for pattern classification applications. The network is presented with each item of data only once and its architecture is dynamically adjusted during training. At the arrival of each example, a decision whether to increase the complexity of the network, or simply train the existing nodes is made based on three heuristic criteria. These criteria measure the position of the new item of data in the input space with respect to the information currently stored in the network. During the training process, each layer is assumed to be an independent entity with its particular input space. By adding nodes to each layer, the algorithm is effectively adding a hyperplane to the input space, hence adding a partition in the input space for that layer. When existing nodes are sufficient to accommodate the incoming input, the corresponding hidden nodes will be trained accordingly. Each hidden unit in the network is trained in closed form by means of a recursive least-squares (RLS) algorithm. A local covariance matrix of the data is maintained at each node and the closed form solution is recursively updated. The three criteria are computed from these covariance matrices to keep low computational cost. The performance of the algorithm is illustrated on two problems. The first problem is the two-dimensional Peterson and Barney vowel data. The second problem is a 33-dimensional data derived from a vision system for classifying wheat grains. The sequential nature of the algorithm has an efficient hardware implementation in the form of systolic arrays, and the incremental training idea has better biological plausibility compared with iterative methods.
Pruning from Adaptive Regularization
Authors: Hansen Lars Kai , Rasmussen Carl Edward
Published: April 2008

Inspired by the recent upsurge of interest in Bayesian methods we consider adaptive regularization. A generalization based scheme for adaptation of regularization parameters is introduced and compared to Bayesian regularization. We show that pruning arises naturally within both adaptive regularization schemes. As model example we have chosen the simplest possible: estimating the mean of a random variable with known variance. Marked similarities are found between the two methods in that they both involve a “noise limit,” below which they regularize with infinite weight decay, i.e., they prune. However, pruning is not always beneficial. We show explicitly that both methods in some cases may increase the generalization error. This corresponds to situations where the underlying assumptions of the regularizer are poorly matched to the environment.
Approximation Capability of Layered Neural Networks with Sigmoid Units on Two Layers
Authors: Ito Yoshifusa
Published: April 2008

Using only an elementary constructive method, we prove the universal approximation capability of three-layered feedforward neural networks that have sigmoid units on two layers. We regard the Heaviside function as a special case of sigmoid function and measure accuracy of approximation in either the supremum norm or in the Lp-norm. Given a continuous function defined on a unit hypercube and the required accuracy of approximation, we can estimate the numbers of necessary units on the respective sigmoid unit layers. In the case where the sigmoid function is the Heaviside function, our result improves the estimation of Kůrková (1992). If the accuracy of approximation is measured in the LP-norm, our estimation also improves that of Kůrková (1992), even when the sigmoid function is not the Heaviside function.
Influence Function Analysis of PCA and BCM Learning
Authors: Liu Yong
Published: April 2008

Based on the influence function (Hampel et al. 1986), we analyze several forms of learning rules under the influence of abnormal inputs (outliers). Principal component analysis (PCA) learning rules (Oja 1982, 1989, 1992) are shown to be sensitive to the influence of outliers. Biologically motivated robust PCA learning rules are proposed. The variants of BCM learning (Bienenstock et al. 1982; Intrator 1990b) with linear neurons are shown to be subject to the influence of outliers, in contrast they are shown to be robust to outliers with sigmoidal neurons.
Bayesian Regularization and Pruning Using a Laplace Prior
Authors: Williams Peter M.
Published: April 2008

Standard techniques for improved generalization from neural networks include weight decay and pruning. Weight decay has a Bayesian interpretation with the decay function corresponding to a prior over weights. The method of transformation groups and maximum entropy suggests a Laplace rather than a gaussian prior. After training, the weights then arrange themselves into two classes: (1) those with a common sensitivity to the data error and (2) those failing to achieve this sensitivity and that therefore vanish. Since the critical value is determined adaptively during training, pruning—in the sense of setting weights to exact zeros—becomes an automatic consequence of regularization alone. The count of free parameters is also reduced automatically as weights are pruned. A comparison is made with results of MacKay using the evidence framework and a gaussian regularizer.
The EM Algorithm and Information Geometry in Neural Network Learning
Authors: Amari Shun-ichi
Published: April 2008

Hidden units play an important role in neural networks, although their activation values are unknown in many learning situations. The EM algorithm (statistical algorithm) and the em algorithm (information-geometric one) have been proposed so far in this connection, and the effectiveness of such algorithms is recognized in many areas of research. The present note points out that these two algorithms are equivalent under a certain condition, although they are different in general.
Real-Time Control of a Tokamak Plasma Using Neural Networks
Authors: Bishop Chris M. , Haynes Paul S. , Smith Mike E. U. , Todd Tom N. , Trotman David L.
Published: April 2008

In this paper we present results from the first use of neural networks for real-time control of the high-temperature plasma in a tokamak fusion experiment. The tokamak is currently the principal experimental device for research into the magnetic confinement approach to controlled fusion. In an effort to improve the energy confinement properties of the high-temperature plasma inside tokamaks, recent experiments have focused on the use of noncircular cross-sectional plasma shapes. However, the accurate generation of such plasmas represents a demanding problem involving simultaneous control of several parameters on a time scale as short as a few tens of microseconds. Application of neural networks to this problem requires fast hardware, for which we have developed a fully parallel custom implementation of a multilayer perceptron, based on a hybrid of digital and analogue techniques.
Similarity Metric Learning for a Variable-Kernel Classifier
Authors: Lowe David G.
Published: April 2008

Nearest-neighbor interpolation algorithms have many useful properties for applications to learning, but they often exhibit poor generalization. In this paper, it is shown that much better generalization can be obtained by using a variable interpolation kernel in combination with conjugate gradient optimization of the similarity metric and kernel size. The resulting method is called variable-kernel similarity metric (VSM) learning. It has been tested on several standard classification data sets, and on these problems it shows better generalization than backpropagation and most other learning methods. The number of parameters that must be determined through optimization are orders of magnitude less than for backpropagation or radial basis function (RBF) networks, which may indicate that the method better captures the essential degrees of variation in learning. Other features of VSM learning are discussed that make it relevant to models for biological learning in the brain.
Regularization Theory and Neural Networks Architectures
Authors: Girosi Federico , Jones Michael , Poggio Tomaso
Published: April 2008

We had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units, called regularization networks. In particular, standard smoothness functionals lead to a subclass of regularization networks, the well known radial basis functions approximation schemes. This paper shows that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models and some of the neural networks. In particular, we introduce new classes of smoothness functionals that lead to different classes of basis functions. Additive splines as well as some tensor product splines can be obtained from appropriate classes of smoothness functionals. Furthermore, the same generalization that extends radial basis functions (RBF) to hyper basis functions (HBF) also leads from additive models to ridge approximation models, containing as special cases Breiman's hinge functions, some forms of projection pursuit regression, and several types of neural networks. We propose to use the term generalized regularization networks for this broad class of approximation schemes that follow from an extension of regularization. In the probabilistic interpretation of regularization, the different classes of basis functions correspond to different classes of prior probabilities on the approximating function spaces, and therefore to different types of smoothness assumptions. In summary, different multilayer networks with one hidden layer, which we collectively call generalized regularization networks, correspond to different classes of priors and associated smoothness functionals in a classical regularization principle. Three broad classes are (1) radial basis functions that can be generalized to hyper basis functions, (2) some tensor product splines, and (3) additive splines that can be generalized to schemes of the type of ridge approximation, hinge functions, and several perceptron-like neural networks with one hidden layer.
New Perceptron Model Using Random Bitstreams
Authors: Lee Eel-wan , Chae Soo-Ik
Published: April 2008

A very high precision is needed to implement the adder using stochastic computation (or pulse arithmetic) in modern VLSI technology. In this paper we propose a new model of perceptron using random bitstreams that alleviates this problem.
A Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTd
Authors: Wang Ruye
Published: April 2008

A simple and biologically plausible model is proposed to simulate the optic flow computation taking place in the dorsal part of medial superior temporal (MSTd) area of the visual cortex in the primates' brain. The model is a neural network composed of competitive learning layers. The input layer of the network simulates the neurons in the middle temporal (MT) area that selectively respond to the visual stimuli of the input motion patterns with different local velocities. The output layer of the network simulates the MSTd neurons that selectively respond to different types of optic flow motion patterns including planar, circular, radial, and spiral motions. Simulation results obtained from this model show that the behaviors of the output nodes of the network resemble very closely the known responsive properties of the MSTd neurons found neurophysiologically, such as the existence of three types of MSTd neurons that respond, respectively, to one, two, or three types of the input motion patterns with different position dependences, and the continuum of response selectivity formed by the three types of neurons.
Synchrony in Excitatory Neural Networks
Authors: Hansel D. , Mato G. , Meunier C.
Published: April 2008

Synchronization properties of fully connected networks of identical oscillatory neurons are studied, assuming purely excitatory interactions. We analyze their dependence on the time course of the synaptic interaction and on the response of the neurons to small depolarizations. Two types of responses are distinguished. In the first type, neurons always respond to small depolarization by advancing the next spike. In the second type, an excitatory postsynaptic potential (EPSP) received after the refractory period delays the firing of the next spike, while an EPSP received at a later time advances the firing. For these two types of responses we derive general conditions under which excitation destabilizes in-phase synchrony. We show that excitation is generally desynchronizing for neurons with a response of type I but can be synchronizing for responses of type II when the synaptic interactions are fast. These results are illustrated on three models of neurons: the Lapicque integrate-and-fire model, the model of Connor et al., and the Hodgkin-Huxley model. The latter exhibits a type II response, at variance with the first two models, that have type I responses. We then examine the consequences of these results for large networks, focusing on the states of partial coherence that emerge. Finally, we study the Lapicque model and the model of Connor et al. at large coupling and show that excitation can be desynchronizing even beyond the weak coupling regime.
Identification Using Feedforward Networks
Authors: Levin Asriel U. , Narendra Kumpati S.
Published: April 2008

This paper is concerned with the identification of an unknown nonlinear dynamic system when only the inputs and outputs are accessible for measurement. Specifically we investigate the use of feedforward neural networks as models for the input-output behavior of such systems. Relying on the approximation capabilities of feedforward neural networks and under mild assumptions regarding the properties of the underlying nonlinear system, it is shown that there exists a feedforward network that for almost all inputs (an open and dense set) will display the input-output behavior of the system.
The Upward Bias in Measures of Information Derived from Limited Data Samples
Authors: Treves Alessandro , Panzeri Stefano
Published: April 2008

Extracting information measures from limited experimental samples, such as those normally available when using data recorded in vivo from mammalian cortical neurons, is known to be plagued by a systematic error, which tends to bias the estimate upward. We calculate here the average of the bias, under certain conditions, as an asymptotic expansion in the inverse of the size of the data sample. The result agrees with numerical simulations, and is applicable, as an additive correction term, to measurements obtained under such conditions. Moreover, we discuss the implications for measurements obtained through other usual procedures.
Representation of Similarity in Three-Dimensional Object Discrimination
Authors: Edelman Shimon
Published: April 2008

How does the brain represent visual objects? In simple perceptual generalization tasks, the human visual system performs as if it represents the stimuli in a low-dimensional metric psychological space (Shepard 1987). In theories of three-dimensional (3D) shape recognition, the role of feature-space representations [as opposed to structural (Biederman 1987) or pictorial (Ullman 1989) descriptions] has long been a major point of contention. If shapes are indeed represented as points in a feature space, patterns of perceived similarity among different objects must reflect the structure of this space. The feature space hypothesis can then be tested by presenting subjects with complex parameterized 3D shapes, and by relating the similarities among subjective representations, as revealed in the response data by multidimensional scaling (Shepard 1980), to the objective parameterization of the stimuli. The results of four such tests, accompanied by computational simulations, support the notion that discrimination among 3D objects may rely on a low-dimensional feature space representation, and suggest that this space may be spanned by explicitly encoded class prototypes.
Quantitative Analysis of Electrotonic Structure and Membrane Properties of NMDA-Activated Lamprey Spinal Neurons
Authors: Murphey C. R. , Moore L. E. , Buchanan J. T.
Published: April 2008

Parameter optimization methods were used to quantitatively analyze frequency-domain-voltage-clamp data of NMDA-activated lamprey spinal neurons simultaneously over a wide range of membrane potentials. A neuronal cable model was used to explicitly take into account receptors located on the dendritic trees. The driving point membrane admittance was measured from the cell soma in response to a Fourier synthesized point voltage clamp stimulus. The data were fitted to an equivalent cable model consisting of a single lumped soma compartment coupled resistively to a series of equal dendritic compartments. The model contains voltage-dependent NMDA sensitive (INMDA), slow potassium (IK), and leakage (IL) currents. Both the passive cable properties and the voltage dependence of ion channel kinetics were estimated, including the electrotonic structure of the cell, the steady-state gating characteristics, and the time constants for particular voltage- and time-dependent ionic conductances. An alternate kinetic formulation was developed that consisted of steady-state values for the gating parameters and their time constants at half-activation values as well as slopes of these parameters at half-activation. This procedure allowed independent restrictions on the magnitude and slope of both the steady-state gating variable and its associated time constant. Quantitative estimates of the voltage-dependent membrane ion conductances and their kinetic parameters were used to solve the nonlinear equations describing dynamic responses. The model accurately predicts current clamp responses and is consistent with experimentally measured TTX-resistant NMDA-induced patterned activity. In summary, an analysis method is developed that provides a pragmatic approach to quantitatively describe a nonlinear neuronal system.
Reduced Representation by Neural Networks with Restricted Receptive Fields
Authors: Idiart Marco , Berk Barry , Abbott L. F.
Published: April 2008

Model neural networks can perform dimensional reductions of input data sets using correlation-based learning rules to adjust their weights. Simple Hebbian learning rules lead to an optimal reduction at the single unit level but result in highly redundant network representations. More complex rules designed to reduce or remove this redundancy can develop optimal principal component representations, but they are not very compelling from a biological perspective. Neurons in biological networks have restricted receptive fields limiting their access to the input data space. We find that, within this restricted receptive field architecture, simple correlation-based learning rules can produce surprisingly efficient reduced representations. When noise is present, the size of the receptive fields can be optimally tuned to maximize the accuracy of reconstructions of input data from a reduced representation.
Stochastic Single Neurons
Authors: Ohira Toru , Cowan Jack D.
Published: April 2008

We study the stochastic behavior of a single self-exciting model neuron with additive noise, a system that has bistable stochastic dynamics. We use Langevin and Fokker-Planck equations to obtain analytical expressions for the stationary distribution of activities and for the crossing rate between two stable states. We adjust the parameters in these expressions to fit observed histograms of neural activity, thus obtaining what we call an “effective single neuron” for a given network. We construct an effective single neuron from an activity histogram of a representative hidden neuron in a recurrent learning network. We also compare our result with an effective single neuron previously obtained analytically through the adiabatic elimination approximation.
Regularization in the Selection of Radial Basis Function Centers
Authors: Orr Mark J. L.
Published: April 2008

Subset selection and regularization are two well-known techniques that can improve the generalization performance of nonparametric linear regression estimators, such as radial basis function networks. This paper examines regularized forward selection (RFS)—a combination of forward subset selection and zero-order regularization. An efficient implementation of RFS into which either delete-1 or generalized cross-validation can be incorporated and a reestimation formula for the regularization parameter are also discussed. Simulation studies are presented that demonstrate improved generalization performance due to regularization in the forward selection of radial basis function centers.
Hints
Authors: Abu-Mostafa Yaser S.
Published: April 2008

The systematic use of hints in the learning-from-examples paradigm is the subject of this review. Hints are the properties of the target function that are known to us independently of the training examples. The use of hints is tantamount to combining rules and data in learning, and is compatible with different learning models, optimization techniques, and regularization techniques. The hints are represented to the learning process by virtual examples, and the training examples of the target function are treated on equal footing with the rest of the hints. A balance is achieved between the information provided by the different hints through the choice of objective functions and learning schedules. The Adaptive Minimization algorithm achieves this balance by relating the performance on each hint to the overall performance. The application of hints in forecasting the very noisy foreign-exchange markets is illustrated. On the theoretical side, the information value of hints is contrasted to the complexity value and related to the VC dimension.
Topology and Geometry of Single Hidden Layer Network, Least Squares Weight Solutions
Authors: Coetzee Frans M. , Stonick Virginia L.
Published: April 2008

In this paper the topological and geometric properties of the weight solutions for multilayer perceptron (MLP) networks under the MSE error criterion are characterized. The characterization is obtained by analyzing a homotopy from linear to nonlinear networks in which the hidden node function is slowly transformed from a linear to the final sigmoidal nonlinearity. Two different geometric perspectives for this optimization process are developed. The generic topology of the nonlinear MLP weight solutions is described and related to the geometric interpretations, error surfaces, and homotopy paths, both analytically and using carefully constructed examples. These results illustrate that although the natural homotopy provides a practically valuable heuristic for training, it suffers from a number of theoretical and practical difficulties. The linear system is a bifurcation point of the homotopy equations, and solution paths are therefore generically discontinuous. Bifurcations and infinite solutions further occur for data sets that are not of measure zero. These results weaken the guarantees on global convergence and exhaustive behavior normally associated with homotopy methods. However, the analyses presented provide a clear understanding of the relationship between linear and nonlinear perceptron networks, and thus a firm foundation for development of more powerful training methods. The geometric perspectives and generic topological results describing the nature of the solutions are further generally applicable to network analysis and algorithm evaluation.
Time-Skew Hebb Rule in a Nonisopotential Neuron
Authors: Pearlmutter Barak A.
Published: April 2008

In an isopotential neuron with rapid response, it has been shown that the receptive fields formed by Hebbian synaptic modulation depend on the principal eigenspace of Q(0), the input autocorrelation matrix, where Qij(τ) = 〈ξi(τ) ξj(t − T)〉 and ξi(t) is the input to synapse i at time t (Oja 1982). We relax the assumption of isopotentiality, introduce a time-skewed Hebb rule, and find that the dynamics of synaptic evolution are determined by the principal eigenspace of [Formula: see text]. This matrix is defined by [Formula: see text], where Kij(τ) is the neuron's voltage response to a unit current injection at synapse j as measured τ seconds later at synapse i, and ψi(τ) is the time course of the opportunity for modulation of synapse i following the arrival of a presynaptic action potential.
Synapse Models for Neural Networks: From Ion Channel Kinetics to Multiplicative Coefficient wij
Authors: Chapeau-Blondeau François , Chambet Nicolas
Published: April 2008

This paper relates different levels at which the modeling of synaptic transmission can be grounded in neural networks: the level of ion channel kinetics, the level of synaptic conductance dynamics, and the level of a scalar synaptic coefficient. The important assumptions to reduce a synapse model from one level to the next are explicitly exhibited. This coherent progression provides control on what is discarded and what is retained in the modeling process, and is useful to appreciate the significance and limitations of the resulting neural networks. This methodic simplification terminates with a scalar synaptic efficacy as it is very often used in neural networks, but here its conditions of validity are explicitly displayed. This scalar synapse also comes with an expression that directly relates it to basic quantities of synaptic functioning, and it can be endowed with meaningful physical units and realistic numerical values. In addition, it is shown that the scalar synapse does not receive the same expression in neural networks operating with spikes or with firing rates. These coherent modeling elements can help to improve, adjust, and refine the investigation of neural systems and their remarkable collective properties for information processing.
Stable Adaptive Control of Robot Manipulators Using “Neural” Networks
Authors: Sanner Robert M. , Slotine Jean-Jacques E.
Published: April 2008

The rapid development and formalization of adaptive signal processing algorithms loosely inspired by biological models can be potentially harnessed for use in flexible new learning control algorithms for nonlinear dynamic systems. However, if such controller designs are to be viable in practice, their stability must be guaranteed and their performance quantified. In this paper, the stable adaptive tracking control designs employing “neural” networks, initially presented in Sanner and Slotine (1992), are extended to classes of multivariable mechanical systems, including robot manipulators, and bounds are developed for the magnitude of the asymptotic tracking errors and the rate of convergence to these bounds. This new algorithm permits simultaneous learning and control, without recourse to an initial identification stage, and is distinguished from previous stable adaptive robotic controllers, e.g. (Slotine and Li 1987), by the relative lack of structure assumed in the design of the control law. The required control is simply considered to contain unknown functions of the measured state variables, and adaptive “neural” networks are used to stably determine, in real time, the entire required functional dependence. While computationally more complex than explicitly model-based techniques, the methods developed in this paper may be effectively applied to the control of many physical systems for which the state dependence of the dynamics is reasonably well understood, but the exact functional form of this dependence, or part thereof, is not, such as underwater robotic vehicles and high performance aircraft.
Test Error Fluctuations in Finite Linear Perceptrons
Authors: Barber D. , Saad D. , Sollich P.
Published: April 2008

We examine the fluctuations in the test error induced by random, finite, training and test sets for the linear perceptron of input dimension n with a spherically constrained weight vector. This variance enables us to address such issues as the partitioning of a data set into a test and training set. We find that the optimal assignment of the test set size scales with n2/3.
Learning and Extracting Initial Mealy Automata with a Modular Neural Network Model
Authors: Tiňo Peter , Šajda Jozef
Published: April 2008

A hybrid recurrent neural network is shown to learn small initial mealy machines (that can be thought of as translation machines translating input strings to corresponding output strings, as opposed to recognition automata that classify strings as either grammatical or nongrammatical) from positive training samples. A well-trained neural net is then presented once again with the training set and a Kohonen self-organizing map with the “star” topology of neurons is used to quantize recurrent network state space into distinct regions representing corresponding states of a mealy machine being learned. This enables us to extract the learned mealy machine from the trained recurrent network. One neural network (Kohonen self-organizing map) is used to extract meaningful information from another network (recurrent neural network).
Dynamic Cell Structure Learns Perfectly Topology Preserving Map
Authors: Bruske Jörg , Sommer Gerald
Published: April 2008

Dynamic cell structures (DCS) represent a family of artificial neural architectures suited both for unsupervised and supervised learning. They belong to the recently (Martinetz 1994) introduced class of topology representing networks (TRN) that build perfectly topology preserving feature maps. DCS employ a modified Kohonen learning rule in conjunction with competitive Hebbian learning. The Kohonen type learning rule serves to adjust the synaptic weight vectors while Hebbian learning establishes a dynamic lateral connection structure between the units reflecting the topology of the feature manifold. In case of supervised learning, i.e., function approximation, each neural unit implements a radial basis function, and an additional layer of linear output units adjusts according to a delta-rule. DCS is the first RBF-based approximation scheme attempting to concurrently learn and utilize a perfectly topology preserving map for improved performance. Simulations on a selection of CMU-Benchmarks indicate that the DCS idea applied to the growing cell structure algorithm (Fritzke 1993c) leads to an efficient and elegant algorithm that can beat conventional models on similar tasks.
Learning and Generalization in Radial Basis Function Networks
Authors: Freeman J. A. S. , Saad D.
Published: April 2008

The two-layer radial basis function network, with fixed centers of the basis functions, is analyzed within a stochastic training paradigm. Various definitions of generalization error are considered, and two such definitions are employed in deriving generic learning curves and generalization properties, both with and without a weight decay term. The generalization error is shown analytically to be related to the evidence and, via the evidence, to the prediction error and free energy. The generalization behavior is explored; the generic learning curve is found to be inversely proportional to the number of training pairs presented. Optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms. Finally, the effect of the joint activations between hidden-layer units is examined and shown to speed training.
Precision and Approximate Flatness in Artificial Neural Networks
Authors: Stinchcombe Maxwell B.
Published: April 2008

Several of the major classes of artificial neural networks' output functions are linear combinations of elements of approximately flat sets. This gives a tool for understanding the precision problem as well as providing a rationale for mixing types of networks. Approximate flatness also helps explain the power of artificial neural network techniques relative to series regressions—series regressions take linear combinations of flat sets, while neural networks take linear combinations of the much larger class of approximately flat sets.
Lower Bounds on the VC Dimension of Smoothly Parameterized Function Classes
Authors: Lee Wee Sun , Bartlett Peter L. , Williamson Robert C.
Published: April 2008

We examine the relationship between the VC dimension and the number of parameters of a threshold smoothly parameterized function class. We show that the VC dimension of such a function class is at least k if there exists a k-dimensional differentiable manifold in the parameter space such that each member of the manifold corresponds to a different decision boundary. Using this result, we are able to obtain lower bounds on the VC dimension proportional to the number of parameters for several thresholded function classes including two-layer neural networks with certain smooth activation functions and radial basis functions with a gaussian basis. These lower bounds hold even if the magnitudes of the parameters are restricted to be arbitrarily small. In Valiant's probably approximately correct learning framework, this implies that the number of examples necessary for learning these function classes is at least linear in the number of parameters.
Agnostic PAC Learning of Functions on Analog Neural Nets
Authors: Maass Wolfgang
Published: April 2008

We consider learning on multilayer neural nets with piecewise polynomial activation functions and a fixed number k of numerical inputs. We exhibit arbitrarily large network architectures for which efficient and provably successful learning algorithms exist in the rather realistic refinement of Valiant's model for probably approximately correct learning ("PAC learning") where no a priori assumptions are required about the "target function" (agnostic learning), arbitrary noise is permitted in the training sample, and the target outputs as well as the network outputs may be arbitrary reals. The number of computation steps of the learning algorithm LEARN that we construct is bounded by a polynomial in the bit-length n of the fixed number of input variables, in the bound s for the allowed bit-length of weights, in 1/ε, where ε is some arbitrary given bound for the true error of the neural net after training, and in 1/δ where δ is some arbitrary given bound for the probability that the learning algorithm fails for a randomly drawn training sample. However, the computation time of LEARN is exponential in the number of weights of the considered network architecture, and therefore only of interest for neural nets of small size. This article provides details to the previously published extended abstract (Maass 1994).
Convex Potentials and their Conjugates in Analog Mean-Field Optimization
Authors: Elfadel I. M.
Published: April 2008

This paper deals with the problem of mapping hybrid (i.e., both discrete and continuous) constrained optimization problems onto analog networks. The saddle-point paradigm of mean-field methods in statistical physics provides a systematic procedure for finding such a mapping via the notion of effective energy. Specifically, it is shown that within this paradigm, to each closed bounded constraint set is associated a smooth convex potential function. Using the conjugate (or the Legendre-Fenchel transform) of the convex potential, the effective energy can be transformed to yield a cost function that is a natural generalization of the analog Hopfield energy. Descent dynamics and deterministic annealing can then be used to find the global minimum of the original minimization problem. When the conjugate is hard to compute explicitly, it is shown that a minimax dynamics, similar to that of Arrow and Hurwicz in Lagrangian optimization, can be used to find the saddle points of the effective energy. As an illustration of its wide applicability, the effective energy framework is used to derive Hopfield-like energy functions and descent dynamics for two classes of networks previously considered in the literature, winner-take-all networks and rotor networks, even when the cost function of the original optimization problem is not quadratic.
Patterns of Functional Damage in Neural Network Models of Associative Memory
Authors: Ruppin Eytan , Reggia James A.
Published: April 2008

Current understanding of the effects of damage on neural networks is rudimentary, even though such understanding could lead to important insights concerning neurological and psychiatric disorders. Motivated by this consideration, we present a simple analytical framework for estimating the functional damage resulting from focal structural lesions to a neural network model. The effects of focal lesions of varying area, shape, and number on the retrieval capacities of a spatially organized associative memory are quantified, leading to specific scaling laws that may be further examined experimentally. It is predicted that multiple focal lesions will impair performance more than a single lesion of the same size, that slit like lesions are more damaging than rounder lesions, and that the same fraction of damage (relative to the total network size) will result in significantly less performance decrease in larger networks. Our study is clinically motivated by the observation that in multi-infarct dementia, the size of metabolically impaired tissue correlates with the level of cognitive impairment more than the size of structural damage. Our results account for the detrimental effect of the number of infarcts rather than their overall size of structural damage, and for the "multiplicative" interaction between Alzheimer's disease and multi-infarct dementia.
Time-Domain Solutions of Oja's Equations
Authors: Jr. J. L. Wyatt, , Elfadel I. M.
Published: April 2008

Oja's equations describe a well-studied system for unsupervised Hebbian learning of principal components. This paper derives the explicit time-domain solution of Oja's equations for the single-neuron case. It also shows that, under a linear change of coordinates, these equations are a gradient system in the general multi-neuron case. This latter result leads to a new Lyapunov-like function for Oja's equations.