bluuebunny/crossref_metadata_2025 huggingface dataset:
An Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural Networks
Authors: Alquézar R. , Sanfeliu A.
Published: April 2008

In this paper we present an algebraic framework to represent finite state machines (FSMs) in single-layer recurrent neural networks (SLRNNs), which unifies and generalizes some of the previous proposals. This framework is based on the formulation of both the state transition function and the output function of an FSM as a linear system of equations, and it permits an analytical explanation of the representational capabilities of first-order and higher-order SLRNNs. The framework can be used to insert symbolic knowledge in RNNs prior to learning from examples and to keep this knowledge while training the network. This approach is valid for a wide range of activation functions, whenever some stability conditions are met. The framework has already been used in practice in a hybrid method for grammatical inference reported elsewhere (Sanfeliu and Alquézar 1994).
Local and Global Optimization Algorithms for Generalized Learning Automata
Authors: Phansalkar V. V. , Thathachar M. A. L.
Published: April 2008

This paper analyzes the long-term behavior of the REINFORCE and related algorithms (Williams 1986, 1988, 1992) for generalized learning automata (Narendra and Thathachar 1989) for the associative reinforcement learning problem (Barto and Anandan 1985). The learning system considered here is a feedforward connectionist network of generalized learning automata units. We show that REINFORCE is a gradient ascent algorithm but can exhibit unbounded behavior. A modified version of this algorithm, based on constrained optimization techniques, is suggested to overcome this disadvantage. The modified algorithm is shown to exhibit local optimization properties. A global version of the algorithm, based on constant temperature heat bath techniques, is also described and shown to converge to the global maximum. All algorithms are analyzed using weak convergence techniques.
Initializing Weights of a Multilayer Perceptron Network by Using the Orthogonal Least Squares Algorithm
Authors: Lehtokangas Mikko , Saarinen Jukka , Kaski Kimmo , Huuhtanen Pentti
Published: April 2008

Usually the training of a multilayer perceptron network starts by initializing the network weights with small random values, and then the weight adjustment is carried out by using an iterative gradient descent-based optimization routine called backpropagation training. If the random initial weights happen to be far from a good solution or they are near a poor local optimum, the training will take a lot of time since many iteration steps are required. Furthermore, it is very possible that the network will not converge to an adequate solution at all. On the other hand, if the initial weights are close to a good solution the training will be much faster and the possibility of obtaining adequate convergence increases. In this paper a new method for initializing the weights is presented. The method is based on the orthogonal least squares algorithm. The simulation results obtained with the proposed initialization method show a considerable improvement in training compared to the randomly initialized networks. In light of practical experiments, the proposed method has proven to be fast and useful for initializing the network weights.
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Authors: Bell Anthony J. , Sejnowski Terrence J.
Published: April 2008

We derive a new self-organizing learning algorithm that maximizes the information transferred in a network of nonlinear units. The algorithm does not assume any knowledge of the input distributions, and is defined here for the zero-noise limit. Under these conditions, information maximization has extra properties not found in the linear case (Linsker 1989). The nonlinearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalization of principal components analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to 10 speakers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information maximization provides a unifying framework for problems in "blind" signal processing.
Self-Organization as an Iterative Kernel Smoothing Process
Authors: Mulier Filip , Cherkassky Vladimir
Published: April 2008

Kohonen's self-organizing map, when described in a batch processing mode, can be interpreted as a statistical kernel smoothing problem. The batch SOM algorithm consists of two steps. First, the training data are partitioned according to the Voronoi regions of the map unit locations. Second, the units are updated by taking weighted centroids of the data falling into the Voronoi regions, with the weighing function given by the neighborhood. Then, the neighborhood width is decreased and steps 1, 2 are repeated. The second step can be interpreted as a statistical kernel smoothing problem where the neighborhood function corresponds to the kernel and neighborhood width corresponds to kernel span. To determine the new unit locations, kernel smoothing is applied to the centroids of the Voronoi regions in the topological space. This interpretation leads to some new insights concerning the role of the neighborhood and dimensionality reduction. It also strengthens the algorithm's connection with the Principal Curve algorithm. A generalized self-organizing algorithm is proposed, where the kernel smoothing step is replaced with an arbitrary nonparametric regression method.
Sorting with Self-Organizing Maps
Authors: Budinich Marco
Published: April 2008

A self-organizing feature map (Von der Malsburg 1973; Kohonen 1984) sorts n real numbers in O(n) time apparently violating the O(n log n) bound. Detailed analysis shows that the net takes advantage of the uniform distribution of the numbers and, in this case, sorting in O(n) is possible. There are, however, an exponentially small fraction of pathological distributions producing O(n2) sorting time. It is interesting to observe that standard learning produced a smart sorting algorithm.
The Target Switch Algorithm: A Constructive Learning Procedure for Feed-Forward Neural Networks
Authors: Campbell Colin , Vicente C. Perez
Published: April 2008

We propose an efficient procedure for constructing and training a feedforward neural network. The network can perform binary classification for binary or analogue input data. We show that the procedure can also be used to construct feedforward neural networks with binary-valued weights. Neural networks with binary-valued weights are potentially straightforward to implement using microelectronic or optical devices and they can also exhibit good generalization.
On the Practical Applicability of VC Dimension Bounds
Authors: Holden Sean B. , Niranjan Mahesan
Published: April 2008

This article addresses the question of whether some recent Vapnik-Chervonenkis (VC) dimension-based bounds on sample complexity can be regarded as a practical design tool. Specifically, we are interested in bounds on the sample complexity for the problem of training a pattern classifier such that we can expect it to perform valid generalization. Early results using the VC dimension, while being extremely powerful, suffered from the fact that their sample complexity predictions were rather impractical. More recent results have begun to improve the situation by attempting to take specific account of the precise algorithm used to train the classifier. We perform a series of experiments based on a task involving the classification of sets of vowel formant frequencies. The results of these experiments indicate that the more recent theories provide sample complexity predictions that are significantly more applicable in practice than those provided by earlier theories; however, we also find that the recent theories still have significant shortcomings.
LeRec: A NN/HMM Hybrid for On-Line Handwriting Recognition
Authors: Bengio Yoshua , LeCun Yann , Nohl Craig , Burges Chris
Published: April 2008

We introduce a new approach for on-line recognition of handwritten words written in unconstrained mixed style. The preprocessor performs a word-level normalization by fitting a model of the word structure using the EM algorithm. Words are then coded into low resolution "annotated images" where each pixel contains information about trajectory direction and curvature. The recognizer is a convolution network that can be spatially replicated. From the network output, a hidden Markov model produces word scores. The entire system is globally trained to minimize word-level errors.
Lower Bounds for the Computational Power of Networks of Spiking Neurons
Authors: Maass Wolfgang
Published: April 2008

We investigate the computational power of a formal model for networks of spiking neurons. It is shown that simple operations on phase differences between spike-trains provide a very powerful computational tool that can in principle be used to carry out highly complex computations on a small network of spiking neurons. We construct networks of spiking neurons that simulate arbitrary threshold circuits, Turing machines, and a certain type of random access machines with real valued inputs. We also show that relatively weak basic assumptions about the response and threshold functions of the spiking neurons are sufficient to employ them for such computations.
On Convergence Properties of the EM Algorithm for Gaussian Mixtures
Authors: Xu Lei , Jordan Michael I.
Published: April 2008

We build up the mathematical connection between the “Expectation-Maximization” (EM) algorithm and gradient-based approaches for maximum likelihood learning of finite gaussian mixtures. We show that the EM step in parameter space is obtained from the gradient via a projection matrix P, and we provide an explicit expression for the matrix. We then analyze the convergence of EM in terms of special properties of P and provide new results analyzing the effect that P has on the likelihood surface. Based on these mathematical results, we present a comparative discussion of the advantages and disadvantages of EM and other algorithms for the learning of gaussian mixture models.
Equivalence of Linear Boltzmann Chains and Hidden Markov Models
Authors: MacKay David J. C.
Published: April 2008

Several authors have studied the relationship between hidden Markov models and “Boltzmann chains” with a linear or “time-sliced” architecture. Boltzmann chains model sequences of states by defining state-state transition energies instead of probabilities. In this note I demonstrate that under the simple condition that the state sequence has a mandatory end state, the probability distribution assigned by a strictly linear Boltzmann chain is identical to that assigned by a hidden Markov model.
Diagrammatic Derivation of Gradient Algorithms for Neural Networks
Authors: Wan Eric A. , Beaufays Françoise
Published: April 2008

Deriving gradient algorithms for time-dependent neural network structures typically requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. In this paper, we show how to derive such algorithms via a set of simple block diagram manipulation rules. The approach provides a common framework to derive popular algorithms including backpropagation and backpropagation-through-time without a single chain rule expansion. Additional examples are provided for a variety of complicated architectures to illustrate both the generality and the simplicity of the approach.
Coding of Time-Varying Signals in Spike Trains of Integrate-and-Fire Neurons with Random Threshold
Authors: Gabbiani Fabrizio , Koch Christof
Published: April 2008

Recently, methods of statistical estimation theory have been applied by Bialek and collaborators (1991) to reconstruct time-varying velocity signals and to investigate the processing of visual information by a directionally selective motion detector in the fly's visual system, the H1 cell. We summarize here our theoretical results obtained by studying these reconstructions starting from a simple model of H1 based on experimental data. Under additional technical assumptions, we derive a closed expression for the Fourier transform of the optimal reconstruction filter in terms of the statistics of the stimulus and the characteristics of the model neuron, such as its firing rate. It is shown that linear reconstruction filters will change in a nontrivial way if the statistics of the signal or the mean firing rate of the cell changes. Analytical expressions are then derived for the mean square error in the reconstructions and the lower bound on the rate of information transmission that was estimated experimentally by Bialek et al. (1991). For plausible values of the parameters, the model is in qualitative agreement with experimental data. We show that the rate of information transmission and mean square error represent different measures of the reconstructions: in particular, satisfactory reconstructions in terms of the mean square error can be achieved only using stimuli that are matched to the properties of the recorded cell. Finally, it is shown that at least for the class of models presented here, reconstruction methods can be understood as a generalization of the more familiar reverse-correlation technique.
A Simple Spike Train Decoder Inspired by the Sampling Theorem
Authors: August David A. , Levy William B
Published: April 2008

Reconstructing a time-varying stimulus estimate from a spike train (Bialek's “decoding” of a spike train) has become an important way to study neural information processing. In this paper, we describe a simple method for reconstructing a time-varying current injection signal from the simulated spike train it produces. This technique extracts most of the information from the spike train, provided that the input signal is appropriately matched to the spike generator. To conceptualize this matching, we consider spikes as instantaneous “samples” of the somatic current. The Sampling Theorem is then applicable, and it suggests that the bandwidth of the injected signal not exceed half the spike generator's average firing rate. The average firing rate, in turn, depends on the amplitude range and DC bias of the injected signal. We hypothesize that nature faces similar problems and constraints when transmitting a time-varying waveform from the soma of one neuron to the dendrite of the postsynaptic cell.
A Model of Spatial Map Formation in the Hippocampus of the Rat
Authors: Blum Kenneth I. , Abbott L. F.
Published: April 2008

Using experimental facts about long-term potentiation (LTP) and hippocampal place cells, we model how a spatial map of the environment can be created in the rat hippocampus. Sequential firing of place cells during exploration induces, in the model, a pattern of LTP between place cells that shifts the location coded by their ensemble activity away from the actual location of the animal. These shifts provide a navigational map that, in a simulation of the Morris maze, can guide the animal toward its goal. The model demonstrates how behaviorally generated modifications of synaptic strengths can be read out to affect subsequent behavior. Our results also suggest a way that navigational maps can be constructed from experimental recordings of hippocampal place cells.
A Neural Model of Olfactory Sensory Memory in the Honeybee's Antennal Lobe
Authors: Linster Christiane , Masson Claudine
Published: April 2008

We present a neural model for olfactory sensory memory in the honeybee's antennal lobe. To investigate the neural mechanisms underlying odor discrimination and memorization, we exploit a variety of morphological, physiological, and behavioral data. The model allows us to study the computational capacities of the known neural circuitry, and to interpret under a new light experimental data on the cellular as well as on the neuronal assembly level. We propose a scheme for memorization of the neural activity pattern after stimulus offset by changing the local balance between excitation and inhibition. This modulation is achieved by changing the intrinsic parameters of local inhibitory neurons or synapses.
Associative Memory with Uncorrelated Inputs
Authors: Michaels Ronald
Published: April 2008

In hybrid learning schemes a layer of unsupervised learning is followed by supervised learning. In this situation a connection between two unsupervised learning algorithms, principal component analysis and decorrelation, and a supervised learning algorithm, associative memory, is shown. When associative memory is preceded by principal component analysis or decorrelation it is possible to take advantage of the lack of correlation among inputs to associative memory to show that correlation matrix memory is a least squares solution to the supervised learning problem.
Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps
Authors: Parra Lucas , Deco Gustavo , Miesbach Stefan
Published: April 2008

According to Barlow (1989), feature extraction can be understood as finding a statistically independent representation of the probability distribution underlying the measured signals. The search for a statistically independent representation can be formulated by the criterion of minimal mutual information, which reduces to decorrelation in the case of gaussian distributions. If nongaussian distributions are to be considered, minimal mutual information is the appropriate generalization of decorrelation as used in linear Principal Component Analyses (PCA). We also generalize to nonlinear transformations by only demanding perfect transmission of information. This leads to a general class of nonlinear transformations, namely symplectic maps. Conservation of information allows us to consider only the statistics of single coordinates. The resulting factorial representation of the joint probability distribution gives a density estimation. We apply this concept to the real world problem of electrical motor fault detection treated as a novelty detection task.
Neural Network Models of Perceptual Learning of Angle Discrimination
Authors: Mato G. , Sompolinsky H.
Published: April 2008

We study neural network models of discriminating between stimuli with two similar angles, using the two-alternative forced choice (2AFC) paradigm. Two network architectures are investigated: a two-layer perceptron network and a gating network. In the two-layer network all hidden units contribute to the decision at all angles, while in the other architecture the gating units select, for each stimulus, the appropriate hidden units that will dominate the decision. We find that both architectures can perform the task reasonably well for all angles. Perceptual learning has been modeled by training the networks to perform the task, using unsupervised Hebb learning algorithms with pairs of stimuli at fixed angles θ and δθ. Perceptual transfer is studied by measuring the performance of the network on stimuli with θ′ ≠ θ. The two-layer perceptron shows a partial transfer for angles that are within a distance a from θ, where a is the angular width of the input tuning curves. The change in performance due to learning is positive for angles close to θ, but for |θ − θ′| ≈ a it is negative, i.e., its performance after training is worse than before. In contrast, negative transfer can be avoided in the gating network by limiting the effects of learning to hidden units that are optimized for angles that are close to the trained angle.
Directional Filling-in
Authors: Arrington Karl Frederick
Published: April 2008

The filling-in theory of brightness perception has gained much attention recently owing to the success of vision models. However, the theory and its instantiations have suffered from incorrectly dealing with transitive brightness relations. This paper describes an advance in the filling-in theory that overcomes the problem. The advance is incorporated into the BCS/FCS neural network model, which allows it, for the first time, to account for all of Arend's test stimuli for assessing brightness perception models. The theory also suggests a new teleology for parallel ON- and OFF-channels.
Annealed Competition of Experts for a Segmentation and Classification of Switching Dynamics
Authors: Pawelzik Klaus , Kohlmorgen Jens , Müller Klaus-Robert
Published: April 2008

We present a method for the unsupervised segmentation of data streams originating from different unknown sources that alternate in time. We use an architecture consisting of competing neural networks. Memory is included to resolve ambiguities of input-output relations. To obtain maximal specialization, the competition is adiabatically increased during training. Our method achieves almost perfect identification and segmentation in the case of switching chaotic dynamics where input manifolds overlap and input-output relations are ambiguous. Only a small dataset is needed for the training procedure. Applications to time series from complex systems demonstrate the potential relevance of our approach for time series analysis and short-term prediction.
The Computational Power of Discrete Hopfield Nets with Hidden Units
Authors: Orponen Pekka
Published: April 2008

We prove that polynomial size discrete Hopfield networks with hidden units compute exactly the class of Boolean functions PSPACE/poly, i.e., the same functions as are computed by polynomial space-bounded nonuniform Turing machines. As a corollary to the construction, we observe also that networks with polynomially bounded interconnection weights compute exactly the class of functions P/poly, i.e., the class computed by polynomial time-bounded nonuniform Turing machines.
A Self-Organizing Neural Network for the Traveling Salesman Problem That Is Competitive with Simulated Annealing
Authors: Budinich Marco
Published: April 2008

Unsupervised learning applied to an unstructured neural network can give approximate solutions to the traveling salesman problem. For 50 cities in the plane this algorithm performs like the elastic net of Durbin and Willshaw (1987) and it improves when increasing the number of cities to get better than simulated annealing for problems with more than 500 cities. In all the tests this algorithm requires a fraction of the time taken by simulated annealing.
The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks
Authors: Thimm Georg , Moerland Perry , Fiesler Emile
Published: April 2008

The backpropagation algorithm is widely used for training multilayer neural networks. In this publication the gain of its activation function(s) is investigated. In specific, it is proven that changing the gain of the activation function is equivalent to changing the learning rate and the weights. This simplifies the backpropagation learning rule by eliminating one of its parameters. The theorem can be extended to hold for some well-known variations on the backpropagation algorithm, such as using a momentum term, flat spot elimination, or adaptive gain. Furthermore, it is successfully applied to compensate for the nonstandard gain of optical sigmoids for optical neural networks.
A Smoothing Regularizer for Feedforward and Recurrent Neural Networks
Authors: Wu Lizhong , Moody John
Published: April 2008

We derive a smoothing regularizer for dynamic network models by requiring robustness in prediction performance to perturbations of the training data. The regularizer can be viewed as a generalization of the first-order Tikhonov stabilizer to dynamic models. For two layer networks with recurrent connections described by

the training criterion with the regularizer is

where Φ = {U, V, W} is the network parameter set, Z(t) are the targets, I(t) = {X(s), s = 1,2, …, t} represents the current and all historical input information, N is the size of the training data set, [Formula: see text] is the regularizer, and λ is a regularization parameter. The closed-form expression for the regularizer for time-lagged recurrent networks is

where ‖ ‖ is the Euclidean matrix norm and γ is a factor that depends upon the maximal value of the first derivatives of the internal unit activations f(). Simplifications of the regularizer are obtained for simultaneous recurrent nets (τ ↦ 0), two-layer feedforward nets, and one layer linear nets. We have successfully tested this regularizer in a number of case studies and found that it performs better than standard quadratic weight decay.
Optimizing Synaptic Conductance Calculation for Network Simulations
Authors: Lytton William W.
Published: April 2008

High computational requirements in realistic neuronal network simulations have led to attempts to realize implementation efficiencies while maintaining as much realism as possible. Since the number of synapses in a network will generally far exceed the number of neurons, simulation of synaptic activation may be a large proportion of total processing time. We present a consolidating algorithm based on a recent biophysically-inspired simplified Markov model of the synapse. Use of a single lumped state variable to represent a large number of converging synaptic inputs results in substantial speed-ups.
Energy Efficient Neural Codes
Authors: Levy William B , Baxter Robert A.
Published: April 2008

In 1969 Barlow introduced the phrase “economy of impulses” to express the tendency for successive neural systems to use lower and lower levels of cell firings to produce equivalent encodings. From this viewpoint, the ultimate economy of impulses is a neural code of minimal redundancy. The hypothesis motivating our research is that energy expenditures, e.g., the metabolic cost of recovering from an action potential relative to the cost of inactivity, should also be factored into the economy of impulses. In fact, coding schemes with the largest representational capacity are not, in general, optimal when energy expenditures are taken into account. We show that for both binary and analog neurons, increased energy expenditure per neuron implies a decrease in average firing rate if energy efficient information transmission is to be maintained.
Predictive Minimum Description Length Criterion for Time Series Modeling with Neural Networks
Authors: Lehtokangas Mikko , Saarinen Jukka , Huuhtanen Pentti , Kaski Kimmo
Published: April 2008

Nonlinear time series modeling with a multilayer perceptron network is presented. An important aspect of this modeling is the model selection, i.e., the problem of determining the size as well as the complexity of the model. To overcome this problem we apply the predictive minimum description length (PMDL) principle as a minimization criterion. In the neural network scheme it means minimizing the number of input and hidden units. Three time series modeling experiments are used to examine the usefulness of the PMDL model selection scheme. A comparison with the widely used cross-validation technique is also presented. In our experiments the PMDL scheme and the cross-validation scheme yield similar results in terms of model complexity. However, the PMDL method was found to be two times faster to compute. This is significant improvement since model selection in general is very time consuming.
Minimum Description Length, Regularization, and Multimodal Data
Authors: Rohwer Richard , van der Rest John C.
Published: April 2008

Relationships between clustering, description length, and regularization are pointed out, motivating the introduction of a cost function with a description length interpretation and the unusual and useful property of having its minimum approximated by the densest mode of a distribution. A simple inverse kinematics example is used to demonstrate that this property can be used to select and learn one branch of a multivalued mapping. This property is also used to develop a method for setting regularization parameters according to the scale on which structure is exhibited in the training data. The regularization technique is demonstrated on two real data sets, a classification problem and a regression problem.
VC Dimension of an Integrate-and-Fire Neuron Model
Authors: Zador Anthony M. , Pearlmutter Barak A.
Published: April 2008

We compute the VC dimension of a leaky integrate-and-fire neuron model. The VC dimension quantifies the ability of a function class to partition an input pattern space, and can be considered a measure of computational capacity. In this case, the function class is the class of integrate-and-fire models generated by varying the integration time constant T and the threshold θ, the input space they partition is the space of continuous-time signals, and the binary partition is specified by whether or not the model reaches threshold at some specified time. We show that the VC dimension diverges only logarithmically with the input signal bandwidth N. We also extend this approach to arbitrary passive dendritic trees. The main contributions of this work are (1) it offers a novel treatment of computational capacity of this class of dynamic system; and (2) it provides a framework for analyzing the computational capabilities of the dynamic systems defined by networks of spiking neurons.
A Theory of the Visual Motion Coding in the Primary Visual Cortex
Authors: Li Zhaoping
Published: April 2008

This paper demonstrates that much of visual motion coding in the primary visual cortex can be understood from a theory of efficient motion coding in a multiscale representation. The theory predicts that cortical cells can have a spectrum of directional indices, be tuned to different directions of motion, and have spatiotemporally separable or inseparable receptive fields (RF). The predictions also include the following correlations between motion coding and spatial, chromatic, and stereo codings: the preferred speed is greater when the cell receptive field size is larger, the color channel prefers lower speed than the luminance channel, and both the optimal speeds and the preferred directions of motion can be different for inputs from different eyes to the same neuron. These predictions agree with experimental observations. In addition, this theory makes predictions that have not been experimentally investigated systematically and provides a testing ground for an efficient multiscale coding framework. These predictions are as follows: (1) if nearby cortical cells of a given preferred orientation and scale prefer opposite directions of motion and have a quadrature RF phase relationship with each other, then they will have the same directional index, (2) a single neuron can have different optimal motion speeds for opposite motion directions of monocular stimuli, and (3) a neuron's ocular dominance may change with motion direction if the neuron prefers opposite directions for inputs from different eyes.
Alignment of Coexisting Cortical Maps in a Motor Control Model
Authors: Chen Yinong , Reggia James A.
Published: April 2008

How do multiple feature maps that coexist in the same region of cerebral cortex align with each other? We hypothesize that such alignment is governed by temporal correlations: features in one map that are temporally correlated with those in another come to occupy the same spatial locations in cortex over time. To examine the feasibility of this hypothesis and to establish some of its detailed implications, we studied a multilayered, closed-loop computational model of primary sensorimotor cortex. A simulated arm moving in three dimensions formed the external environment for the model cortical regions. Coexisting proprioceptive and motor maps formed and generally aligned in a fashion consistent with the temporal correlation hypothesis. For example, in simulated proprioceptive sensory cortex the map of elements responding strongly to stretch of a particular muscle matched the map of tension sensitivity in antagonist muscles. In simulated primary motor cortex the map of elements responding strongly to increased tension in specific muscles matched the map of output elements for the same muscles. These computational results suggest specific experimental measurements that can support or refute the temporal correlation hypothesis for map alignments.
Semilinear Predictability Minimization Produces Well-Known Feature Detectors
Authors: Schmidhuber Jürgen , Eldracher Martin , Foltin Bernhard
Published: April 2008

Predictability minimization (PM—Schmidhuber 1992) exhibits various intuitive and theoretical advantages over many other methods for unsupervised redundancy reduction. So far, however, there have not been any serious practical applications of PM. In this paper, we apply semilinear PM to static real world images and find that without a teacher and without any significant preprocessing, the system automatically learns to generate distributed representations based on well-known feature detectors, such as orientation-sensitive edge detectors and off-center–on-surround detectors, thus extracting simple features related to those considered useful for image preprocessing and compression.
Learning with Preknowledge: Clustering with Point and Graph Matching Distance Measures
Authors: Gold Steven , Rangarajan Anand , Mjolsness Eric
Published: April 2008

Prior knowledge constraints are imposed upon a learning problem in the form of distance measures. Prototypical 2D point sets and graphs are learned by clustering with point-matching and graph-matching distance measures. The point-matching distance measure is approximately invariant under affine transformations—translation, rotation, scale, and shear—and permutations. It operates between noisy images with missing and spurious points. The graph-matching distance measure operates on weighted graphs and is invariant under permutations. Learning is formulated as an optimization problem. Large objectives so formulated (∼ million variables) are efficiently minimized using a combination of optimization techniques—softassign, algebraic transformations, clocked objectives, and deterministic annealing.
On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions
Authors: Niyogi Partha , Girosi Federico
Published: April 2008

Feedforward networks together with their training algorithms are a class of regression techniques that can be used to learn to perform some task from a set of examples. The question of generalization of network performance from a finite training set to unseen data is clearly of crucial importance. In this article we first show that the generalization error can be decomposed into two terms: the approximation error, due to the insufficient representational capacity of a finite sized network, and the estimation error, due to insufficient information about the target function because of the finite number of samples. We then consider the problem of learning functions belonging to certain Sobolev spaces with gaussian radial basis functions. Using the above-mentioned decomposition we bound the generalization error in terms of the number of basis functions and number of examples. While the bound that we derive is specific for radial basis functions, a number of observations deriving from it apply to any approximation technique. Our result also sheds light on ways to choose an appropriate network architecture for a particular problem and the kinds of problems that can be effectively solved with finite resources, i.e., with a finite number of parameters and finite amounts of data.
On Neurodynamics with Limiter Function and Linsker's Developmental Model
Authors: Feng Jianfeng , Pan Hong , Roychowdhury Vwani P.
Published: April 2008

The limiter function is used in many learning and retrieval models as the constraint controlling the magnitude of the weight or state vectors. In this paper, we developed a new method to relate the set of saturated fixed points to the set of system parameters of the models that use the limiter function, and then, as a case study, applied this method to Linsker's Hebbian learning network. We derived a necessary and sufficient condition to test whether a given saturated weight or state vector is stable or not for any given set of system parameters, and used this condition to determine the whole regime in the parameter space over which the given state is stable. This approach allows us to investigate the relative stability of the major receptive fields reported in Linsker's simulations, and to demonstrate the crucial role played by the synaptic density functions.
Effect of Binocular Cortical Misalignment on Ocular Dominance and Orientation Selectivity
Authors: Shouval Harel , Intrator Nathan , Law C. Charles , Cooper Leon N
Published: April 2008

We model a two-eye visual environment composed of natural images and study its effect on single cell synaptic modification. In particular, we study the effect of binocular cortical misalignment on receptive field formation after eye opening. We show that binocular misalignment affects principal component analysis (PCA) and Bienenstock, Cooper, and Munro (BCM) learning in different ways. For the BCM learning rule this misalignment is sufficient to produce varying degrees of ocular dominance, whereas for PCA learning binocular neurons emerge in every case.
Gradient Projection Network: Analog Solver for Linearly Constrained Nonlinear Programming
Authors: Urahama Kiichi
Published: April 2008

An analog approach is presented for solving nonlinear programming problems with linear constraint conditions. The present method is based on transformation of variables with exponential functions, which enables every trajectory to pass through an interior of feasible regions along a gradient direction projected onto the feasible space. Convergence of its trajectory to the solution of optimization problems is guaranteed and it is shown that the present scheme is an extension of the affine scaling method for linear programming to nonlinear programs under a slight modification of Riemannian metric. An analog electronic circuit is also presented that implements the proposed scheme in real time.
Rate of Convergence in Density Estimation Using Neural Networks
Authors: Modha Dharmendra S. , Masry Elias
Published: April 2008

Given N i.i.d. observations {Xi}Ni=1 taking values in a compact subset of Rd, such that p* denotes their common probability density function, we estimate p* from an exponential family of densities based on single hidden layer sigmoidal networks using a certain minimum complexity density estimation scheme. Assuming that p* possesses a certain exponential representation, we establish a rate of convergence, independent of the dimension d, for the expected Hellinger distance between the proposed minimum complexity density estimator and the true underlying density p*.
Modeling Slowly Bursting Neurons via Calcium Store and Voltage-Independent Calcium Current
Authors: Chay Teresa Ree
Published: April 2008

Recent experiments indicate that the calcium store (e.g., endoplasmic reticulum) is involved in electrical bursting and [Ca2+]i oscillation in bursting neuronal cells. In this paper, we formulate a mathematical model for bursting neurons, which includes Ca2+ in the intracellular Ca2+ stores and a voltage-independent calcium channel (VICC). This VICC is activated by a depletion of Ca2+ concentration in the store, [Ca2+]CS. In this model, [Ca2+]CS oscillates slowly, and this slow dynamic in turn gives rise to electrical bursting. The newly formulated model thus is radically different from existing models of bursting excitable cells, whose mechanism owes its origin to the ion channels in the plasma membrane and the [Ca2+]i dynamics. In addition, this model is capable of providing answers to some puzzling phenomena, which the previous models could not (e.g., why cAMP, glucagon, and caffeine have ability to change the burst periodicity). Using mag-fura-2 fluorescent dyes, it would be interesting to verify the prediction of the model that (1) [Ca2+]CS oscillates in bursting neurons such as Aplysia neuron and (2) the neurotransmitters and hormones that affect the adenylate cyclase pathway can influence this oscillation.
The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction
Authors: Casey Mike
Published: April 2008

Recurrent neural networks (RNNs) can learn to perform finite state computations. It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given. This knowledge effectively predicts activation space dynamics, which allows one to understand RNN computation dynamics in spite of complexity in activation dynamics. This theory provides a theoretical framework for understanding finite state machine (FSM) extraction techniques and can be used to improve training methods for RNNs performing FSM computations. This provides an example of a successful approach to understanding a general class of complex systems that has not been explicitly designed, e.g., systems that have evolved or learned their internal structure.
Temporal Precision of Spike Trains in Extrastriate Cortex of the Behaving Macaque Monkey
Authors: Bair Wyeth , Koch Christof
Published: April 2008

How reliably do action potentials in cortical neurons encode information about a visual stimulus? Most physiological studies do not weigh the occurrences of particular action potentials as significant but treat them only as reflections of average neuronal excitation. We report that single neurons recorded in a previous study by Newsome et al. (1989; see also Britten et al. 1992) from cortical area MT in the behaving monkey respond to dynamic and unpredictable motion stimuli with a markedly reproducible temporal modulation that is precise to a few milliseconds. This temporal modulation is stimulus dependent, being present for highly dynamic random motion but absent when the stimulus translates rigidly.
Neural Network for Dynamic Binding with Graph Representation: Form, Linking, and Depth-from-Occlusion
Authors: Williamson James R.
Published: April 2008

A neural network is presented that explicitly represents form attributes and relations between them, thus solving the binding problem without temporal coding. Rather, the network creates a graph representation by dynamically allocating nodes to code local form attributes and establishing arcs to link them. With this representation, the network selectively groups and segments in depth objects based on line junction information, producing results consistent with those of several recent visual search experiments. In addition to depth-from-occlusion, the network provides a sufficient framework for local line-labeling processes to recover other three-dimensional (3-D) variables, such as edge/surface contiguity, edge slant, and edge convexity.
Spike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in Dendrites
Authors: Northmore David P. M. , Elias John G.
Published: April 2008

A dendritic tree, as part of a silicon neuromorph, was modeled in VLSI as a multibranched, passive cable structure with multiple synaptic sites that either depolarize or hyperpolarize local “membrane patches,” thereby raising or lowering the probability of spike generation of an integrate-and-fire “soma.” As expected from previous theoretical analyses, contemporaneous synaptic activation at widely separated sites on the artificial tree resulted in near-linear summation, as did neighboring excitatory and inhibitory activations. Activation of synapses of the same type close in time and space produced local saturation of potential, resulting in spike train processing capabilities not possible with linear summation alone. The resulting sublinear synaptic summation, as well as being physiologically plausible, is sufficient for a variety of spike train processing functions. With the appropriate arrangement of synaptic inputs on its dendritic tree, a neuromorph was shown to discriminate input pulse intervals and patterns, pulse train frequencies, and detect correlation between input trains.
Vapnik-Chervonenkis Generalization Bounds for Real Valued Neural Networks
Authors: Hole Arne
Published: April 2008

We show how lower bounds on the generalization ability of feedforward neural nets with real outputs can be derived within a formalism based directly on the concept of VC dimension and Vapnik's theorem on uniform convergence of estimated probabilities.
The Lack of A Priori Distinctions Between Learning Algorithms
Authors: Wolpert David H.
Published: April 2008

This is the first of two papers that use off-training set (OTS) error to investigate the assumption-free relationship between learning algorithms. This first paper discusses the senses in which there are no a priori distinctions between learning algorithms. (The second paper discusses the senses in which there are such distinctions.) In this first paper it is shown, loosely speaking, that for any two algorithms A and B, there are “as many” targets (or priors over targets) for which A has lower expected OTS error than B as vice versa, for loss functions like zero-one loss. In particular, this is true if A is cross-validation and B is “anti-cross-validation” (choose the learning algorithm with largest cross-validation error). This paper ends with a discussion of the implications of these results for computational learning theory. It is shown that one cannot say: if empirical misclassification rate is low, the Vapnik-Chervonenkis dimension of your generalizer is small, and the training set is large, then with high probability your OTS error is small. Other implications for “membership queries” algorithms and “punting” algorithms are also discussed.
The Existence of A Priori Distinctions Between Learning Algorithms
Authors: Wolpert David H.
Published: April 2008

This is the second of two papers that use off-training set (OTS) error to investigate the assumption-free relationship between learning algorithms. The first paper discusses a particular set of ways to compare learning algorithms, according to which there are no distinctions between learning algorithms. This second paper concentrates on different ways of comparing learning algorithms from those used in the first paper. In particular this second paper discusses the associated a priori distinctions that do exist between learning algorithms. In this second paper it is shown, loosely speaking, that for loss functions other than zero-one (e.g., quadratic loss), there are a priori distinctions between algorithms. However, even for such loss functions, it is shown here that any algorithm is equivalent on average to its “randomized” version, and in this still has no first principles justification in terms of average error. Nonetheless, as this paper discusses, it may be that (for example) cross-validation has better head-to-head minimax properties than “anti-cross-validation” (choose the learning algorithm with the largest cross-validation error). This may be true even for zero-one loss, a loss function for which the notion of “randomization” would not be relevant. This paper also analyzes averages over hypotheses rather than targets. Such analyses hold for all possible priors over targets. Accordingly they prove, as a particular example, that cross-validation cannot be justified as a Bayesian procedure. In fact, for a very natural restriction of the class of learning algorithms, one should use anti-cross-validation rather than cross-validation (!).