bluuebunny/crossref_metadata_2025 huggingface dataset:
Feature Linking via Synchronization among Distributed Assemblies: Simulations of Results from Cat Visual Cortex
Authors: Eckhorn R. , Reitboeck H. J. , Arndt M. , Dicke P.
Published: March 2008

We recently discovered stimulus-specific interactions between cell assemblies in cat primary visual cortex that could constitute a global linking principle for feature associations in sensory and motor systems: stimulus-induced oscillatory activities (35-80 Hz) in remote cell assemblies of the same and of different visual cortex areas mutually synchronize, if common stimulus features drive the assemblies simultaneously. Based on our neurophysiological findings we simulated feature linking via synchronizations in networks of model neurons. The networks consisted of two one-dimensional layers of neurons, coupled in a forward direction via feeding connections and in lateral and backward directions via modulatory linking connections. The models' performance is demonstrated in examples of region linking with spatiotemporally varying inputs, where the rhythmic activities in response to an input, that initially are uncorrelated, become phase locked. We propose that synchronization is a general principle for the coding of associations in and among sensory systems and that at least two distinct types of synchronization do exist: stimulus-forced (event-locked) synchronizations support “crude instantaneous” associations and stimulus-induced (oscillatory) synchronizations support more complex iterative association processes. In order to bring neural linking mechanisms into correspondence with perceptual feature linking, we introduce the concept of the linking field (association field) of a local assembly of visual neurons. The linking field extends the concept of the invariant receptive field (RF) of single neurons to the flexible association of RFs in neural assemblies.
Towards a Theory of Early Visual Processing
Authors: Atick Joseph J. , Redlich A. Norman
Published: March 2008

We propose a theory of the early processing in the mammalian visual pathway. The theory is formulated in the language of information theory and hypothesizes that the goal of this processing is to recode in order to reduce a “generalized redundancy” subject to a constraint that specifies the amount of average information preserved. In the limit of no noise, this theory becomes equivalent to Barlow's redundancy reduction hypothesis, but it leads to very different computational strategies when noise is present. A tractable approach for finding the optimal encoding is to solve the problem in successive stages where at each stage the optimization is performed within a restricted class of transfer functions. We explicitly find the solution for the class of encodings to which the parvocellular retinal processing belongs, namely linear and nondivergent transformations. The solution shows agreement with the experimentally observed transfer functions at all levels of signal to noise.
Spontaneous Development of Modularity in Simple Cortical Models
Authors: Chernjavsky Alex , Moody John
Published: March 2008

The existence of modular structures in the organization of nervous systems (e.g., cortical columns, patches of neostriatum, and olfactory glomeruli) is well known. However, the detailed dynamic mechanisms by which such structures develop remain a mystery. We propose a mechanism for the formation of modular structures that utilizes a combination of intrinsic network dynamics and Hebbian learning. Specifically, we show that under certain conditions, layered networks can support spontaneous localized activity patterns, which we call collective excitations, even in the absence of localized or spatially correlated afferent stimulation. These collective excitations can then induce the formation of modular structures in both the afferent and lateral connections via a Hebbian learning mechanism. The networks we consider are spatially homogeneous before learning, but the spontaneous emergence of localized collective excitations and the consequent development of modules in the connection patterns break translational symmetry. The essential conditions required to support collective excitations include internal units with sufficiently high gains and certain patterns of lateral connectivity. Our proposed mechanism is likely to play a role in understanding more complex (and more biologically realistic) systems.
The Bootstrap Widrow-Hoff Rule as a Cluster-Formation Algorithm
Authors: Hinton Geoffrey E. , Nowlan Steven J.
Published: March 2008

An algorithm that is widely used for adaptive equalization in current modems is the “bootstrap” or “decision-directed” version of the Widrow-Hoff rule. We show that this algorithm can be viewed as an unsupervised clustering algorithm in which the data points are transformed so that they form two clusters that are as tight as possible. The standard algorithm performs gradient ascent in a crude model of the log likelihood of generating the transformed data points from two gaussian distributions with fixed centers. Better convergence is achieved by using the exact gradient of the log likelihood.
Discovering the Structure of a Reactive Environment by Exploration
Authors: Mozer Michael C. , Bachrach Jonathan
Published: March 2008

Consider a robot wandering around an unfamiliar environment, performing actions and observing the consequences. The robot's task is to construct an internal model of its environment, a model that will allow it to predict the effects of its actions and to determine what sequences of actions to take to reach particular goal states. Rivest and Schapire (1987a,b; Schapire 1988) have studied this problem and have designed a symbolic algorithm to strategically explore and infer the structure of “finite state” environments. The heart of this algorithm is a clever representation of the environment called an update graph. We have developed a connectionist implementation of the update graph using a highly specialized network architecture. With backpropagation learning and a trivial exploration strategy — choosing random actions — the connectionist network can outperform the Rivest and Schapire algorithm on simple problems. Our approach has additional virtues, including the fact that the network can accommodate stochastic environments and that it suggests generalizations of the update graph representation that do not arise from a traditional, symbolic perspective.
Robust Classifiers without Robust Features
Authors: Katz Alan J. , Gately Michael T. , Collins Dean R.
Published: March 2008

We develop a two-stage, modular neural network classifier and apply it to an automatic target recognition problem. The data are features extracted from infrared and TV images. We discuss the problem of robust classification in terms of a family of decision surfaces, the members of which are functions of a set of global variables. The global variables characterize how the feature space changes from one image to the next. We obtain rapid training times and robust classification with this modular neural network approach.
Phase Transitions in Connectionist Models Having Rapidly Varying Connection Strengths
Authors: Reggia James A. , Edwards Mark
Published: March 2008

A phase transition in a connectionist model refers to a qualitative change in the model's behavior as parameters determining the spread of activation (gain, decay rate, etc.) pass through certain critical values. As connectionist methods have been increasingly adopted to model various problems in neuroscience, artificial intelligence, and cognitive science, there has been an increased need to understand and predict these phase transitions to assure meaningful model behavior. This paper extends previous results on phase transitions to encompass a class of connectionist models having rapidly varying connection strengths (“fast weights”). Phase transitions are predicted theoretically and then verified through a series of computer simulations. These results broaden the range of connectionist models for which phase transitions are identified and lay the foundation for future studies comparing models with rapidly varying and slowly varying connection strengths.
An M-ary Neural Network Model
Authors: Bijjani R. , Das P.
Published: March 2008

An M-ary neural network model is described, and is shown to have a higher error correction capability than the bipolar Hopfield neural net. The M-ary model is then applied to an image recognition problem.
Dynamics and Formation of Self-Organizing Maps
Authors: Zhang Jun
Published: March 2008

Amari (1983, 1989) proposed a mathematical formulation on the self-organization of synaptic efficacies and neural response fields under the influence of external stimuli. The dynamics as well as the equilibrium properties of the cortical map were obtained analytically for neurons with binary input-output transfer functions. Here we extend this approach to neurons with arbitrary sigmoidal transfer function. Under the assumption that both the intracortical connection and the stimulus-driven thalamic activity are well localized, we are able to derive expressions for the cortical magnification factor, the point-spread resolution, and the bandwidth resolution of the map. As a highlight, we show analytically that the receptive field size of a cortical neuron in the map is inversely proportional to the cortical magnification factor at that map location, the experimentally well-established rule of inverse magnification in retinotopic and somatotopic maps.
Sequence Manipulation Using Parallel Mapping Networks
Authors: Touretzky David S. , Wheeler Deirdre W.
Published: March 2008

We describe a parallel mapping matrix that performs several types of sequence manipulations that are the building blocks of well-known phonological processes. Our results indicate that human phonological behavior can by modeled by a highly constrained connectionist architecture, one that uses purely feedforward circuitry and imposes tight limits on depth of derivations.
Pacemaker-Induced Coherence in Cortical Networks
Authors: Destexhe A. , Babloyantz A.
Published: March 2008

A simple mathematical model of cortical tissue is introduced and the system's dynamics is monitored when a small subset of neurons is submitted to oscillatory inputs of various frequency and waveform. In the absence of input, the system shows desynchronized or “turbulent” behavior. The oscillatory input synchronizes the neuronal activity, which is strongest for inputs of low frequency. The increase of spatial coherence is estimated from the spatial autocorrelation function whereas the increase in temporal coherence is evaluated from correlation dimensions. The model accounts qualitatively for some of the features of the thalamocortical system.
Stimulus-Dependent Assembly Formation of Oscillatory Responses: I. Synchronization
Authors: König Peter , Schillen Thomas B.
Published: March 2008

Current concepts in neurobiology of vision assume that local object features are represented by distributed neuronal populations in the brain. Such representations can lead to ambiguities if several distinct objects are simultaneously present in the visual field. Temporal characteristics of the neuronal activity have been proposed as a possible solution to this problem and have been found in various cortical areas.

In this paper we introduce a delayed nonlinear oscillator to investigate temporal coding in neuronal networks. We show synchronization within two-dimensional layers consisting of oscillatory elements coupled by excitatory delay connections. The observed correlation length is large compared to coupling length. Following the experimental situation, we then demonstrate the response of such layers to two short stimulus bars of varying gap distance. Coherency of stimuli is reflected by the temporal correlation of the responses, which closely resembles the experimental observations.
Universal Approximation Using Radial-Basis-Function Networks
Authors: Park J. , Sandberg I. W.
Published: March 2008

There have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables. Some of these studies deal with cases in which the hidden-layer nonlinearity is not a sigmoid. This was motivated by successful applications of feedforward networks with nonsigmoidal hidden-layer units.

This paper reports on a related study of radial-basis-function (RBF) networks, and it is proved that RBF networks having one hidden layer are capable of universal approximation. Here the emphasis is on the case of typical RBF networks, and the results show that a certain class of RBF networks with the same smoothing factor in each kernel node is broad enough for universal approximation.
Recognizing Hand-Printed Letters and Digits Using Backpropagation Learning
Authors: Martin Gale L. , Pittman James A.
Published: March 2008

We report on results of training backpropagation nets with samples of hand-printed digits scanned off of bank checks and hand-printed letters interactively entered into a computer through a stylus digitizer. Generalization results are reported as a function of training set size and network capacity. Given a large training set, and a net with sufficient capacity to achieve high performance on the training set, nets typically achieved error rates of 4-5% at a 0% reject rate and 1-2% at a 10% reject rate. The topology and capacity of the system, as measured by the number of connections in the net, have surprisingly little effect on generalization. For those developing hand-printed character recognition systems, these results suggest that a large and representative training sample may be the single, most important factor in achieving high recognition accuracy. Benefits of reducing the number of net connections, other than improving generalization, are discussed.
A Practical Approach for Representing Context and for Performing Word Sense Disambiguation Using Neural Networks
Authors: Gallant Stephen I.
Published: March 2008

Representing and manipulating context information is one of the hardest problems in natural language processing. This paper proposes a method for representing some context information so that the correct meaning for a word in a sentence can be selected. The approach is primarily based on work by Waltz and Pollack (1985, 1984), who emphasized neutrally plausible systems. By contrast this paper focuses on computationally feasible methods applicable to full-scale natural language processing systems.

There are two key elements: a collection of context vectors defined for every word used by a natural language processing system, and a context algorithm that computes a dynamic context vector at any position in a body of text. Once the dynamic context vector has been computed it is easy to choose among competing meanings for a word. This choice of definitions is essentially a neural network computation, and neural network learning algorithms should be able to improve such choices.

Although context vectors do not represent all context information, their use should improve those full-scale systems that have avoided context as being too difficult to deal with. Good candidates for full-scale context vector implementations are machine translation systems and Japanese word processors. A main goal of this paper is to encourage such large-scale implementations and tests of context vector approaches.

A variety of interesting directions for research in natural language processing and machine learning will be possible once a full set of context vectors has been created. In particular the development of more powerful context algorithms will be an important topic for future research.
Parameter Sensitivity of the Elastic Net Approach to the Traveling Salesman Problem
Authors: Simmen Martin W.
Published: March 2008

Durbin and Willshaw's elastic net algorithm can find good solutions to the TSP. The purpose of this paper is to point out that for certain ranges of parameter values, the algorithm converges into local minima that do not correspond to valid tours. The key parameter is the ratio governing the relative strengths of the two competing terms in the elastic net energy function. Based on recent work by Durbin, Szeliski and Yuille, the parameter regime in which the net may visit some cities twice is examined. Further analysis predicts the regime in which the net may fail to visit some cities at all. Understanding these limitations allows one to select the parameter value most likely to avoid either type of problem. Simulation data support the theoretical work.
FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling
Authors: Back A. D. , Tsoi A. C.
Published: March 2008

A new neural network architecture involving either local feedforward global feedforward, and/or local recurrent global feedforward structure is proposed. A learning rule minimizing a mean square error criterion is derived. The performance of this algorithm (local recurrent global feedforward architecture) is compared with a local-feedforward global-feedforward architecture. It is shown that the local-recurrent global-feedforward model performs better than the local-feedforward global-feedforward model.
Including Hints in Training Neural Nets
Authors: Al-Mashouq Khalid A. , Reed Irving S.
Published: March 2008

The aim of a neural net is to partition the data space into near optimal decision regions. Learning such a partitioning solely from examples has proven to be a very hard problem (Blum and Rivest 1988; Judd 1988). To remedy this, we use the idea of supplying hints to the network—as discussed by Abu-Mostafa (1990). Hints reduce the solution space, and as a consequence speed up the learning process. The minimum Hamming distance between the patterns serves as the hint. Next, it is shown how to learn such a hint and how to incorporate it into the learning algorithm. Modifications in the net structure and its operation are suggested, which allow for a better generalization. The sensitivity to errors in such a hint is studied through some simulations.
Handwritten Digit Recognition Using K Nearest-Neighbor, Radial-Basis Function, and Backpropagation Neural Networks
Authors: Lee Yuchun
Published: March 2008

Results of recent research suggest that carefully designed multilayer neural networks with local “receptive fields” and shared weights may be unique in providing low error rates on handwritten digit recognition tasks. This study, however, demonstrates that these networks, radial basis function (RBF) networks, and k nearest-neighbor (kNN) classifiers, all provide similar low error rates on a large handwritten digit database. The backpropagation network is overall superior in memory usage and classification time but can provide “false positive” classifications when the input is not a digit. The backpropagation network also has the longest training time. The RBF classifier requires more memory and more classification time, but less training time. When high accuracy is warranted, the RBF classifier can generate a more effective confidence judgment for rejecting ambiguous inputs. The simple kNN classifier can also perform handwritten digit recognition, but requires a prohibitively large amount of memory and is much slower at classification. Nevertheless, the simplicity of the algorithm and fast training characteristics makes the kNN classifier an attractive candidate in hardware-assisted classification tasks. These results on a large, high input dimensional problem demonstrate that practical constraints including training time, memory usage, and classification time often constrain classifier selection more strongly than small differences in overall error rate.
A Matrix Method for Optimizing a Neural Network
Authors: Barton Simon A.
Published: March 2008

A matrix method is described that optimizes the set of weights and biases for the output side of a network with a single hidden layer of neurons, given any set of weights and biases for the input side of the hidden layer. All the input patterns are included in a single optimization cycle. A simple iterative minimization procedure is used to optimize the weights and biases on the input side of the hidden layer. Many test problems have been solved, confirming the validity of the method. The results suggest that for a network with a single layer of hidden sigmoidal nodes, the accuracy of a functional representation is reduced as the nonlinearity of the function increases.
A Computer Simulation of Oscillatory Behavior in Primary Visual Cortex
Authors: Wilson Matthew A. , Bower James M.
Published: March 2008

Periodic variations in correlated cellular activity have been observed in many regions of the cerebral cortex. The recent discovery of stimulus-dependent, spatially-coherent oscillations in primary visual cortex of the cat has led to suggestions of neural information encoding schemes based on phase and/or frequency variation. To explore the mechanisms underlying this behavior and their possible functional consequences, we have developed a realistic neural model, based on structural features of visual cortex, which replicates observed oscillatory phenomena. In the model, this oscillatory behavior emerges directly from the structure of the cortical network and the properties of its intrinsic neurons; however, phase coherence is shown to be an average phenomenon seen only when measurements are made over multiple trials. Because average coherence does not ensure synchrony of firing over the course of single stimuli, oscillatory phase may not be a robust strategy for directly encoding stimulus-specific information. Instead, the phase and frequency of cortical oscillations may reflect the coordination of general computational processes within and between cortical areas. Under this interpretation, coherence emerges as a result of horizontal interactions that could be involved in the formation of receptive field properties.
Segmentation, Binding, and Illusory Conjunctions
Authors: Horn D. , Sagi D. , Usher M.
Published: March 2008

We investigate binding within the framework of a model of excitatory and inhibitory cell assemblies that form an oscillating neural network. Our model is composed of two such networks that are connected through their inhibitory neurons. The excitatory cell assemblies represent memory patterns. The latter have different meanings in the two networks, representing two different attributes of an object, such as shape and color. The networks segment an input that contains mixtures of such pairs into staggered oscillations of the relevant activities. Moreover, the phases of the oscillating activities representing the two attributes in each pair lock with each other to demonstrate binding. The system works very well for two inputs, but displays faulty correlations when the number of objects is larger than two. In other words, the network conjoins attributes of different objects, thus showing the phenomenon of “illusory conjunctions,” as in human vision.
Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks
Authors: Jabri Marwan , Flower Barry
Published: March 2008

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms like backpropagation. Although backpropagation is efficient, its implementation in analog VLSI requires excessive computational hardware. In this paper we show that, for analog parallel implementations, the use of gradient descent with direct approximation of the gradient using “weight perturbation” instead of backpropagation significantly reduces hardware complexity. Gradient descent by weight perturbation eliminates the need for derivative and bidirectional circuits for on-chip learning, and access to the output states of neurons in hidden layers for off-chip learning. We also show that weight perturbation can be used to implement recurrent networks. A discrete level analog implementation showing the training of an XOR network as an example is described.
Temporal Evolution of Generalization during Learning in Linear Networks
Authors: Baldi Pierre , Chauvin Yves
Published: March 2008

We study generalization in a simple framework of feedforward linear networks with n inputs and n outputs, trained from examples by gradient descent on the usual quadratic error function. We derive analytical results on the behavior of the validation function corresponding to the LMS error function calculated on a set of validation patterns. We show that the behavior of the validation function depends critically on the initial conditions and on the characteristics of the noise. Under certain simple assumptions, if the initial weights are sufficiently small, the validation function has a unique minimum corresponding to an optimal stopping time for training for which simple bounds can be calculated. There exists also situations where the validation function can have more complicated and somewhat unexpected behavior such as multiple local minima (at most n) of variable depth and long but finite plateau effects. Additional results and possible extensions are briefly discussed.
Learning the Unlearnable
Authors: Nabutovsky Dan , Domany Eytan
Published: March 2008

We present a local perceptron-learning rule that either converges to a solution, or establishes linear nonseparability. We prove that when no solution exists, the algorithm detects this in a finite time (number of learning steps). This time is polynomial in typical cases and exponential in the worst case, when the set of patterns is nonstrictly linearly separable. The algorithm is local and has no arbitrary parameters.
Kolmogorov's Theorem Is Relevant
Authors: Kůrková Věra
Published: March 2008

We show that Kolmogorov's theorem on representations of continuous functions of n-variables by sums and superpositions of continuous functions of one variable is relevant in the context of neural networks. We give a version of this theorem with all of the one-variable functions approximated arbitrarily well by linear combinations of compositions of affine functions with some given sigmoidal function. We derive an upper estimate of the number of hidden units.
Neural Networks and the Bias/Variance Dilemma
Authors: Geman Stuart , Bienenstock Elie , Doursat René
Published: March 2008

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Authors: Schmidhuber Jürgen
Published: March 2008

Previous algorithms for supervised sequence learning are based on dynamic recurrent networks. This paper describes an alternative class of gradient-based systems consisting of two feedforward nets that learn to deal with temporal sequences using fast weights: The first net learns to produce context-dependent weight changes for the second net whose weights may vary very quickly. The method offers the potential for STM storage efficiency: A single weight (instead of a full-fledged unit) may be sufficient for storing temporal information. Various learning methods are derived. Two experiments with unknown time delays illustrate the approach. One experiment shows how the system can be used for adaptive temporary variable binding.
What Does the Retina Know about Natural Scenes?
Authors: Atick Joseph J. , Redlich A. Norman
Published: March 2008

By examining the experimental data on the statistical properties of natural scenes together with (retinal) contrast sensitivity data, we arrive at a first principle, theoretical hypothesis for the purpose of retinal processing and its relationship to an animal's environment. We argue that the retinal goal is to transform the visual input as much as possible into a statistically independent basis as the first step in creating a redundancy reduced representation in the cortex, as suggested by Barlow. The extent of this whitening of the input is limited, however, by the need to suppress input noise. Our explicit theoretical solutions for the retinal filters also show a simple dependence on mean stimulus luminance: they predict an approximate Weber law at low spatial frequencies and a De Vries-Rose law at high frequencies. Assuming that the dominant source of noise is quantum, we generate a family of contrast sensitivity curves as a function of mean luminance. This family is compared to psychophysical data.
A Simple Network Showing Burst Synchronization without Frequency Locking
Authors: Koch Christof , Schuster Heinz
Published: March 2008

The dynamic behavior of a network model consisting of all-to-all excitatory coupled binary neurons with global inhibition is studied analytically and numerically. We prove that for random input signals, the output of the network consists of synchronized bursts with apparently random intermissions of noisy activity. We introduce the fraction of simultaneously firing neurons as a measure for synchrony and prove that its temporal correlation function displays, besides a delta peak at zero indicating random processes, strongly dampened oscillations. Our results suggest that synchronous bursts can be generated by a simple neuronal architecture that amplifies incoming coincident signals. This synchronization process is accompanied by dampened oscillations that, by themselves, however, do not play any constructive role in this and can therefore be considered to be an epiphenomenon.
Learning Complex, Extended Sequences Using the Principle of History Compression
Authors: Schmidhuber Jürgen
Published: March 2008

Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to “divide and conquer” by recursively decomposing sequences. I describe two architectures. The first functions as a self-organizing multilevel hierarchy of recurrent networks. The second, involving only two recurrent networks, tries to collapse a multilevel predictor hierarchy into a single recurrent net. Experiments show that the system can require less computation per time step and many fewer training sequences than conventional training algorithms for recurrent nets.
A Competitive Distribution Theory of Neocortical Dynamics
Authors: Reggia James A. , D'Autrechy C. Lynne , Sutton Granger G. , Weinrich Michael
Published: March 2008

Peristimulus inhibition in sensory pathways is generally attributed to lateral inhibitory connections. However, in the neocortex circuitry is incompletely understood at present, and in some cases there is an apparent mismatch between observed inhibitory effects and intracortical inhibitory connections. This paper studies the hypothesis that an additional mechanism, competitive distribution of activation, underlies some inhibitory effects in cortex. Analysis of a mathematical model based on this hypothesis predicts that per stimulus inhibitory effects can be caused by competitive distribution of activation, and computer simulations confirm these predictions by demonstrating Mexican Hat patterns of lateral interactions, transformation of initially diffuse activity patterns into tightly focused "islands" of activation, and edge enhancement. The amount of inhibition can be adjusted by varying the intensity of the underlying competitive process. The concept of competitive distribution of activation provides an important perspective for interpreting neocortical and thalamocortical circuitry and can serve as a guide for further morphological and physiological studies. For example, it provides an explanation for the existence of recurrent cortex-to-thalamus connections that perform a logical AND-operation, and predicts the existence of analogous neocortical circuitry.
Generation of Direction Selectivity by Isotropic Intracortical Connections
Authors: Wörgötter Florentin , Niebur Ernst , Koch Christof
Published: March 2008

To what extent do the mechanisms generating different receptive field properties of neurons depend on each other? We investigated this question theoretically within the context of orientation and direction tuning of simple cells in the mammalian visual cortex. In our model a cortical cell of the "simple" type receives its orientation tuning by afferent convergence of aligned receptive fields of the lateral geniculate nucleus (Hubel and Wiesel 1962). We sharpen this orientation bias by postulating a special type of radially symmetric long-range lateral inhibition called circular inhibition. Surprisingly, this isotropic mechanism leads to the emergence of a strong bias for the direction of motion of a bar. We show that this directional anisotropy is neither caused by the probabilistic nature of the connections nor is it a consequence of the specific columnar structure chosen but that it is an inherent feature of the architecture of visual cortex.
Binding Hierarchies: A Basis for Dynamic Perceptual Grouping
Authors: Lumer Erik D. , Huberman Bernardo A.
Published: March 2008

Since it has been suggested that the brain binds its fragmentary representations of perceptual events via phase-locking of stimulated neural oscillators, it is important to determine how extended synchronization can occur in a clustered organization of cells possessing a distribution of firing rates. To answer that question, we establish the basic conditions for the existence of a binding mechanism based on synchronized oscillations. In addition, we present a simple hierarchical architecture of feedback units that not only induces robust phase-locking within and segregation between perceptual groups, but also serves as a generic binding machine.
A Distributed Neural Network Architecture for Hexapod Robot Locomotion
Authors: Beer Randall D. , Chiel Hillel J. , Quinn Roger D. , Espenschied Kenneth S. , Larsson Patrik
Published: March 2008

We present fully distributed neural network architecture for controlling the locomotion of a hexapod robot. The design of this network is directly based on work on the neuroethology of insect locomotion. Previously, we demonstrated in simulation that this controller could generate a continuous range of statically stable insect-like gaits as the activity of a single command neuron was varied and that it was robust to a variety of lesions. We now report that the controller can be utilized to direct the locomotion of an actual six-legged robot, and that it exhibits a range of gaits and degree of robustness in the real world that is quite similar to that observed in simulation.
Computing the Karhunen-Loeve Expansion with a Parallel, Unsupervised Filter System
Authors: Lenz Reiner , Österberg Mats
Published: March 2008

We use the invariance principle and the principles of maximum information extraction and maximum signal concentration to design a parallel, linear filter system that learns the Karhunen-Loeve expansion of a process from examples. In this paper we prove that the learning rule based on these principles forces the system into stable states that are pure eigenfunctions of the input process.
Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks
Authors: Giles C. L. , Miller C. B. , Chen D. , Chen H. H. , Sun G. Z. , Lee Y. C.
Published: March 2008

We show that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples. We present simulations that show the effect of initial conditions, training set size and order, and neural network architecture. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. We discuss a quantization algorithm for dynamically extracting finite state automata during and after training. For a well-trained neural net, the extracted automata constitute an equivalence class of state machines that are reducible to the minimal machine of the inferred grammar. We then show through simulations that many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. In addition, some of these extracted automata actually outperform the trained neural network for classification of unseen strings.
Induction of Finite-State Languages Using Second-Order Recurrent Networks
Authors: Watrous Raymond L. , Kuhn Gary M.
Published: March 2008

Second-order recurrent networks that recognize simple finite state languages over {0,1}* are induced from positive and negative examples. Using the complete gradient of the recurrent network and sufficient training examples to constrain the definition of the language to be induced, solutions are obtained that correctly recognize strings of arbitrary length.
Bayesian Interpolation
Authors: MacKay David J. C.
Published: March 2008

Although Bayesian analysis has been in use since Laplace, the Bayesian method of model-comparison has only recently been developed in depth. In this paper, the Bayesian approach to regularization and model-comparison is demonstrated by studying the inference problem of interpolating noisy data. The concepts and methods described are quite general and can be applied to many other data modeling problems. Regularizing constants are set by examining their posterior probability distribution. Alternative regularizers (priors) and alternative basis sets are objectively compared by evaluating the evidence for them. “Occam's razor” is automatically embodied by this process. The way in which Bayes infers the values of regularizing constants and noise levels has an elegant interpretation in terms of the effective number of parameters determined by the data set. This framework is due to Gull and Skilling.
A Practical Bayesian Framework for Backpropagation Networks
Authors: MacKay David J. C.
Published: March 2008

A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network pruning or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on network parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian "evidence" automatically embodies "Occam's razor," penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evidence is obtained.
Activity Patterns of a Slow Synapse Network Predicted by Explicitly Averaging Spike Dynamics
Authors: Rinzel John , Frankel Paul
Published: March 2008

When postsynaptic conductance varies slowly compared to the spike generation process, a straightforward averaging scheme can be used to reduce the system's complexity. Our model consists of a Hodgkin-Huxley-like membrane description for each cell; synaptic activation is described by first order kinetics, with slow rates, in which the equilibrium activation is a sigmoidal function of the presynaptic voltage. Our work concentrates on a two-cell network and it applies qualitatively to the activity patterns, including bistable behavior, recently observed in simple in vitro circuits with slow synapses (Kleinfeld et al. 1990). The fact that our averaged system is derived from a realistic biophysical model has important consequences. In particular, it can preserve certain hysteresis behavior near threshold that is not represented in a simple ad hoc sigmoidal input-output network. This behavior enables a coupled pair of cells, one excitatory and one inhibitory, to generate an alternating burst rhythm even though neither cell has fatiguing properties.
Refining PID Controllers Using Neural Networks
Authors: Scott Gary M. , Shavlik Jude W. , Ray W. Harmon
Published: March 2008

The KBANN (Knowledge-Based Artificial Neural Networks) approach uses neural networks to refine knowledge that can be written in the form of simple propositional rules. We extend this idea further by presenting the MANNCON (Multivariable Artificial Neural Network Control) algorithm by which the mathematical equations governing a PID (Proportional-Integral-Derivative) controller determine the topology and initial weights of a network, which is further trained using backpropagation. We apply this method to the task of controlling the outflow and temperature of a water tank, producing statistically significant gains in accuracy over both a standard neural network approach and a nonlearning PID controller. Furthermore, using the PID knowledge to initialize the weights of the network produces statistically less variation in test set accuracy when compared to networks initialized with small random numbers.
Rule-Based Neural Networks for Classification and Probability Estimation
Authors: Goodman Rodney M. , Higgins Charles M. , Miller John W. , Smyth Padhraic
Published: March 2008

In this paper we propose a network architecture that combines a rule-based approach with that of the neural network paradigm. Our primary motivation for this is to ensure that the knowledge embodied in the network is explicitly encoded in the form of understandable rules. This enables the network's decision to be understood, and provides an audit trail of how that decision was arrived at. We utilize an information theoretic approach to learning a model of the domain knowledge from examples. This model takes the form of a set of probabilistic conjunctive rules between discrete input evidence variables and output class variables. These rules are then mapped onto the weights and nodes of a feedforward neural network resulting in a directly specified architecture. The network acts as parallel Bayesian classifier, but more importantly, can also output posterior probability estimates of the class variables. Empirical tests on a number of data sets show that the rule-based classifier performs comparably with standard neural network classifiers, while possessing unique advantages in terms of knowledge representation and probability estimation.
Complex Scheduling with Potts Neural Networks
Authors: Gislén Lars , Peterson Carsten , Söderberg Bo
Published: March 2008

In a recent paper (Gislén et al. 1989) a convenient encoding and an efficient mean field algorithm for solving scheduling problems using a Potts neural network was developed and numerically explored on simplified and synthetic problems. In this work the approach is extended to realistic applications both with respect to problem complexity and size. This extension requires among other things the interaction of Potts neurons with different number of components. We analyze the corresponding linearized mean field equations with respect to estimating the phase transition temperature. Also a brief comparison with the linear programming approach is given. Testbeds consisting of generated problems within the Swedish high school system are solved efficiently with high quality solutions as results.
Local Learning Algorithms
Authors: Bottou Léon , Vapnik Vladimir
Published: March 2008

Very rarely are training data evenly distributed in the input space. Local learning algorithms attempt to locally adjust the capacity of the training system to the properties of the training set in each area of the input space. The family of local learning algorithms contains known methods, like the k-nearest neighbors method (kNN) or the radial basis function networks (RBF), as well as new algorithms. A single analysis models some aspects of these algorithms. In particular, it suggests that neither kNN or RBF, nor nonlocal classifiers, achieve the best compromise between locality and capacity. A careful control of these parameters in a simple local learning algorithm has provided a performance breakthrough for an optical character recognition problem. Both the error rate and the rejection performance have been significantly improved.
A Model Circuit for Cortical Temporal Low-Pass Filtering
Authors: Maex R. , Orban G. A.
Published: March 2008

We propose that the low-pass characteristics in the temporal and velocity domain of area 17 cells are generated by the abundant excitatory connections between cortical neurons. We have incorporated this anatomical feature in a model circuit in which simple cells' firing is initiated by geniculocortical excitatory synaptic input, with a short time course, and firing is maintained by feedback corticocortical excitatory synapses, which have a longer time course. The low-pass performance of the model is demonstrated by computing the model simple cells' velocity response curves (VRC) elicited by moving bars, and comparing these to those of its LGN (lateral geniculate nucleus) inputs. For the same parameter set, the VRCs of sustained and transient LGN cells are transformed into VRCs typical of central area 17 and central area 18 cells, respectively.
A Study on Fabrication, Morphological and Optical Properties of Lead Sulfide Nanocrystals
Authors: Chen Jung-Hsuan , Lo Shen-Chuan , Chao Chuen-Guang , Liu Tzeng-Feng
Published: March 2008

To investigate the properties of lead sulfide (PbS) nanocrystals, we have prepared PbS nanocrystals on/in the porous alumina membrane with a pore diameter 20 nm. Utilizing the reaction of Pb wires and the hydrogen sulfide (H2S) gas, PbS nanocrystals produced and grew as the
reaction time increased. The composition identification of the nanocrystals was performed by the XPS and EDS analyses. More structure characteristics of the PbS nanocrystals obtained from the TEM analysis. As indicated in the PL spectra, an orange-red emission band appeared and the emission
intensities were obviously related to the defects in the nanocrytals. A significant quantum confinement effect made the energy gap of PbS nanocrystals produce a blue shift from 0.41 eV to 1.89 eV. Furthermore, the growth mechanism of the PbS nanocrystals was also discussed.
Gas Sensing Properties of Columnar CeO2 Nanostructures Prepared by Chemical Vapor Deposition
Authors: Barreca Davide , Comini Elisabetta , Gasparotto Alberto , Maccato Chiara , Maragno Cinzia , Sberveglieri Giorgio , Tondello Eugenio
Published: March 2008

Columnar CeO2 nanostructures are grown on alumina substrates by a template- and catalyst-free Chemical Vapor Deposition (CVD) approach and subsequently tested as resistive gas sensors of CH3COCH3, H2, NO2. The sensor response is
stable and reproducible throughout the whole working temperature range (200–500 °C) and directly dependent on the analyte gas and the adopted operating conditions. The higher sensitivity with respect to that displayed by continuous CeO2 thin films demonstrates the potential
of fabricating nanostructured sensing devices characterized by improved functional performances.