optimal channel efficiency in a sensory network spontaneous neural activity has been increasingly recognized as a subject of key relevance in neuroscience it exhibits nontrivial spatiotemporal structure reflecting the organization of the underlying neural network and has proved to be closely intertwined with stimulus induced activity patterns as an additional contribution in this regard we report computational studies that strongly suggest that a stimulus free feature rules the behavior of an important psychophysical measure of the sensibility of a sensory system to a stimulus the so called dynamic range indeed in this paper we show that the entropy of the distribution of avalanche lifetimes information efficiency since it can be interpreted as the efficiency of the network seen as a communication channel always accompanies the dynamic range in the benchmark model for sensory systems specifically by simulating the kinouchi copelli kc model on two broad families of model networks we generically observed that both quantities always increase or decrease together as functions of the average branching ratio the control parameter of the kc model and that the information efficiency typically exhibits critical optimization jointly with the dynamic range i e both quantities are optimized at the same value of that control parameter that turns out to be the critical point of a nonequilibrium phase transition in contrast with the practice of taking power laws to identify critical points in most studies describing measured neuronal avalanches we rely on data collapses as more robust signatures of criticality to claim that critical optimization may happen even when the distribution of avalanche lifetimes is not a power law as suggested by a recent experiment finally we note that the entropy of the size distribution of avalanches information capacity does not always follow the dynamic range and the information efficiency when they are critically optimized despite being more widely used than the latter to describe the computational capabilities of a neural network this strongly suggests that dynamical rules allowing a proper temporal matching of the states of the interacting neurons is the key for achieving good performance in information processing rather than increasing the number of available units

statistical mechanics and phase transitions in clustering a new approach to clustering based on statistical physics is presented the problem is formulated as fuzzy clustering and the association probability distribution is obtained by maximizing the entropy at a given average variance the corresponding lagrange multiplier is related to the temperature and motivates a deterministic annealing process where the free energy is minimized at each temperature critical temperatures are derived for phase transitions when existing clusters split it is a hierarchical clustering estimating the most probable cluster parameters at various average variances

excitable greenberg hastings cellular automaton model on scale free networks we study the excitable greenberg hastings cellular automaton model on scale free networks we obtain analytical expressions for no external stimulus the uncoupled case it is found that the curves the average activity f versus the external stimulus rate r can be fitted by a hill function but not exactly there exists a relation f ensuremath sim r ensuremath alpha for the low stimulus response where the stevens hill exponent ensuremath alpha ranges from ensuremath alpha 1 in the subcritical regime to ensuremath alpha 0 5 at criticality at the critical point the range is maximal but not divergent we also calculate the average activity f k r and the dynamic range ensuremath delta k p for nodes with given connectivity k it is interesting that nodes with larger connectivity have larger optimal range which could be applied in biological experiments to reveal the network topology

deterministic excitable media under poisson drive power law responses spiral waves and dynamic range when each site of a spatially extended excitable medium is independently driven by a poisson stimulus with rate h the interplay between creation and annihilation of excitable waves leads to an average activity f it has recently been suggested that in the low stimulus regime h approximately 0 the response function f h of hypercubic deterministic systems behaves as a power law f approximately h m moreover the response exponent m has been predicted to depend only on the dimensionality d of the lattice m 1 1 d t ohta and t yoshimura physica d 205 189 2005 in order to test this prediction we study the response function of excitable lattices modeled by either coupled morris lecar equations or greenberg hastings cellular automata we show that the prediction is verified in our model systems for d 1 2 and 3 provided that a minimum set of conditions is satisfied under these conditions the dynamic range which measures the range of stimulus intensities that can be coded by the network activity increases with the dimensionality d of the network the power law scenario breaks down however if the system can exhibit self sustained activity spiral waves in this case we recover a scenario that is common to probabilistic excitable media as a function of the conductance coupling g among the excitable elements the dynamic range is maximized precisely at the critical value g c above which self sustained activity becomes stable we discuss the implications of these results in the context of neural coding

blind separation of a mixture of uniformly distributed source signals a novel approach a new efficient algorithm for blind separation of uniformly distributed sources is proposed the mixing matrix is assumed to be orthogonal by prewhitening the observed signals the learning rule adaptively estimates the mixing matrix by conceptually rotating a unit hypercube so that all output signal components are contained within or on the hypercube under some ideal constraints it has been theoretically shown that the algorithm is very similar to an ideal convergent algorithm which is much faster than the existing convergent algorithms the algorithm has been generalized to take care of the noisy signals by adaptively dilating the hypercube in conjunction with its rotation

faithful representation of separable distributions a geometric approach to data representation incorporating information theoretic ideas is presented the task of finding a faithful representation where the input distribution is evenly partitioned into regions of equal mass is addressed for input consisting of mixtures of statistically independent sources we treat independent component analysis ica as a computational geometry problem first we consider the separation of sources with sharply peaked distribution functions where the ica problem becomes that of finding high density directions in the input distribution second we consider the more general problem for arbitrary input distributions where ica is transformed into the task of finding an aligned equipartition by modifying the kohonen self organized feature maps we arrive at neural networks with local interactions that optimize coding while simultaneously performing source separation the local nature of our approach results in networks with nonlinear ica capabilities

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

response of electrically coupled spiking neurons a cellular automaton approach experimental data suggest that some classes of spiking neurons in the first layers of sensory systems are electrically coupled via gap junctions or ephaptic interactions when the electrical coupling is removed the response function firing rate vs stimulus intensity of the uncoupled neurons typically shows a decrease in dynamic range and sensitivity in order to assess the effect of electrical coupling in the sensory periphery we calculate the response to a poisson stimulus of a chain of excitable neurons modeled by n state greenberg hastings cellular automata in two approximation levels the single site mean field approximation is shown to give poor results failing to predict the absorbing state of the lattice while the results for the pair approximation are in good agreement with computer simulations in the whole stimulus range in particular the dynamic range is substantially enlarged due to the propagation of excitable waves which suggests a functional role for lateral electrical coupling for probabilistic spike propagation the hill exponent of the response function is alpha 1 while for deterministic spike propagation we obtain alpha 1 2 which is close to the experimental values of the psychophysical stevens exponents for odor and light intensities our calculations are in qualitative agreement with experimental response functions of ganglion cells in the mammalian retina

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

adaptive online learning algorithms for blind separation maximum entropy and minimum mutual information there are two major approaches for blind separation maximum entropy me and minimum mutual information mmi both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix the mi is the contrast function for blind separation the entropy is not to justify the me the relation between me and mmi is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the me and at the solution points determined by the mi the me will not update the demixing matrix in the directions of increasing the cross talking second the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms because the parameter space is a riemannian space consisting of matrices the mutual information is calculated by applying the gram charlier expansion to approximate probability density functions of the outputs finally we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants it is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants

dynamic range of hypercubic stochastic excitable media we study the response properties of d dimensional hypercubic excitable networks to a stochastic stimulus each site modeled either by a three state stochastic susceptible infected recovered susceptible system or by the probabilistic greenberg hastings cellular automaton is continuously and independently stimulated by an external poisson rate h the response function mean density of active sites ensuremath rho versus h is obtained via simulations for d 1 2 3 4 and mean field approximations at the single site and pair levels ensuremath forall text d in any dimension the dynamic range and sensitivity of the response function are maximized precisely at the nonequilibrium phase transition to self sustained activity in agreement with a reasoning recently proposed moreover the maximum dynamic range attained at a given dimension d is a decreasing function of d

independent component analysis using potts models we explore the extending application of potts encoding to the task of independent component analysis which primarily deals with the problem of minimizing the kullback leibler divergence between the joint distribution and the product of all marginal distributions of output components the competitive mechanism of potts neurons is used to encode the overlapping projections from observations to output components based on these projections the marginal distributions and the entropy of output components are made tractable for computation and the adaptation of the de mixing matrix toward independent output components is obtained the potts model for ica is well formulated by an objective function subject to a set of constraints which leads to a novel energy function a hybrid of the mean field annealing and the gradient descent method is applied to the energy function our approach to independent component analysis presents a new criterion for ica the performance of the potts model for ica given by our numerical simulations is encouraging

a deterministic annealing approach for parsimonious design of piecewise regression models a new learning algorithm is proposed for piecewise regression modeling it employs the technique of deterministic annealing to design space partition regression functions while the performance of traditional space partition regression functions such as cart and mars is limited by a simple tree structured partition and by a hierarchical approach for design the deterministic annealing algorithm enables the joint optimization of a more powerful piecewise structure based on a voronoi partition the new method is demonstrated to achieve consistent performance improvements over regular cart as well as over its extension to allow arbitrary hyperplane boundaries comparison tests on several benchmark data sets from the regression literature are provided

fast and robust fixed point algorithms for independent component analysis independent component analysis ica is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible we use a combination of two different approaches for linear ica comon s information theoretic approach and the projection pursuit approach using maximum entropy approximations of differential entropy we introduce a family of new contrast functions for ica these contrast functions enable both the estimation of the whole decomposition by minimizing mutual information and estimation of individual independent components as projection pursuit directions the statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model and it is shown how to choose contrast functions that are robust and or of minimum variance finally we introduce simple fixed point algorithms for practical optimization of the contrast functions

variational and generalized variational inequalities with discontinuous mappings abstract in this paper we derive several existence results for the variational and the generalized variational inequality problems where the mappings under consideration need not have any continuity properties uniqueness of the solution of the generalized variational inequality problem is also investigated as a by product we obtain several existence and uniqueness results for the generalized complementarity problem applications to minimization problems with generalized convex functions are considered

epidemic dynamics and endemic states in complex networks we study by analytical methods and large scale simulations a dynamical model for the spreading of epidemics in complex networks in networks with exponentially bounded connectivity we recover the usual epidemic behavior with a threshold defining a critical point below that the infection prevalence is null on the contrary on a wide range of scale free networks we observe the absence of an epidemic threshold and its associated critical behavior this implies that scale free networks are prone to the spreading and the persistence of infections whatever spreading rate the epidemic agents might possess these results can help understanding computer virus epidemics and other spreading phenomena on communication and social networks

intensity coding in two dimensional excitable neural networks abstract in the light of recent experimental findings that gap junctions are essential for low level intensity detection in the sensory periphery the greenberg hastings cellular automaton is employed to model the response of a two dimensional sensory network to external stimuli we show that excitable elements sensory neurons that have a small dynamical range are shown to give rise to a collective large dynamical range therefore the network transfer gain function which is hill or stevens law like is an emergent property generated from a pool of small dynamical range cells providing a basis for a neural psychophysics the growth of the dynamical range with the system size is approximately logarithmic suggesting a functional role for electrical coupling for a fixed number of neurons the dynamical range displays a maximum as a function of the refractory period which suggests experimental tests for the model a biological application to ephaptic interactions in olfactory nerve fascicles is proposed

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

the influence of connectivity on the firing rate in a neuronal network with electrical and chemical synapses a b s t r a c t we study the firing rate properties of a cellular automaton model for a neuronal network with chemical synapses we propose a simple mechanism in which the nonlocal connections are included through electrical and chemical synapses in the latter case we introduce a time delay which produces self sustained activity nonlocal connections or shortcuts are randomly introduced according to a specified connection probability there is a range of connection probabilities for which neuron firing occurs as well as a critical probability for which the firing ceases in the absence of time delay the critical probability for nonlocal shortcuts depends on the network size according to a power law we also compute the firing rate amplification factor by varying both the connection probability and the time delay for different network sizes

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

mean field theory for scale free random networks random networks with complex topology are common in nature describing systems as diverse as the world wide web or social and business networks recently it has been demonstrated that most large networks for which topological information is available display scale free features here we study the scaling properties of the recently introduced scale free model that can account for the observed power law distribution of the connectivities we develop a mean eld method to predict the growth dynamics of the individual vertices and use this to calculate analytically the connectivity distribution and the scaling exponents the mean eld method can be used to address the properties of two variants of the scale free model that do not display power law scaling c 1999 elsevier science b v all rights reserved