measuring the strangeness of strange attractors we study the correlation exponent v introduced recently as a characteristic measure of strange attractors which allows one to distinguish between deterministic chaos and random noise the exponent v is closely related to the fractal dimension and the information dimension but its computation is considerably easier its usefulness in characterizing experimental data which stem from very high dimensional systems is stressed algorithms for extracting v from the time series of a single variable are proposed the relations between the various measures of strange attractors and between them and the lyapunov exponents are discussed it is shown that the conjecture of kaplan and yorke for the dimension gives an upper bound for v various examples of finite and infinite dimensional systems are treated both numerically and analytically

signal tuned gabor functions as models for stimulus dependent cortical receptive fields we propose and analyze a model based on signal tuned gabor functions for the receptive fields and responses of v1 cells signal tuned gabor functions are gaussian modulated sinusoids whose parameters are obtained from a given spatial or spectral tuning signal these functions can be proven to yield exact representations of their tuning signals and have recently been proposed as the kernels of a variant gabor transform the signal tuned gabor transform stgt which allows the accurate detection of spatial and spectral events here we show that by modeling the receptive fields of simple and complex cells as signal tuned gabor functions and expressing their responses as stgts we are able to replicate the properties of these cells when tested with standard grating and slit inputs at the same time emulating their stimulus dependent character as revealed by recent neurophysiological studies

nonlinear dynamics in driven damped sine gordon systems abstract many quasi one and two dimensional systems can be modeled by sine gordon equations illustrations are given for driven and damped sine gordon equations of behaviors that are typical of ways that complexity in time and space can be manifested examples include an ac response to a dc driver and spontaneous pattern formation and low dimensional chaos under the influence of ac or ac dc drivers the importance of the coexistence of coherence and chaos for transport phenomena is emphasized

uniformity of the phase space and fluctuations in thermal equilibrium abstract general relations are found between the measure of the uniformity of distributions on the phase space and the first moments and correlations of extensive variables for systems close to thermal equilibrium the role played by the parameter of the renyi entropy for the analysis of their fluctuations and correlations is studied analytical results are verified and illustrated by direct simulations of quantum systems of ideal fermions and bosons problems of finite statistics usual in experiments and simulations are addressed and discussed and solved by finding unbiased estimators for renyi entropies and uniformities

instantaneous frequency decomposition an application to spectrally sparse sounds with fast frequency modulations classical time frequency analysis is based on the amplitude responses of bandpass filters discarding phase information instantaneous frequency analysis in contrast is based on the derivatives of these phases this method of frequency calculation is of interest for its high precision and for reasons of similarity to cochlear encoding of sound this article describes a methodology for high resolution analysis of sparse sounds based on instantaneous frequencies in this method a comparison between tonotopic and instantaneous frequency information is introduced to select filter positions that are well matched to the signal second a cross check that compares frequency estimates from neighboring channels is used to optimize filter bandwidth and to signal the quality of the analysis these cross checks lead to an optimal time frequency representation without requiring any prior information about the signal when applied to a signal that is sufficiently sparse the method decomposes the signal into separate time frequency contours that are tracked with high precision alternatively if the signal is spectrally too dense neighboring channels generate inconsistent estimates a feature that allows the method to assess its own validity in particular contexts similar optimization principles may be present in cochlear encoding

characterization of minimum error linear coding with sensory and neural noise robust coding has been proposed as a solution to the problem of minimizing decoding error in the presence of neural noise many real world problems however have degradation in the input signal not just in neural representations this generalized problem is more relevant to biological sensory coding where internal noise arises from limited neural precision and external noise from distortion of sensory signal such as blurring and phototransduction noise in this note we show that the optimal linear encoder for this problem can be decomposed exactly into two serial processes that can be optimized separately one is wiener filtering which optimally compensates for input degradation the other is robust coding which best uses the available representational capacity for signal transmission with a noisy population of linear neurons we also present spectral analysis of the decomposition that characterizes how the reconstruction error is minimized under different input signal spectra types and amounts of degradation degrees of neural precision and neural population sizes

estimating dimension from small samples abstract details of a new algorithm to estimate the dimension of a measure from point samples are described the new algorithm emphasizes that dimension need not be the same at all scales that is there need not be a scaling region the algorithm is verified with gaussian measures and is found to be reliable and have less demanding data requirements than conventional estimators using certain convergence properties of gaussian measures it is shown that good estimates can be made from very small samples

dynamical complexity detection in short term physiological series using base scale entropy physiological systems generate complex fluctuations in their output signals that reflect the underlying dynamics the base scale entropy method was proposed as a complexity measure to investigate the complexity of time series the advantages of this method are simplicity and extremely fast calculation for very short data sets this method enables analyzing very short nonstationary and noisy data sets we employed this method for short term physiological time series for analysis of heart rate variability signals the results show that the simple and easily calculated measure can effectively detect the complexity dissimilarity of physiological time series in different physiological or pathological states which is convenient for clinical applications

learning chaotic attractors by neural networks an algorithm is introduced that trains a neural network to identify chaotic dynamics from a single measured time series during training the algorithm learns to short term predict the time series at the same time a criterion developed by diks van zwet takens and de goede 1996 is monitored that tests the hypothesis that the reconstructed attractors of model generated and measured data are the same training is stopped when the prediction error is low and the model passes this test two other features of the algorithm are 1 the way the state of the system consisting of delays from the time series has its dimension reduced by weighted principal component analysis data reduction and 2 the user adjustable prediction horizon obtained by error propagation partially propagating prediction errors to the next time step the algorithm is first applied to data from an experimental driven chaotic pendulum of which two of the three state variables are known this is a comprehensive example that shows how well the diks test can distinguish between slightly different attractors second the algorithm is applied to the same problem but now one of the two known state variables is ignored finally we present a model for the laser data from the santa fe time series competition set a it is the first model for these data that is not only useful for short term predictions but also generates time series with similar chaotic characteristics as the measured data

dependency reduction with divisive normalization justification and effectiveness efficient coding transforms that reduce or remove statistical dependencies in natural sensory signals are important for both biology and engineering in recent years divisive normalization dn has been advocated as a simple and effective nonlinear efficient coding transform in this work we first elaborate on the theoretical justification for dn as an efficient coding transform specifically we use the multivariate t model to represent several important statistical properties of natural sensory signals and show that dn approximates the optimal transforms that eliminate statistical dependencies in the multivariate t model second we show that several forms of dn used in the literature are equivalent in their effects as efficient coding transforms third we provide a quantitative evaluation of the overall dependency reduction performance of dn for both the multivariate t models and natural sensory signals finally we find that statistical dependencies in the multivariate t model and natural sensory signals are increased by the dn transform with low input dimensions this implies that for dn to be an effective efficient coding transform it has to pool over a sufficiently large number of inputs

a technique for measuring fractal dimensions from time series on a real time scale abstract a dedicated electronic instrument for measuring fractal dimension of chaotic systems called dimensiometer is described the pointwise correlation dimension can be obtained from experimental time series on a real time scale this technique employs a discrete sequence of the time intervals between the successive intersections of a single variable x t and an arbitrary level c the dimensiometer is tested by means of a simple electronic chaos oscillator the experimental results are compared with the numerical calculations carried out for the corresponding dynamical model

the hilbert phenomenon in chaotic motions abstract it is proved that chaotic dynamical systems have d 1 singularity due to such singularity and its periodicity in hausdorff metric spaces a chaotic dynamical system induces a hilbert iterated system which was discovered at the end of the 19th century this implies that the most puzzling discovery in nature in the 20th century can be put into deep correspondence with that in thoughts in the 19th century all this is verified by the numerical experiments of chaotic vibrations of a beam and a tension slack oscillator

model of frequency analysis in the visual cortex and the shape from texture problem this paper addresses the question at the level of cortical cells present in the primary area v1 is the information sufficient to extract the local perspective from the texture starting from a model of complex cells in visual area v1 we propose a biologically plausible algorithm for frequency analysis applied to the shape from texture problem first specific log normal filters are designed in replacement of the classical gabor filters because of their theoretical properties and of their biological plausibility these filters are separable in frequency and orientation and they better sample the image spectrum which makes them appropriate for any pattern analysis technique a method to estimate the local frequency in the image which discards the need to choose the best local scale is designed based on this frequency analysis model a local decomposition of the image into patches leads to the estimation of the local frequency variation which is used to solve the problem of recovering the shape from the texture from the analytical relation between the local frequency and the geometrical parameters under perspective projection it is possible to recover the orientation and the shape of the original image the accuracy of the method is evaluated and discussed on different kind of textures both regular and irregular with planar and curved surfaces and also on natural scenes and psychophysical stimuli it compares favorably to the best existing methods with in addition a low computational cost the biological plausibility of the model is finally discussed

what is the goal of sensory coding a number of recent attempts have been made to describe early sensory coding in terms of a general information processing strategy in this paper two strategies are contrasted both strategies take advantage of the redundancy in the environment to produce more effective representations the first is described as a compact coding scheme a compact code performs a transform that allows the input to be represented with a reduced number of vectors cells with minimal rms error this approach has recently become popular in the neural network literature and is related to a process called principal components analysis pca a number of recent papers have suggested that the optimal compact code for representing natural scenes will have units with receptive field profiles much like those found in the retina and primary visual cortex however in this paper it is proposed that compact coding schemes are insufficient to account for the receptive field properties of cells in the mammalian visual pathway in contrast it is proposed that the visual system is near to optimal in representing natural scenes only if optimality is defined in terms of sparse distributed coding in a sparse distributed code all cells in the code have an equal response probability across the class of images but have a low response probability for any single image in such a code the dimensionality is not reduced rather the redundancy of the input is transformed into the redundancy of the firing pattern of cells it is proposed that the signature for a sparse code is found in the fourth moment of the response distribution i e the kurtosis in measurements with 55 calibrated natural scenes the kurtosis was found to peak when the bandwidths of the visual code matched those of cells in the mammalian visual cortex codes resembling wavelet transforms are proposed to be effective because the response histograms of such codes are sparse i e show high kurtosis when presented with natural scenes it is proposed that the structure of the image that allows sparse coding is found in the phase spectrum of the image it is suggested that natural scenes to a first approximation can be considered as a sum of self similar local functions the inverse of a wavelet possible reasons for why sensory systems would evolve toward sparse coding are presented

dimension measurement on high dimensional systems abstract using experimental data from a variety of laboratory electrical systems the upper limits on dimension measurement are examined and it is shown that contrary to standard scaling estimates it can be possible to reliably measure correlation dimensions larger than 10 the practical implications of the constraints on such a measurement are examined including the role of statistical errors geometrical artifacts dimension algorithm metric choice and the process power spectrum these ideas are applied to electrical 1 f noise from both macroscopic and microscopic sources to show that it cannot be explained by a deterministic model with fewer than roughly 20 degrees of freedom

relative commutator calculus in chevalley groups abstract we revisit localisation and patching method in the setting of chevalley groups introducing certain subgroups of relative elementary chevalley groups we develop relative versions of the conjugation calculus and the commutator calculus in chevalley groups g r rk 2 which are both more general and substantially easier than the ones available in the literature for classical groups such relative commutator calculus has been recently developed by the authors in hazrat zhang 2011 34 hazrat et al 2011 33 as an application we prove the mixed commutator formula e r a g r b e r a e r b for two ideals a b r this answers a problem posed in a paper by alexei stepanov and the second author

characterization of the spatial complex behavior and transition to chaos in flow systems abstract we introduce a spatial lyapunov exponent to characterize the complex behavior of non chaotic but convectively unstable flow sytems this complexity is of spatial type and is due to sensitivity to the boundary conditions we show that there exists a relation between the spatial complexity index we define and the comoving lyapunov exponents in such systems the transition to chaos i e the occurrence of a positive lyapunov exponent can manifest itself in two different ways in the first case from neither chaotic nor spatially complex behavior to chaos one observes the typical scenario i e as the system size grows up the spectrum of the lyapunov exponents gives rise to a density in the second case when the chaos develops from a convectively unstable situation one observes only a finite number of positive lyapunov exponents

information maximization and independent component analysis is there a difference this article provides a detailed and rigorous analysis of the two commonly used methods for redundancy reduction linear independent component analysis ica posed as a direct minimization of a suitably chosen redundancy measure and information maximization infomax of a continuous stochastic signal transmitted through an appropriate nonlinear network the article shows analytically that ica based on the kullbackleibler information as a redundancy measure and infomax lead to the same solution if the parameterization of the output nonlinear functions in the latter method is sufficiently rich furthermore this work discusses the alternative redundancy measures not based on the kullback leibler information distance the practical issues of applying ica and infomax are also discussed and illustrated on the problem of extracting statistically independent factors from a linear pixel by pixel mixture of images

faithful representation of separable distributions a geometric approach to data representation incorporating information theoretic ideas is presented the task of finding a faithful representation where the input distribution is evenly partitioned into regions of equal mass is addressed for input consisting of mixtures of statistically independent sources we treat independent component analysis ica as a computational geometry problem first we consider the separation of sources with sharply peaked distribution functions where the ica problem becomes that of finding high density directions in the input distribution second we consider the more general problem for arbitrary input distributions where ica is transformed into the task of finding an aligned equipartition by modifying the kohonen self organized feature maps we arrive at neural networks with local interactions that optimize coding while simultaneously performing source separation the local nature of our approach results in networks with nonlinear ica capabilities

supervised factorial learning factorial learning finding a statistically independent representation of a sensory image a factorial code is applied here to solve multilayer supervised learning problems that have traditionally required backpropagation this lends support to barlow s argument for factorial sensory processing by demonstrating how it can solve actual pattern recognition problems two techniques for supervised factorial learning are explored one of which gives a novel distributed solution requiring only positive examples also a new nonlinear technique for factorial learning is introduced that uses neural networks based on almost reversible cellular automata due to the special functional connectivity of these networks which resemble some biological microcircuits learning requires only simple local algorithms also supervised factorial learning is shown to be a viable alternative to backpropagation one significant advantage is the existence of a measure for the performance of intermediate learning stages

do simple cells in primary visual cortex form a tight frame sets of neuronal tuning curves which describe the responses of neurons as functions of a stimulus can serve as a basis for approximating other functions of stimulus parameters in a function approximating network synaptic weights determined by a correlation based hebbian rule are closely related to the coefficients that result when a function is expanded in an orthogonal basis although neuronal tuning curves typically are not orthogonal functions the relationship between function approximation and correlation based synaptic weights can be retained if the tuning curves satisfy the conditions of a tight frame we examine whether the spatial receptive fields of simple cells in cat and monkey primary visual cortex v1 form a tight frame allowing them to serve as a basis for constructing more complicated extrastriate receptive fields using correlationbased synaptic weights our calculations show that the set of v1 simple cell receptive fields is not tight enough to account for the acuity observed psychophysically

the infinite number of generalized dimensions of fractals and strange attractors abstract we show that fractals in general and strange attractors in particular are characterized by an infinite number of generalized dimensions dq q 0 to this aim we develop a rescaling transformation group which yields analytic expressions for all the quantities dq we prove that lim q 0 dq fractal dimension d limq 1dq information dimension and dq 2 correlation exponent v dq with other integer q s correspond to exponents associated with ternary quaternary and higher correlation functions we prove that generally dq dq for any q q for homogeneous fractals dq dq a particularly interesting dimension is dq for two examples feigenbaum attractor generalized baker s transformation we calculate the generalized dimensions and find that d is a non trivial number all the other generalized dimensions are bounded between the fractal dimension and d

learning time series evolution by unsupervised extraction of correlations as a consequence we are able to model chaotic and nonchaotic time series furthermore one critical point in modeling time series is the determination of the dimension of the embedding vector used i e the number of components of the past that are needed to predict the future with this method we can detect the embedding dimension by extracting the influence of the past on the future i e the correlation of remote past and future optimal embedding dimensions are obtained for the henon map and the mackey glass series when noisy data corrupted by colored noise are used a model is still possible the noise will then be decorrelated by the network in the case of modeling a chemical reaction the most natural architecture that conserves the volume is a symplectic network which describes a system that conserves the entropy and therefore the transmitted information

stochastic resonance in the mutual information between input and output spike trains of noisy central neurons abstract within the framework of information theory we investigate the phenomenon of stochastic resonance in a realistic stochastic model of central neurons the essential point is that for central neurons the input is not a continuous signal but a train of spikes which has in general a non periodic distribution the behavior of an individual neuron is modeled by a combination of an ornstein uhlenbeck diffusion process and a jump process corresponding to a discrete input spike train the input spike train is assumed to be given by a poisson distribution if the parameters are such that the input spike train provokes a membrane potential that hovers under the threshold the noise given by the diffusive part of the neuron model induces a maximum in the mutual information between the input and output spike trains thus the stochastic resonance effect results in a strategy for efficient information transmission of subthreshold signals

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

redundancy reduction and independent component analysis conditions on cumulants and adaptive approaches in the context of both sensory coding and signal processing building factorized codes has been shown to be an efficient strategy in a wide variety of situations the signal to be processed is a linear mixture of statistically independent sources building a factorized code is then equivalent to performing blind source separation thanks to the linear structure of the data this can be done in the language of signal processing by finding an appropriate linear filter or equivalently in the language of neural modeling by using a simple feedforward neural network in this article we discuss several aspects of the source separation problem we give simple conditions on the network output that if satisfied guarantee that source separation has been obtained then we study adaptive approaches in particular those based on redundancy reduction and maximization of mutual information we show how the resulting updating rules are related to the bcm theory of synaptic plasticity eventually we briefly discuss extensions to the case of nonlinear mixtures through out this article we take care to put into perspective our work with other studies on source separation and redundancy reduction in particular we review algebraic solutions pointing out their simplicity but also their drawbacks

statistical edge detection learning and evaluating edge cues we formulate edge detection as statistical inference this statistical edge detection is data driven unlike standard methods for edge detection which are model based for any set of edge detection filters implementing local edge cues we use presegmented images to learn the probability distributions of filter responses conditioned on whether they are evaluated on or off an edge edge detection is formulated as a discrimination task specified by a likelihood ratio test on the filter responses this approach emphasizes the necessity of modeling the image background the off edges we represent the conditional probability distributions nonparametrically and illustrate them on two different data sets of 100 sowerby and 50 south florida images multiple edges cues including chrominance and multiple scale are combined by using their joint distributions hence this cue combination is optimal in the statistical sense we evaluate the effectiveness of different visual cues using the chernoff information and receiver operator characteristic roc curves this shows that our approach gives quantitatively better results than the canny edge detector when the image background contains significant clutter in addition it enables us to determine the effectiveness of different edge cues and gives quantitative measures for the advantages of multilevel processing for the use of chrominance and for the relative effectiveness of different detectors furthermore we show that we can learn these conditional distributions on one data set and adapt them to the other with only slight degradation of performance without knowing the ground truth on the second data set this shows that our results are not purely domain specific we apply the same approach to the spatial grouping of edge cues and obtain analogies to nonmaximal suppression and hysteresis

chaos in a variable mass relaxation oscillator model for the leaky tap abstract a variable mass relaxation oscillator model for the behaviour of the leaky tap is numerically investigated different regions of its three dimensional parameter space have been searched and interesting dynamical behaviour has been found the dynamics of the system is thoroughly characterized using projections of phase space trajectories 2 and 3 dimensional time delay plots power spectra lyapunov exponents hausdorff and correlation dimensions and bifurcation diagrams dynamical variables obtained from the model like drop masses velocities and drip intervals are used for reconstructing attractors we found several periodic intermittent and chaotic attractors we conclude that the rich dynamics of the model investigated represents an improvement in describing the experimentally observed behaviour of the actual dripping faucet system

coherence and chaos in the driven damped sine gordon equation measurement of the soliton spectrum abstract a numerical procedure is developed which measures the sine gordon soliton and radiation content of any field t which is periodic in space the procedure is applied to the field generated by a damped driven sine gordon equation this field can be either temporally periodic locked to the driver or chaotic in either case the numerical measurement shows that the spatial structure can be described by only a few spatially localized soliton wave train modes the numerical procedure quantitatively identifies the presence number and properties of these soliton wave trains for example an increase of spatial symmetry is accompanied by the injection of additional solitons into the field

subpixel registration with gradient correlation we address the problem of subpixel registration of images assumed to be related by a pure translation we present a method which extends gradient correlation to achieve subpixel accuracy our scheme is based on modeling the dominant singular vectors of the 2 d gradient correlation matrix with a generic kernel which we derive by studying the structure of gradient correlation assuming natural image statistics our kernel has a parametric form which offers flexibility in modeling the functions obtained from various types of image data we estimate the kernel parameters including the unknown subpixel shifts using the levenberg marquardt algorithm experiments with landsat and mri data show that our scheme outperforms recently proposed state of the art phase correlation methods

what are textons textons refer to fundamental micro structures in natural images and videos and are considered as the atoms of pre attentive human visual perception julesz 1981 unfortunately the word texton remains a vague concept in the literature for lack of a good mathematical model in this article we first present a three level generative image model for learning textons from texture images in this model an image is a superposition of a number of image bases selected from an over complete dictionary including various gabor and laplacian of gaussian functions at various locations scales and orientations these image bases are in turn generated by a smaller number of texton elements selected from a dictionary of textons by analogy to the waveform phoneme word hierarchy in speech the pixel base texton hierarchy presents an increasingly abstract visual description and leads to dimension reduction and variable decoupling by fitting the generative model to observed images we can learn the texton dictionary as parameters of the generative model then the paper proceeds to study the geometric dynamic and photometric structures of the texton representation by further extending the generative model to account for motion and illumination variations 1 for the geometric structures a texton consists of a number of image bases with deformable spatial configurations the geometric structures are learned from static texture images 2 for the dynamic structures the motion of a texton is characterized by a markov chain model in time which sometimes can switch geometric configurations during the movement we call the moving textons as motons the dynamic models are learned using the trajectories of the textons inferred from video sequence 3 for photometric structures a texton represents the set of images of a 3d surface element under varying illuminations and is called a lighton in this paper we adopt an illumination cone representation where a lighton is a texton triplet for a given light source a lighton image is generated as a linear sum of the three texton bases we present a sequence of experiments for learning the geometric dynamic and photometric structures from images and videos and we also present some comparison studies with k mean clustering sparse coding independent component analysis and transformed component analysis we shall discuss how general textons can be learned from generic natural images

predicting chaotic time series with wavelet networks a new technique wavelet network is introduced to predict chaotic time series by using this technique firstly we make accurate short term predictions of the time series from chaotic attractors secondly we make accurate predictions of the values and bifurcation structures of the time series from dynamical systems whose parameter values are changing with time finally we predict chaotic attractors by making long term predictions based on remarkably few data points where the correlation dimensions of predicted attractors are calculated and are found to be almost identical to those of actual attractors

redundancy reduction as a strategy for unsupervised learning a redundancy reduction strategy which can be applied in stages is proposed as a way to learn as efficiently as possible the statistical properties of an ensemble of sensory messages the method works best for inputs consisting of strongly correlated groups that is features with weaker statistical dependence between different features this is the case for localized objects in an image or for words in a text a local feature measure determining how much a single feature reduces the total redundancy is derived which turns out to depend only on the probability of the feature and of its components but not on the statistical properties of any other features the locality of this measure makes it ideal as the basis for a neural implementation of redundancy reduction and an example of a very simple non hebbian algorithm is given the effect of noise on learning redundancy is also discussed

relative unitary commutator calculus and applications abstract this note revisits localisation and patching method in the setting of generalised unitary groups introducing certain subgroups of relative elementary unitary groups we develop relative versions of the conjugation calculus and the commutator calculus in unitary groups which are both more general and substantially easier than the ones available in the literature for the general linear group such relative commutator calculus has been recently developed by the first and the third authors as an application we prove the mixed commutator formula eu 2 n i gu 2 n j eu 2 n i eu 2 n j for two form ideals i and j of a form ring a this answers two problems posed in a paper by alexei stepanov and the second author

nonlinear independent component analysis and multivariate time series analysis abstract we derive an information theory based unsupervised learning paradigm for nonlinear independent component analysis nica with neural networks we demonstrate that under the constraint of bounded and invertible output transfer functions the two main goals of unsupervised learning redundancy reduction and maximization of the transmitted information between input and output infomax principle are equivalent no assumptions are made concerning the kind of input and output distributions i e the kind of nonlinearity of correlations an adapted version of the general nica network is used for the modeling of multivariate time series by unsupervised learning given time series of various observables of a dynamical system our net learns their evolution in time by extracting statistical dependencies between past and present elements of the time series multivariate modeling is obtained by making present value of each time series statistically independent not only from their own past but also from the past of the other series therefore in contrast to univariate methods the information lying in the couplings between the observables is also used and a detection of higher order cross correlations is possible we apply our method to time series of the two dimensional henon map and to experimental time series obtained from the measurements of axial velocities in different locations in weakly turbulent taylor couette flow

using neural nets to look for chaos abstract backpropagating neural networks are used to reconstruct the attractors of two low dimensional chaotic systems using small input sets of noise corrupted data the nets are able to reconstruct attractors that are visually similar to and have the same correlation dimensions as attractors constructed from noise free data

estimation of time delay and lag relationship of quadratic systems abstract this paper is concerned with the identification of a quadratic non linear system the identification includes the estimation of the lag delay relationship of the input process the lead lag time delay of the input output system and the unknown parameters the estimation procedures are based on the second order covariances cross covariances and spectral densities the estimation methods described here are illustrated with numerical examples

a recursive on line identification of bilinear systems a recursive identification of a discrete dynamical system is presented in this paper two methods of recursive identification namely the recursive extended least squares method and the recursive maximum likelihood method are described and analysed the techniques are illustrated numerically with simulations

extracting qualitative dynamics from experimental data abstract we consider the notion of qualitative information and the practicalities of extracting it from experimental data our approach based on a theorem of takens draws on ideas from the generalized theory of information known as singular system analysis due to bertero pike and co workers we illustrate our technique with numerical data from the chaotic regime of the lorenz model

the mutual information of a stochastic binary channel validity of the replica symmetry ansatz we calculate the mutual information mi of a two layered neural network with noiseless continuous inputs and binary stochastic outputs under several assumptions on the synaptic efficiencies the interesting regime corresponds to the limit where the number of both input and output units is large but their ratio is kept fixed at a value we first present a solution for the mi using the replica technique with a replica symmetric rs ansatz then we find an exact solution for this quantity valid in a neighbourhood of an analysis of this solution shows that the system must have a phase transition at some finite value of this transition shows a singularity in the third derivative of the mi as the rs solution turns out to be infinitely differentiable it could be regarded as a smooth approximation to the mi this is checked numerically in the validity domain of the exact solution

on the exit times approach for e entropy and turbulent signals abstract we review a recently proposed approach to the computation of the e entropy of a given signal based on the exit time statistics i e one codes the signal by looking at the instants when the fluctuations are larger than a given threshold e moreover we show how the exit times statistics when applied to experimental turbulent data is able to highlight the intermediate dissipative range of turbulent fluctuations

towards a theory of early visual processing we propose a theory of the early processing in the mammalian visual pathway the theory is formulated in the language of information theory and hypothesizes that the goal of this processing is to recode in order to reduce a generalized redundancy subject to a constraint that specifies the amount of average information preserved in the limit of no noise this theory becomes equivalent to barlow s redundancy reduction hypothesis but it leads to very different computational strategies when noise is present a tractable approach for finding the optimal encoding is to solve the problem in successive stages where at each stage the optimization is performed within a restricted class of transfer functions we explicitly find the solution for the class of encodings to which the parvocellular retinal processing belongs namely linear and nondivergent transformations the solution shows agreement with the experimentally observed transfer functions at all levels of signal to noise

modeling the shape of the scene a holistic representation of the spatial envelope in this paper we propose a computational model of the recognition of real world scenes that bypasses the segmentation and the processing of individual objects or regions the procedure is based on a very low dimensional representation of the scene that we term the spatial envelope we propose a set of perceptual dimensions naturalness openness roughness expansion ruggedness that represent the dominant spatial structure of a scene then we show that these dimensions may be reliably estimated using spectral and coarsely localized information the model generates a multidimensional space in which scenes sharing membership in semantic categories e g streets highways coasts are projected closed together the performance of the spatial envelope model shows that specific information about object shape or identity is not a requirement for scene categorization and that modeling a holistic representation of the scene informs about its probable semantic category

statistical independence and novelty detection with information preserving nonlinear maps according to barlow 1989 feature extraction can be understood as finding a statistically independent representation of the probability distribution underlying the measured signals the search for a statistically independent representation can be formulated by the criterion of minimal mutual information which reduces to decorrelation in the case of gaussian distributions if nongaussian distributions are to be considered minimal mutual information is the appropriate generalization of decorrelation as used in linear principal component analyses pca we also generalize to nonlinear transformations by only demanding perfect transmission of information this leads to a general class of nonlinear transformations namely symplectic maps conservation of information allows us to consider only the statistics of single coordinates the resulting factorial representation of the joint probability distribution gives a density estimation we apply this concept to the real world problem of electrical motor fault detection treated as a novelty detection task

some results on the behavior and estimation of the fractal dimensions of distributions on attractors the strong interest in recent years in analyzing chaotic dynamical systems according to their asymptotic behavior has led to various definitions of fractal dimension and corresponding methods of statistical estimation in this paper we first provide a rigorous mathematical framework for the study of dimension focusing on pointwise dimension x and the generalized renyi dimensionsd q and give a rigorous proof of inequalities first derived by grassberger and procaccia and hentschel and procaccia we then specialize to the problem of statistical estimation of the correlation dimension and information dimension it has been recognized for some time that the error estimates accompanying the usual procedures which generally involve least squares methods and nearest neighbor calculations grossly underestimate the true statistical error involved in least squares analyses of and we identify sources of error not previously discussed in the literature and address the problem of obtaining accurate error estimates we then develop an estimation procedure for which corrects for an important bias term the local measure density and provides confidence intervals for the general applicability of this method is illustrated with various numerical examples

testing nonlinear markovian hypotheses in dynamical systems abstract we present a statistical approach for detecting the markovian character of dynamical systems by analyzing their flow of information especially in the presence of noise which is mostly the case for real world time series the calculation of the information flow of the underlying system via the concept of symbolic dynamics is rather problematic since one has to use infinitesimal partitions we circumvent this difficulty by measuring the information flow indirectly more precisely we calculate a measure based on higher order cumulants which quantifies the statistical dependencies between the past values of the time series and the point r steps ahead as an extension of theiler s method of surrogate data theiler et al 1992 this cumulant based information flow a function of the look ahead r is used as the discriminating statistic in testing the observed dynamics against a hierarchy of null hypotheses corresponding to nonlinear markov processes of increasing order this procedure is iterative in the sense that whenever a null hypothesis is rejected new data sets can be generated corresponding to better approximations of the original process in terms of information flow since we use higher order cumulants for calculating the discriminating statistic our method is also applicable to small data sets numerical results on artificial and real world examples including non chaotic nonlinear processes autoregressive models and noisy chaos show the effectiveness of our approach

robust coding over noisy overcomplete channels we address the problem of robust coding in which the signal information should be preserved in spite of intrinsic noise in the representation we present a theoretical analysis for 1 and 2 d cases and characterize the optimal linear encoder and decoder in the mean squared error sense our analysis allows for an arbitrary number of coding units thus including both under and over complete representations and provides insights into optimal coding strategies in particular we show how the form of the code adapts to the number of coding units and to different data and noise conditions in order to achieve robustness we also present numerical solutions of robust coding for high dimensional image data demonstrating that these codes are substantially more robust than other linear image coding methods such as pca ica and wavelets

limit behaviour and scaling relations of two kinds of noisy logistic map in the vicinity of chaos threshold and their robustness abstract in this paper we numerically investigate the distribution of the sums of the iterates of the logistic map and the relationships among the important properties of the nonlinear dynamics in the vicinity of the chaos threshold by adding two kinds of contributions with different densities the first one is the well known white noise whereas the second is a newly defined one named as quartic term which makes contributions from the own structure of the map as the chaos threshold is approached the iterates of the standard logistic map i e noise free have strong correlations and the standard central limit theorem is not valid anymore in a recent work tirnakli 2009 it has been shown that the limit distribution seems to converge to q gaussian distribution which maximizes the nonadditive entropy s q 1 i p i q q 1 under appropriate conditions in this work we investigate the effect of these contributions i e white noise and quartic term on the limit distribution and on the range of the obtained q gaussian distribution as a result of these findings under the existence of white noise and also the quartic term we analyse the validity of the scaling relations among correlation fractality the lyapunov divergence and q gaussian distributions which have recently been observed in afsar 2014 the results obtained here strengthen the argument that the central limit behaviour is given by a q gaussian as the chaos threshold is approached and indicate that the scaling relations obtained for the standard logistic map among the range of the q gaussian the correlation dimension the correlation length the lyapunov exponent fractality and the distance from the chaos threshold are robust under the existence of white noise and the quartic term

computation of lyapunov spectra effect of interactive noise and application to a chemical oscillator abstract a refined version of the sano sawada algorithm to compute lyapunov spectra has been implemented and tested with several models which include periodic chaotic and hyperchaotic attractors under conditions designed to mimic experimental situations it was found that reliable lyapunov exponents can be obtained for time series of moderate length and point density also in the presence of interactive noise as long as its amplitude does not exceed 1 of the oscillation amplitude the algorithm was then applied to experimental data from the co oxidation on pt 110 clearly showing the occurrence of chaos as well as hyperchaos

small scale turbulence in the plankton low order deterministic chaos or high order stochasticity abstract almost all regions of the oceans are heavily influenced by the effects of physical processes such as turbulence which in turn influence the distribution and ecology of organisms that occupy these regions there is a real need for additional focus placed on the precise knowledge of both physical and biological processes which is often difficult using basic time series analysis in that way we applied nonlinear analysis techniques to high frequency time series of temperature salinity and phytoplankton concentrations recorded in different hydrodynamical regimes related to tidal forcing in a tidally mixed coastal ecosystem techniques devoted to the identification of low order deterministic chaos cannot find evidence of chaos while a lower dimensionality was identified in low hydrodynamic conditions the results rather suggest stochastic time series with many degrees of freedom no obvious attractor in phase space trajectory lle indistinguishable from zero absence of convergence of the correlation integral we then applied to these data specific multifractal analysis techniques and showed that these time series clearly exhibit high order stochasticity in addition the stochastic structure of purely passive scalars i e temperature and salinity remained invariant while the one of phytoplankton biomass must be regarded as highly structured in time by both hydrodynamic and advective processes

on the identification of bilinear systems from operating records in this paper the identification of a non linear system when the input output record is available is considered the system discussed is an open loop system it is assumed that the unobservable output x t at time t and the observable input w s s t are related by a bilinear model it is further assumed that only a contaminated version of the output is observed i e y t whore for each t y t x t z t and x t and z t are mutually independent the identification of the system is considered in the following cases i where the external noise z t is a gaussian white noise and ii where z t is a gaussian stationary time series satisfying an autoregressive moving average arma model of order g j the estimation procedures are illustrated with examples

what does the retina know about natural scenes by examining the experimental data on the statistical properties of natural scenes together with retinal contrast sensitivity data we arrive at a first principle theoretical hypothesis for the purpose of retinal processing and its relationship to an animal s environment we argue that the retinal goal is to transform the visual input as much as possible into a statistically independent basis as the first step in creating a redundancy reduced representation in the cortex as suggested by barlow the extent of this whitening of the input is limited however by the need to suppress input noise our explicit theoretical solutions for the retinal filters also show a simple dependence on mean stimulus luminance they predict an approximate weber law at low spatial frequencies and a de vries rose law at high frequencies assuming that the dominant source of noise is quantum we generate a family of contrast sensitivity curves as a function of mean luminance this family is compared to psychophysical data

face recognition using ipca ica algorithm in this paper a fast incremental principal non gaussian directions analysis algorithm called ipca ica is introduced this algorithm computes the principal components of a sequence of image vectors incrementally without estimating the covariance matrix so covariance free and at the same time transforming these principal components to the independent directions that maximize the non gaussianity of the source two major techniques are used sequentially in a real time fashion in order to obtain the most efficient and independent components that describe a whole set of human faces database this procedure is done by merging the runs of two algorithms based on principal component analysis pca and independent component analysis ica running sequentially this algorithm is applied to face recognition problem simulation results on different databases showed high average success rate of this algorithm compared to others

exploring dynamic property of traffic flow time series in multi states based on complex networks phase space reconstruction versus visibility graph abstract a new method based on complex network theory is proposed to analyze traffic flow time series in different states we use the data collected from loop detectors on freeway to establish traffic flow model and classify the flow into three states based on k means method we then introduced two widely used methods to convert time series into networks phase space reconstruction and visibility graph furthermore in phase space reconstruction we discuss how to determine delay time constant and embedding dimension and how to select optimal critical threshold in terms of cumulative degree distribution in the visibility graph we design a method to construct network from multi variables time series based on logical or finally we study and compare the statistic features of the networks converted from original traffic time series in three states based on phase space and visibility by using the degree distribution network structure correlation of the cluster coefficient to betweenness and degree degree correlation

matching pursuit filters applied to face identification we present a face identification algorithm that automatically processes an unknown image by locating and identifying the face the heart of the algorithm is the use of pursuit filters a matching pursuit filter is an adapted wavelet expansion where the expansion is adapted to both the data and the pattern recognition problem being addressed for identification the filters find the features that differentiate among faces whereas for detection the filters encode the similarities among faces the filters are designed though a simultaneous decomposition of a training set into a two dimensional 2 d wavelet expansion this yields a representation that is explicitly 2 d and encodes information locally the algorithm uses coarse to fine processing to locate a small set of key facial features which are restricted to the nose and eye regions of the face the result is an algorithm that is robust to variations in facial expression hair style and the surrounding environment based on the locations of the facial features the identification module searches the data base for the identity of the unknown face using matching pursuit filters to make the identification the algorithm was demonstrated on three sets of images the first set was images from the feret data base the second set was infrared and visible images of the same people this demonstration was done to compare performance on infrared and visible images individually and on fusing the results from both modalities the third set was mugshot data from a law enforcement application

convergent algorithm for sensory receptive field development an unsupervised developmental algorithm for linear maps is derived which reduces the pixel entropy using the measure introduced in previous work at every update and thus removes pairwise correlations between pixels since the measure of pixel entropy has only a global minimum the algorithm is guaranteed to converge to the minimum entropy map such optimal maps have recently been shown to possess cognitively desirable properties and are likely to be used by the nervous system to organize sensory information the algorithm derived here turns out to be one proposed by goodall for pairwise decorrelation it is biologically plausible since in a neural network implementation it requires only data available locally to a neuron in training over ensembles of two dimensional input signals with the same spatial power spectrum as natural scenes networks develop output neurons with center surround receptive fields similar to those of ganglion cells in the retina some technical issues pertinent to developmental algorithms of this sort such as symmetry fixing are also discussed

characterization of strange attractors a new measure of strange attractors is introduced which offers a practical algorithm to determine their character from the time series of a single observable the relation of this new measure to fractal dimension and information theoretic entropy is discussed

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples