reciprocal focus profile a focus profile having a steeper peak is more resistant to image noise in the autofocus af process of a digital camera however a focus profile of such shape normally has a flatter out of focus region on either side of the profile resulting in a slow af process due to the lack of clue about where the lens should move when the lens is in such regions to address the problem we provide a statistical analysis of the focus profile and show that a strictly monotonic transformation of the focus profile preserves the accuracy of the af on the basis of this analysis we propose a new focus profile representation that transforms the focus profile to the reciprocal domain in which the reciprocal focus profile is modeled by a polynomial function this transformation makes the af mathematically tractable and boosts the search speed experimental results are shown to demonstrate the advantage of the proposed representation

complexity issues in natural gradient descent method for training multilayer perceptrons the natural gradient descent method is applied to train an n m 1 multilayer perceptron based on an efficient scheme to represent the fisher information matrix for an n m 1 stochastic multilayer perceptron a new algorithm is proposed to calculate the natural gradient without inverting the fisher information matrix explicitly when the input dimension n is much larger than the number of hidden neurons m the time complexity of computing the natural gradient is o n

equivariant adaptive selective transmission in this paper we consider the problem of selective transmission the dual of the blind source separation task in which a set of independent source signals are adaptively premixed prior to a nondispersive physical mixing process so that each source can be independently monitored in the far field following similar procedures for information theoretic blind source separation we derive a stochastic gradient algorithm for iteratively estimating the premixing matrix in the selective transmission problem and through a simple modification we obtain a second algorithm whose performance is equivariant with respect to the channel s mixing characteristics the local stability conditions for the algorithms about any selective transmission solution are shown to be the same as those for similar source separation algorithms practical implementation issues are discussed including the estimation of the combined system matrix and the reordering and scaling of the received signals within the algorithm mean square error based selective transmission algorithms are also derived for performance comparison purposes simulations indicate the useful behavior of the premixing algorithms for selective transmission

blind separation of a mixture of uniformly distributed source signals a novel approach a new efficient algorithm for blind separation of uniformly distributed sources is proposed the mixing matrix is assumed to be orthogonal by prewhitening the observed signals the learning rule adaptively estimates the mixing matrix by conceptually rotating a unit hypercube so that all output signal components are contained within or on the hypercube under some ideal constraints it has been theoretically shown that the algorithm is very similar to an ideal convergent algorithm which is much faster than the existing convergent algorithms the algorithm has been generalized to take care of the noisy signals by adaptively dilating the hypercube in conjunction with its rotation

robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size independent component analysis ica aims at decomposing an observed random vector into statistically independent variables deflation based implementations such as the popular one unit fastica algorithm and its variants extract the independent components one after another a novel method for deflationary ica referred to as robustica is put forward in this paper this simple technique consists of performing exact line search optimization of the kurtosis contrast function the step size leading to the global maximum of the contrast along the search direction is found among the roots of a fourth degree polynomial this polynomial rooting can be performed algebraically and thus at low cost at each iteration among other practical benefits robustica can avoid prewhitening and deals with real and complex valued mixtures of possibly noncircular sources alike the absence of prewhitening improves asymptotic performance the algorithm is robust to local extrema and shows a very high convergence speed in terms of the computational cost required to reach a given source extraction quality particularly for short data records these features are demonstrated by a comparative numerical analysis on synthetic data robustica s capabilities in processing real world data involving noncircular complex strongly super gaussian sources are illustrated by the biomedical problem of atrial activity aa extraction in atrial fibrillation af electrocardiograms ecgs where it outperforms an alternative ica based technique

energy efficient distributed filtering with wireless sensor networks we consider a wireless sensor network wsn that monitors a physical field and communicates pertinent data to a distant fusion center fc we study the case of a binary valued hidden natural field observed in a significant amount of gaussian clutter which is relevant to applications like detection of plumes or oil slicks the considerable spatio temporal dependencies found in natural fields can be exploited to improve the reliability of the detection estimation of hidden phenomena while this problem has been previously treated using kernel regression techniques we formulate it as a task of delay free filtering on a process observed by the sensors we propose a distributed scalable implementation of the filter within the network this is achieved by i exploiting the localized spatio temporal dependencies to define a hidden markov model hmm in terms of an exponential family with o n parameters where n is the size of the wsn ii using a reduced state approximation of the propagated probability mass function and iii making a tractable approximation of model marginals by using iterated decoding algorithms like the gibbs sampler gs mean field decoding mfd iterated conditional modes icm and broadcast belief propagation bbp we compare the marginalization algorithms in terms of their information geometry performance complexity and communication load finally we analyze the energy efficiency of the proposed distributed filter relative to brute force data fusion it is demonstrated that when the fc is sufficiently far away from the sensor array distributed filtering is significantly more energy efficient and can increase the lifetime of the wsn by one to two orders of magnitude

blind source separation semiparametric statistical approach the semiparametric statistical model is used to formulate the problem of blind source separation the method of estimating functions is applied to this problem it is shown that an estimator of the mixing matrix or its learning version can be described in terms of an estimating function the statistical efficiencies of these algorithms are studied the main results are as follows 1 the space consisting of all the estimating functions is derived 2 the space is decomposed into the orthogonal sum of the admissible part and a redundant ancillary part for any estimating function one can find a better or equally good estimator in the admissible part 3 the fisher efficient that is asymptotically best estimating functions are derived 4 the stability of learning algorithms is studied

a game theoretic approach to robust filtering a game theoretic approach to the filtering or smoothing problem is presented a family of stationary information carrying processes and generalized models for the noise channel and the filter is considered sufficient conditions for the existence of saddle point type solutions are stated in addition the solution for a special case of noise channel a family of information carrying processes and a nonlinear filter are found

information geometry of boltzmann machines a boltzmann machine is a network of stochastic neurons the set of all the boltzmann machines with a fixed topology forms a geometric manifold of high dimension where modifiable synaptic weights of connections play the role of a coordinate system to specify networks a learning trajectory for example is a curve in this manifold it is important to study the geometry of the neural manifold rather than the behavior of a single network in order to know the capabilities and limitations of neural networks of a fixed topology using the new theory of information geometry a natural invariant riemannian metric and a dual pair of affine connections on the boltzmann neural network manifold are established the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view this leads to a natural modification of the boltzmann machine learning rule

learning dynamics on different timescales the special character of certain degrees of freedom in two layered neural networks is investigated for on line learning of realizable rules our analysis shows that the dynamics of these degrees of freedom can be put on a faster timescale than those remaining with the profit of speeding up the overall adaptation process this is shown for two groups of degrees of freedom second layer weights and bias weights for the former case our analysis provides a theoretical explanation of phenomenological findings the resulting learning algorithm is compared with natural gradient descent in order to check whether the proposed scaling can be naturally derived from that type of learning rule

a generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures in this letter the problem of blind separation of n independent sources from their m linear instantaneous mixtures is considered first a generalized contrast function is defined as a valuable extension of the existing classical and nonsymmetrical contrast functions it is applicable to the overdetermined blind separation m n with an unknown number of sources because not only independent components but also redundant ones are allowed in the outputs of a separation system second a natural gradient learning algorithm developed primarily for the complete case m n is shown to work as well with an n m or m m separating matrix for each optimizes a certain mutual information contrast function finally we present stability analysis for a newly proposed generalized orthogonal natural gradient algorithm which can perform the overdetermined blind separation when n is unknown obtaining an expectable result that its local stability conditions are slightly stricter than those of the conventional natural gradient algorithm using an invertible mixing matrix m n

rational filters for passive depth from defocus a fundamental problem in depth from defocus is the measurement of relative defocus between images the performance of previously proposed focus operators are inevitably sensitive to the frequency spectra of local scene textures as a result focus operators such as the laplacian of gaussian result in poor depth estimates an alternative is to use large filter banks that densely sample the frequency space though this approach can result in better depth accuracy it sacrifices the computational efficiency that depth from defocus offers over stereo and structure from motion we propose a class of broadband operators that when used together provide invariance to scene texture and produce accurate and dense depth maps since the operators are broadband a small number of them are sufficient for depth estimation of scenes with complex textural properties in addition a depth confidence measure is derived that can be computed from the outputs of the operators this confidence measure permits further refinement of computed depth maps experiments are conducted on both synthetic and real scenes to evaluate the performance of the proposed operators the depth detection gain error is less than 1 irrespective of texture frequency depth accuracy is found to be 0 5 1 2 of the distance of the object from the imaging optics

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

higher order boltzmann machines and entropy bounds we examine some aspects of the interface area between mathematical statistics and statistical physics relevant to the study of boltzmann machines the boltzmann machine learning algorithm is based on a variational principle gibbs lemma for relative entropy this fact suggests the possibility of a scheme of successive approximations here we consider successive approximations parametrized by the order of many body interactions among individual units we prove bounds on the gain in relative entropy in the crucial step of adding and estimating by hebb s rule a new parameter we address the problem of providing on the basis of local observations upper and lower bounds on the entropy while upper bounds are easily obtained by subadditivity lower bounds involve localization of hirschman bounds on a dual quantum system

information geometric algorithm for estimating switching probabilities in space varying hmm this paper proposes an iterative natural gradient algorithm to perform the optimization of switching probabilities in a space varying hidden markov model in the context of human activity recognition in long range surveillance the proposed method is a version of the gradient method developed under an information geometric viewpoint where the usual euclidean metric is replaced by a riemannian metric on the space of transition probabilities it is shown that the change in metric provides advantages over more traditional approaches namely 1 it turns the original constrained optimization into an unconstrained optimization problem 2 the optimization behaves asymptotically as a newton method and yields faster convergence than other methods for the same computational complexity and 3 the natural gradient vector is an actual contravariant vector on the space of probability distributions for which an interpretation as the steepest descent direction is formally correct experiments on synthetic and real world problems focused on human activity recognition in long range surveillance settings show that the proposed methodology compares favorably with the state of the art algorithms developed for the same purpose

self calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters in this paper the theoretical and practical feasibility of self calibration in the presence of varying intrinsic camera parameters is under investigation the paper s main contribution is to propose a self calibration method which efficiently deals with all kinds of constraints on the intrinsic camera parameters within this framework a practical method is proposed which can retrieve metric reconstruction from image sequences obtained with uncalibrated zooming focusing cameras the feasibility of the approach is illustrated on real and synthetic examples besides this a theoretical proof is given which shows that the absence of skew in the image plane is sufficient to allow for self calibration a counting argument is developed which depending on the set of constraints gives the minimum sequence length for self calibration and a method to detect critical motion sequences is proposed

robust recursive estimation in the presence of heavy tailed observation noise a first order approximation is derived for the conditional prior distribution of the state of a discrete time stochastic linear dynamic system in the presence of contaminated normal observation noise this distribution is then used to derive a first order approximation of the conditional mean minimum variance estimator if the observation noise distribution can be represented as a member of the e contaminated normal neighborhood then the conditional prior is also to first order an analogous perturbation from a normal distribution whose first two moments are given by the kalman filter moreover the perturbation is itself of a special form combining distributions whose parameters are given by banks of parallel kalman filters and optimal smoothers

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

from shrinking to percolation in an optimization model a model of noise reduction for signal processing and other optimization tasks is introduced each noise source puts a symmetric constraint on the space of the signal vector within a tolerance bound when the number of noise sources increases sequences of transitions take place causing the solution space to vanish we find that the transition from an extended solution space to a shrunk space is retarded because of the symmetry of the constraints in contrast with the analogous problem of pattern storage for low tolerance the solution space vanishes by volume reduction whereas for high tolerance the vanishing becomes more and more like percolation

outlier resistant algorithms for detecting a change in a stochastic process outlier resistant algorithms that detect a change from a given nominal stationary process to another such process are given the nominal processes are assumed to be mutually independent and to satisfy some general regularity conditions the outlier sequences are assumed to be independently and identically distributed and independent of the nominal processes the proposed algorithms are sequential and consist of uniformly bounded steps the asymptotic performance of the algorithms is analyzed both in the absence and the presence of outliers breakdown points and influence functions are defined and analyzed the algorithms are studied in more detail for gaussian autoregressive nominal processes

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

robust wiener filtering for multiple inputs with channel distortion corresp robust wiener filtering has previously been considered for the single input scalar case where there is no channel distortion and where the signal to be estimated is the source signal itself here these results are extended to the multiple input vector case where linear channel distortion is allowed and the signal to be estimated is a linear filtered version of the source signal the results are obtained from those for the single input ease by modifying the constraints on signal and noise characteristics such a modification is motivated by examining the expression of the mean squared error for the optimum filter

on minimax robustness a general approach and applications the minimax approach to the design of systems that are robust with respect to modeling uncertainties is studied using a game theoretic formulation in which the performance functional and the sets of modeling uncertainties and admissible design policies are arbitrary the existence and characterization of minimax robust solutions that form saddle points are discussed through various methods that take into account several common features of the games encountered in applications in particular it is shown that if the performance functional and the uncertainty set are convex then a certain type of regularity condition on the functional is sufficient to ensure that the optimal strategy for a least favorable element of the uncertainty set is minimax robust the efficacy of the methods proposed for a general game is tested in the problems of matched filtering wiener filtering quadratic detection and output energy filtering in which uncertainties in their respective signal and noise models are assumed to exist these problems are analyzed in a common hilbert space framework and they serve to point out the advantages and limitations of the proposed techniques

recurrent sampling models for the helmholtz machine many recent analysis by synthesis density estimation models of cortical learning and processing have made the crucial simplifying assumption that units within a single layer are mutually independent given the states of units in the layer below or the layer above in this article we suggest using either a markov random field or an alternative stochastic sampling architecture to capture explicitly particular forms of dependence within each layer we develop the architectures in the context of real and binary helmholtz machines recurrent sampling can be used to capture correlations within layers in the generative or the recognition models and we also show how these can be combined

on line versus off line learning from random examples general results i propose a general model of on line learning from random examples which when applied to a smooth realizable stochastic rule yields the same asymptotic generalization error rate as optimal batch algorithms the approach is based on an iterative gaussian approximation to the posterior gibbs distribution of rule parameters

serial updating rule for blind separation derived from the method of scoring in the context of blind source separation the method of scoring based on the inverse of the fisher information matrix fim becomes the serial updating learning rule with an equivariant property this learning rule can be simplified to a low complexity algorithm by using the asymptotic form of the ftm around the equilibrium the simplified learning rule is still general enough to include some existing equivariant blind separation algorithms as its special cases

a globally convergent mc algorithm with an adaptive learning rate this brief deals with the problem of minor component analysis mca artificial neural networks can be exploited to achieve the task of mca recent research works show that convergence of neural networks based mca algorithms can be guaranteed if the learning rates are less than certain thresholds however the computation of these thresholds needs information about the eigenvalues of the autocorrelation matrix of data set which is unavailable in online extraction of minor component from input data stream in this correspondence we introduce an adaptive learning rate into the ojan mca algorithm such that its convergence condition does not depend on any unobtainable information and can be easily satisfied in practical applications

an iterative method using conditional second order statistics applied to the blind source separation problem this paper is concerned with the problem of blind separation of an instantaneous mixture of sources bss which has been addressed in many ways when power spectral densities of the sources are different methods using second order statistics are sufficient to solve this problem otherwise these methods fail and others higher order statistics etc must be used in this paper we propose an iterative method to process the case of sources with the same power spectral density this method is based on an evaluation of conditional first and second order statistics only restrictions on characteristics of sources are given to reach a solution and proofs of convergence of the algorithm are provided for particular cases of probability density functions robustness of this algorithm with respect to the number of sources is shown through computer simulations a particular case of sources that have a probability density function with unbounded domain of definition is described here the algorithm does not lead directly to a separation state but to an a priori known mixture state finally prospects of links with contrast functions are mentioned with a possible generalization of them based on results obtained with particular sources

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

consistent estimation of continuous time signals from nonlinear transformations of noisy samples a signal cannot in general be reconstructed from its sign i e from its hard limited version however by the deliberate addition of noise to samples of the signal prior to hard limiting it is shown that the signal can be estimated consistently from its hard limited noisy samples as the sampling rate tends to infinity in fact such estimates are shown to converge with probability one to the signal and to be asymptotically normal although the estimates are in general nonlinear they can be made linear by a proper choice of the noise distribution these rather unexpected results hold for all bounded and uniformly continuous signals in addition to the hard limiter such results are also established for certain monotonic and nonmonotonic nonlinearities

dynamics of learning in multilayer perceptrons near singularities the dynamical behavior of learning is known to be very slow for the multilayer perceptron being often trapped in the plateau it has been recently understood that this is due to the singularity in the parameter space of perceptrons in which trajectories of learning are drawn the space is riemannian from the point of view of information geometry and contains singular regions where the riemannian metric or the fisher information matrix degenerates this paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity we give explicit asymptotic analytical solutions trajectories both for the standard gradient sgd and natural gradient ngd methods it is clearly shown in the case of the sgd method that the plateau phenomenon appears in a neighborhood of the critical regions where the dynamical behavior is extremely slow the analysis of the ngd method is much more difficult because the inverse of the fisher information matrix diverges we conquer the difficulty by introducing the blow down technique used in algebraic geometry the ngd method works efficiently and the state converges directly to the true parameters very quickly while it staggers in the case of the sgd method the analytical results are compared with computer simulations showing good agreement the effects of singularities on learning are thus qualitatively clarified for both standard and ngd methods

analysis of natural gradient descent for multilayer neural networks natural gradient descent ngd is an on line algorithm for redefining the steepest descent direction an analysis of ngd for training a multilayer neural network is presented using statistical mechanics the performance can be significantly improved using ngd algorithm and can be used for both the transient and asymptotic stages of learning

statistical inference for the population landscape via moment adjusted stochastic gradients modern statistical inference tasks often require iterative optimization methods to compute the solution convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data in contrast recognizing the randomness in the data statisticians are keen to provide uncertainty quantification or confidence for the solution obtained by using iterative optimization methods the paper makes progress along this direction by introducing moment adjusted stochastic gradient descent a new stochastic optimization method for statistical inference we establish non asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees on the statistical front the theory allows for model misspecification with very mild conditions on the data for optimization the theory is flexible for both convex and non convex cases remarkably the moment adjusting idea motivated from error standardization in statistics achieves a similar effect to acceleration in first order optimization methods that are used to fit generalized linear models we also demonstrate this acceleration effect in the non convex setting through numerical experiments

qualitative robustness in time series abstract we consider robust operations in time series we present a definition and subsequent qualitative analysis of robustness we also present meaningful definitions of performance criteria such as the breakdown point and the sensitivity of robust operations we present some specific classes of robust operations and we discuss and analyze their properties finally we fully analyze a particular class of robust predictors and interpolators for a linearly contaminated class of stationary stochastic processes

sliding block encoders that are rho bar continuous functions of their input corresp sliding block encoders operating within a statistically contaminated source environment are considered stationary analog sources are studied using the rho bar distance as the measure of contamination sufficient constructive conditions are developed for the design of sliding block encoders that guarantee rho bar stability of their output when operating within a statistically contaminated source environment the relationship of this stability to the robustness of the encoding scheme is discussed where robustness is a local stability property applied on the output entropy and the average distortion

nonlinear kalman filtering with divergence minimization we consider the nonlinear kalman filtering problem using kullback leibler kl and alpha divergence measures as optimization criteria unlike linear kalman filters nonlinear kalman filters do not have closed form gaussian posteriors because of a lack of conjugacy due to the nonlinearity in the likelihood in this paper we propose novel algorithms to approximate this posterior by optimizing the forward and reverse forms of the kl divergence as well as the alpha divergence that contains these two as limiting cases unlike previous approaches our algorithms do not make approximations to the divergences being optimized but use monte carlo techniques to derive unbiased algorithms for direct optimization we assess performance on radar and sensor tracking and options pricing showing general improvement over the extended unscented and ensemble kalman filters as well as competitive performance with particle filtering

robust linear filtering for multivariable stationary time series we consider the problem of asymptotic noncausal linear filtering for multivariable second order stationary time series under spectral uncertainty in both the signal and the noise processes the spectral uncertainty is modeled by e contaminated and p point classes for the case where both the signal and noise spectra are e contaminated we assume that the eigenvectors of the corresponding nominal spectral density matrices are identical the problem is formulated as a game whose saddle point solutions are found and analyzed

sampling for shape from focus in optical microscopy shape from focus sff which relies on image focus as a cue within sequenced images represents a passive technique in recovering object shapes in scenes although numerous methods have been recently proposed less attention has been paid to particular factors affecting them in regard to sff one such critical factor impacting system application is the total number of images a large data set requires a huge amount of computation power whereas decreasing the number of images causes shape reconstruction to be crude and erroneous the total number of images is inversely proportional to interframe distance or sampling step size in this paper interframe distance or sampling step size criteria for sff systems have been formulated in particular light ray focusing is approximated by the use of a gaussian beam followed by the formulation of a sampling expression using nyquist sampling consequently a fitting function for focus curves is also obtained experiments are performed on simulated and real objects to validate the proposed schemes

learning dynamics of a single polar variable complex valued neuron this letter investigates the characteristics of the complex valued neuron model with parameters represented by polar coordinates called polar variable complex valued neuron the parameters of the polar variable complex valued neuron are unidentifiable the plateau phenomenon can occur during learning of the polar variable complex valued neuron furthermore computer simulations suggest that a single polar variable complex valued neuron has the following characteristics in the case of using the steepest gradient descent method with square error 1 unidentifiable parameters singular points degrade the learning speed and 2 a plateau can occur during learning when the weight is attracted to the singular point the learning tends to become stuck however computer simulations also show that the steepest gradient descent method with amplitude phase error and the complex valued natural gradient method could reduce the effects of the singular points the learning dynamics near singular points depends on the error functions and the training algorithms used

efficient focus sampling through depth of field calibration due to the limited depth of field dof of conventional digital cameras only the objects within a certain distance range from the camera are in focus objects outside the dof are observed with different amounts of defocus depending on their position focus sampling consists of capturing different images of the same scene by changing the focus configuration of the camera in order to alternately bring objects at different depths into focus focus sampling is an important part of different focus related applications such as autofocus focus stacking and depth estimation this work proposes a calibration procedure for modeling the depth of field of conventional cameras in order to perform an efficient focus sampling the method is simple in terms of repeatability and can be easily implemented in different imaging devices experimental tests are presented in order to illustrate the effectiveness of the proposed approach in autofocus results demonstrate that a significant reduction in the number of frames required to capture during autofocusing can be achieved by means of the proposed methodology

superefficiency in blind source separation blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to eeg and meg imaging of the brain it is already known that certain algorithms work well for the extraction of independent components the present paper is concerned with superefficiency of these based on the statistical and dynamical analysis in a statistical estimation using t examples the covariance of any two extracted independent signals converges to 0 of the order of 1 t on line dynamics shows that the covariance is of the order of spl eta when the learning rate spl eta is fixed to a small constant in contrast with the above general properties a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1 t sup 2 or of spl eta sup 2 the paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on line learning a standardized estimating function is introduced to this end superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1 t sup 2 or spl eta sup 2 but implies that their covariances and independencies do

natural gradient learning for over and under complete bases in ica independent component analysis or blind source separation is a new technique of extracting independent signals from mixtures it is applicable even when the number of independent sources is unknown and is larger or smaller than the number of observed mixture signals this article extends the natural gradient learning algorithm to be applicable to these overcomplete and undercomplete cases here the observed signals are assumed to be whitened by preprocessing so that we use the natural riemannian gradient in stiefel manifolds

matrix momentum for practical natural gradient learning an on line learning rule based on the introduction of a matrix momentum term is presented aimed at alleviating the computational costs of standard natural gradient learning the new rule natural gradient matrix momentum is analysed in the case of two layer feed forward neural network learning via methods of statistical physics it appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity

sufficient dimension reduction via squared loss mutual information estimation the goal of sufficient dimension reduction in supervised learning is to find the low dimensional subspace of input features that contains all of the information about the output values that the input features possess in this letter we propose a novel sufficient dimension reduction method using a squared loss variant of mutual information as a dependency measure we apply a density ratio estimator for approximating squared loss mutual information that is formulated as a minimum contrast estimator on parametric or nonparametric models since cross validation is available for choosing an appropriate model our method does not require any prespecified structure on the underlying distributions we elucidate the asymptotic bias of our estimator on parametric models and the asymptotic convergence rate on nonparametric models the convergence analysis utilizes the uniform tail bound of a u process and the convergence rate is characterized by the bracketing entropy of the model we then develop a natural gradient algorithm on the grassmann manifold for sufficient subspace search the analytic formula of our estimator allows us to compute the gradient efficiently numerical experiments show that the proposed method compares favorably with existing dimension reduction approaches on artificial and benchmark data sets

robust source coding of weakly compact classes a fixed rate block code is said to be strongly robust for a class of sources lambda if its maximum distortion over lambda h is no larger than the maximum of the distortion rate functions of the sources in the class at the rate of the code it is shown that such codes exist at all positive rates whenever the class is compact with resect to the topology of weak convergence and satisfies certain additional but not very strong constraints on the alphabet and distortion measure examples of classes that satisfy these conditions are given in addition classes are exhibited for which there are no strongly robust codes these help to demarcate the boundary between classes for which strongly robust codes do and do not exist more specifically they show that strongly robust codes exist more widely than strongly universal codes but ess widely than weakly universal codes

a mean field optimal control formulation for global optimization this paper is concerned with variational optimal control constructions whose solution yields a sampling algorithm the particular form of the sampling algorithm considered here is a particle filter designed to numerically approximate the solution to the global optimization problem the theoretical significance of this study comes from its variational aspects specifically the control input represents the solution of a mean field type optimal control problem its parametric counterpart obtained when a parametric form of density is known a priori is shown to be equivalent to the natural gradient algorithm explicit formulae for the filter are derived when the objective function is quadratic and the density is gaussian the optimal control construction of the particle filter is a significant departure from the classical importance sampling resampling based approaches

predictive analog to digital conversion for resistance to data outliers abstract we consider predictive source encoding for analog to digital conversion that is resistant to data outliers we focus on a gaussian stationary source and we develop a sequence of outlier resistant source oncoders the studied performance criteria include mean squared distortion and output entropy at the nominal gaussian source when data outliers are absent as well as outlier resistance measures such as breakdown point and influence function the proposed sequence of predictive encoders attains high resistance to data outliers at the expense of some increase in mean squared distortion and differential entropy at the nominal gaussian source

stochastic quantization for performance stability we consider stochastic stationary analog signals for such signals we consider quantizers whose performance remains stable under perturbations in the statistical description of the signals performance is defined as the induced average distortion the output entropy and the asymptotic consistency in the preservation of important signal characteristics we found that stochastic quantizers can in general realize performance stability while deterministic quantizers do not we consider block quantization for lower average distortion under fixed output rate and we propose predictive stochastic quantization for highly correlated signals the proposed predictive quantizers compress automatically to the output entropy we also propose nonpredictive block stochastic quantization for weakly correlated or uncorrelated signals we describe an elias type noiseless compressor for channel transmission and a matching decoder

nonlinear filters for linear models a robust approach we consider the altering problem for linear models where the driving noises may be quite general nonwhite and non gaussian and where the observation noise may only be known to belong to a finite family of possible disturbances using diffusion approximation methods we show that a certain nonlinear filter minimizes the asymptotic filter variance this nonlinear filter is obtained by choosing at each moment on the basis of the observations one of a finite number of kalman type filters driven by a suitable nonlinear transformation of the innovations as a byproduct we obtain also the asymptotic identification of the a priori unknown observation noise disturbance by yielding an asymptotically efficient filter in face of an unknown observation noise our approach may also be viewed as a robust approach to filtering for linear models

approximate riemannian conjugate gradient learning for fixed form variational bayes variational bayesian vb methods are typically only applied to models in the conjugate exponential family using the variational bayesian expectation maximisation vb em algorithm or one of its variants in this paper we present an efficient algorithm for applying vb to more general models the method is based on specifying the functional form of the approximation such as multivariate gaussian the parameters of the approximation are optimised using a conjugate gradient algorithm that utilises the riemannian geometry of the space of the approximations this leads to a very efficient algorithm for suitably structured approximations it is shown empirically that the proposed method is comparable or superior in efficiency to the vb em in a case where both are applicable we also apply the algorithm to learning a nonlinear state space model and a nonlinear factor analysis model for which the vb em is not applicable for these models the proposed algorithm outperforms alternative gradient based methods by a significant margin

adaptive method of realizing natural gradient learning for multilayer perceptrons the natural gradient learning method is known to have ideal performances for on line training of multilayer perceptrons it avoids plateaus which give rise to slow convergence of the backpropagation method it is fisher efficient whereas the conventional method is not however for implementing the method it is necessary to calculate the fisher information matrix and its inverse which is practically very difficult this article proposes an adaptive method of directly obtaining the inverse of the fisher information matrix it generalizes the adaptive gauss newton algorithms and provides a solid theoretical justification of them simulations show that the proposed adaptive method works very well for realizing natural gradient learning

natural evolution strategies this paper presents natural evolution strategies nes a recent family of black box optimization algorithms that use the natural gradient to update a parameterized search distribution in the direction of higher expected fitness we introduce a collection of techniques that address issues of convergence robustness sample complexity computational complexity and sensitivity to hyperparameters this paper explores a number of implementations of the nes family such as general purpose multi variate normal distributions and separable distributions tailored towards search in high dimensional spaces experimental results show best published performance on various standard benchmarks as well as competitive performance on others

blind separation of uniformly distributed signals a general approach a general algorithm for blind separation of uniformly distributed signals is presented first maximum likelihood equations are obtained for dealing with this task it is difficult to obtain a closed form maximum likelihood solution for arbitrary mixing matrix the learning rules are obtained based on the geometric interpretation of the maximum likelihood estimator the algorithm under special constraint of orthogonal mixing matrix is the same as the o 1 t sup 2 convergent algorithm special noise correction mechanisms are incorporated in the algorithm and it has been found that the algorithm exhibits stable performance even in the presence of large amount of noise

statistical dynamics of on line independent component analysis the learning dynamics of on line independent component analysis is analysed in the limit of large data dimension we study a simple hebbian learning algorithm that can be used to separate out a small number of non gaussian components from a high dimensional data set the de mixing matrix parameters are confined to a stiefel manifold of tall orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold for large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points each described by a diffusion process in a polynomial potential choosing the learning rate too large increases the escape time from each of these fixed points effectively trapping the learning in a sub optimal state in order to avoid these trapping states a very low learning rate must be chosen during the learning transient resulting in learning time scales of o n2 or o n3 iterations where n is the data dimension escape from each sub optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn this is in marked contrast to the learning dynamics displayed by related on line learning algorithms for multilayer neural networks and principal component analysis although the natural gradient variant of the algorithm has nice asymptotic convergence properties it has an equivalent transient dynamics to the standard hebbian algorithm

generation of all in focus images by noise robust selective fusion of limited depth of field images the limited depth of field of some cameras prevents them from capturing perfectly focused images when the imaged scene covers a large distance range in order to compensate for this problem image fusion has been exploited for combining images captured with different camera settings thus yielding a higher quality all in focus image since most current approaches for image fusion rely on maximizing the spatial frequency of the composed image the fusion process is sensitive to noise in this paper a new algorithm for computing the all in focus image from a sequence of images captured with a low depth of field camera is presented the proposed approach adaptively fuses the different frames of the focus sequence in order to reduce noise while preserving image features the algorithm consists of three stages 1 focus measure 2 selectivity measure 3 and image fusion an extensive set of experimental tests has been carried out in order to compare the proposed algorithm with state of the art all in focus methods using both synthetic and real sequences the obtained results show the advantages of the proposed scheme even for high levels of noise

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

brief paper an analysis of the effects of spectral uncertainty on wiener filtering a representative selection of results from an extensive study of the performance of wiener filtering under spectral uncertainty is presented for a variety of spectral uncertainty models the wiener filter is shown to have undesirable sensitivity to even small deviations from those signal and noise spectral densities which were used to design the filter performance of a robust filter designed to have the best possible performance when the uncertainty is worst is also examined in most cases the robust filter s insensitivity to spectral uncertainty makes it preferable to the traditional wiener filter

the reconstruction of analog signals from the sign of their noisy samples the reconstruction of a continuous time signal s t from the sign of its deliberately contaminated samples is considered sequential generally nonlinear estimates of s t are established and their performance is studied error bounds and convergence rates are derived the signal s t need not be band limited the convergence rates obtained here are faster than those obtained previously for nonsequential estimates the degradation in the reconstruction of the signal due to transmission over a noisy channel is also investigated and bounds on the additional error are obtained

outlier resistant filtering and smoothing abstract we consider a stationary gaussian information process transmitted through an additive noise channel we assume that the noise and information processes are mutually independent and we model the noise process as nominally gaussian with additive outliers for the above system model we first develop a theory for outlier resistant filtering and smoothing operations we then design specific such nonlinear operations and we study their performance the performance criteria are the asymptotic mean squared error at the gaussian nominal model the breakdown point and the influence function we find that the proposed operations combine excellent performance at the nominal model with strong resistance to outliers

independent component analysis using potts models we explore the extending application of potts encoding to the task of independent component analysis which primarily deals with the problem of minimizing the kullback leibler divergence between the joint distribution and the product of all marginal distributions of output components the competitive mechanism of potts neurons is used to encode the overlapping projections from observations to output components based on these projections the marginal distributions and the entropy of output components are made tractable for computation and the adaptation of the de mixing matrix toward independent output components is obtained the potts model for ica is well formulated by an objective function subject to a set of constraints which leads to a novel energy function a hybrid of the mean field annealing and the gradient descent method is applied to the energy function our approach to independent component analysis presents a new criterion for ica the performance of the potts model for ica given by our numerical simulations is encouraging

source separation in post nonlinear mixtures we address the problem of separation of mutually independent sources in nonlinear mixtures first we propose theoretical results and prove that in the general case it is not possible to separate the sources without nonlinear distortion therefore we focus our work on specific nonlinear mixtures known as post nonlinear mixtures these mixtures constituted by a linear instantaneous mixture linear memoryless channel followed by an unknown and invertible memoryless nonlinear distortion are realistic models in many situations and emphasize interesting properties i e in such nonlinear mixtures sources can be estimated with the same indeterminacies as in instantaneous linear mixtures the separation structure of nonlinear mixtures is a two stage system namely a nonlinear stage followed by a linear stage the parameters of which are updated to minimize an output independence criterion expressed as a mutual information criterion the minimization of this criterion requires knowledge or estimation of source densities or of their log derivatives a first algorithm based on a gram charlier expansion of densities is proposed unfortunately it fails for hard nonlinear mixtures a second algorithm based on an adaptive estimation of the log derivative of densities leads to very good performance even with hard nonlinearities experiments are proposed to illustrate these results

an alternative perspective on adaptive independent component analysis algorithms this article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources the gaussian mixture model of pearson is employed in deriving a closedform generic score function for strictly subgaussian sources this is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources

preconditioned stochastic gradient descent stochastic gradient descent sgd still is the workhorse for many practical problems however it converges slow and can be difficult to tune it is possible to precondition sgd to accelerate its convergence remarkably but many attempts in this direction either aim at solving specialized problems or result in significantly more complicated methods than sgd this paper proposes a new method to adaptively estimate a preconditioner such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to newton method for deterministic optimization unlike the preconditioners based on secant equation fitting as done in deterministic quasi newton methods which assume positive definite hessian and approximate its inverse the new preconditioner works equally well for both convex and nonconvex optimizations with exact or noisy gradients when stochastic gradient is used it can naturally damp the gradient noise to stabilize sgd efficient preconditioner estimation methods are developed and with reasonable simplifications they are applicable to large scale problems experimental results demonstrate that equipped with the new preconditioner without any tuning effort preconditioned sgd can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long term memories

robust wiener kolmogorov theory a minimax formulation is considered for the problem of designing robust linear causal estimators of linear functions of discrete time wide sense stationary signals when knowledge of the signal and or noise spectra is inexact the solution is given under mild regularity conditions in terms of a least favorable pair of spectra thus reducing the minimax problem to a direct maximization problem which in many cases can be solved easily it is noted that this design method leads in particular to robust n step predictors robust causal filters and robust n lag smoothers the method of design is illustrated by a thorough development of the special case of one step noiseless prediction further solutions are given explicitly for the problem of robust causal filtering of an uncertain signal in white noise and numerical examples are given for this case which illustrate the effectiveness of this design

newton stein method an optimization method for glms via stein s lemma we consider the problem of efficiently computing the maximum likelihood estimator in generalized linear models glms when the number of observations is much larger than the number of coefficients n p 1 in this regime optimization algorithms can immensely benefit from approximate second order information we propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a stein type lemma which allows further improvements through subsampling and eigenvalue thresholding our algorithm enjoys fast convergence rates resembling that of second order methods with modest per iteration cost we provide its convergence analysis for the general case where the rows of the design matrix are samples from a sub gaussian distribution we show that the convergence has two phases a quadratic phase followed by a linear phase finally we empirically demonstrate that our algorithm achieves the highest performance compared to various optimization algorithms on several data sets

adaptive online learning algorithms for blind separation maximum entropy and minimum mutual information there are two major approaches for blind separation maximum entropy me and minimum mutual information mmi both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix the mi is the contrast function for blind separation the entropy is not to justify the me the relation between me and mmi is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the me and at the solution points determined by the mi the me will not update the demixing matrix in the directions of increasing the cross talking second the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms because the parameter space is a riemannian space consisting of matrices the mutual information is calculated by applying the gram charlier expansion to approximate probability density functions of the outputs finally we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants it is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants

robust prediction and interpolation for vector stationary processes robust multivariate prediction and interpolation problems for statistically contaminated vector valued second order stationary processes are considered the statistical contamination is modeled by requiring that the spectral density matrices of the processes lie within certain nonparametric classes both prediction and interpolation are then formalized as games whose saddle point solutions are sought finally such solutions are found and analyzed for two specific multivariate spectral classes

analysis of focus measure operators for shape from focus shape from focus sff has widely been studied in computer vision as a passive depth recovery and 3d reconstruction method one of the main stages in sff is the computation of the focus level for every pixel of an image by means of a focus measure operator in this work a methodology to compare the performance of different focus measure operators for shape from focus is presented and applied the selected operators have been chosen from an extensive review of the state of the art the performance of the different operators has been assessed through experiments carried out under different conditions such as image noise level contrast saturation and window size such performance is discussed in terms of the working principles of the analyzed operators

estimating functions of independent component analysis for temporally correlated signals this article studies a general theory of estimating functions of independent component analysis when the independent source signals are temporarily correlated estimating functions are used for deriving both batch and on line learning algorithms and they are applicable to blind cases where spatial and temporal probability structures of the sources are unknown most algorithms proposed so far can be analyzed in the framework of estimating functions an admissible class of estimating functions is derived and related efficient on line learning algorithms are introduced we analyze dynamical stability and statistical efficiency of these algorithms different from the independently and identically distributed case the algorithms work even when only the second order moments are used the method of simultaneous diagonalization of cross covariance matrices is also studied from the point of view of estimating functions

pruning using parameter and neuronal metrics in this article we introduce a measure of optimality for architecture selection algorithms for neural networks the distance from the original network to the new network in a metric defined by the probability distributions of all possible networks we derive two pruning algorithms one based on a metric in parameter space and the other based on a metric in neuron space which are closely related to well known architecture selection algorithms such as gobs our framework extends the theoretically range of validity of gobs and therefore can explain results observed in previous experiments in addition we give some computational improvements for these algorithms

minimum volume constrained nonnegative matrix factorization enhanced ability of learning parts nonnegative matrix factorization nmf with minimum volume constraint mvc is exploited in this paper our results show that mvc can actually improve the sparseness of the results of nmf this sparseness is l0 norm oriented and can give desirable results even in very weak sparseness situations thereby leading to the significantly enhanced ability of learning parts of nmf the close relation between nmf sparse nmf and the mvc nmf is discussed first then two algorithms are proposed to solve the mvc nmf model one is called quadratic programming mvc nmf qp mvc nmf which is based on quadratic programming and the other is called negative glow mvc nmf ng mvc nmf because it uses multiplicative updates incorporating natural gradient ingeniously the qp mvc nmf algorithm is quite efficient for small scale problems and the ng mvc nmf algorithm is more suitable for large scale problems simulations show the efficiency and validity of the proposed methods in applications of blind source separation and human face images analysis

efficient universal noiseless source codes although the existence of universal noiseless variable rate codes for the class of discrete stationary ergodic sources has previously been established very few practical universal encoding methods are available efficient implementable universal source coding techniques are discussed in this paper results are presented on source codes for which a small value of the maximum redundancy is achieved with a relatively short block length a constructive proof of the existence of universal noiseless codes for discrete stationary sources is first presented the proof is shown to provide a method for obtaining efficient universal noiseless variable rate codes for various classes of sources for memoryless sources upper and lower bounds are obtained for the minimax redundancy as a function of the block length of the code several techniques for constructing universal noiseless source codes for memoryless sources are presented and their redundancies are compared with the bounds consideration is given to possible applications to data compression for certain nonstationary sources

on the p point uncertainty class corresp the p point uncertainty class also known as sakrison s class b is considered although the p point class is not a 2 alternating choquet capacity class it is shown how many of the robust signal processing and communication theory results which have been developed for capacity classes can be applied to the p point class by embedding it in a slightly larger extended p point class

the relation between information theory and the differential geometry approach to statistics abstract cencov has shown that the riemannian metric on the probability simplex xi 1 defined by ds 2 dx i 2 x i has an invariance property under certain probabilistically natural mappings no other riemannian metric has the same property the geometry associated with this metric is shown to lead almost automatically to measures of divergence between probability distributions which are associated with kullback bhattacharyya and matusita certain vector fields are associated in a natural way with random variables the integral curves of these vector fields yield the maximum entropy or minimum divergence estimates of probabilities some other consequences of this geometric view are also explored

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

online model selection based on the variational bayes the bayesian framework provides a principled way of model selection this framework estimates a probability distribution over an ensemble of models and the prediction is done by averaging over the ensemble of models accordingly the uncertainty of the models is taken into account and complex models with more degrees of freedom are penalized however integration over model parameters is often intractable and some approximation scheme is needed recently a powerful approximation scheme called the variational bayes vb method has been proposed this approach defines the free energy for a trial probability distribution which approximates a joint posterior probability distribution over model parameters and hidden variables the exact maximization of the free energy gives the true posterior distribution the vb method uses factorized trial distributions the integration over model parameters can be done analytically and an iterative expectation maximization like algorithm whose convergence is guaranteed is derived in this article we derive an online version of the vb algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy by combining sequential model selection procedures the online vb method provides a fully online learning method with a model selection mechanism in preliminary experiments using synthetic data the online vb method was able to adapt the model structure to dynamic environments