one bit matching theorem for ica convex concave programming on polyhedral set and distribution approximation for combinatorics according to the proof by liu chiu and xu 2004 on the so called one bit matching conjecture xu cheung and amari 1998a all the sources can be separated as long as there is an one to one same sign correspondence between the kurtosis signs of all source probability density functions pdf s and the kurtosis signs of all model pdf s which is widely believed and implicitly supported by many empirical studies however this proof is made only in a weak sense that the conjecture is true when the global optimal solution of an independent component analysis criterion is reached thus it cannot support the successes of many existing iterative algorithms that usually converge at one of the local optimal solutions this article presents a new mathematical proof that is obtained in a strong sense that the conjecture is also true when any one of local optimal solutions is reached in helping to investigating convex concave programming on a polyhedral set theorems are also provided not only on partial separation of sources when there is a partial matching between the kurtosis signs but also on an interesting duality of maximization and minimization on source separation moreover corollaries are obtained on an interesting duality with supergaussian sources separated by maximization and subgaussian sources separated by minimization also a corollary is obtained to confirm the symmetric orthogonalization implementation of the kurtosis extreme approach for separating multiple sources in parallel which works empirically but lacks mathematical proof furthermore a linkage has been set up to combinatorial optimization from a distribution approximation perspective and a stiefel manifold perspective with algorithms that guarantee convergence as well as satisfaction of constraints

an alternative perspective on adaptive independent component analysis algorithms this article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources the gaussian mixture model of pearson is employed in deriving a closedform generic score function for strictly subgaussian sources this is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources

independent component analysis a flexible nonlinearity and decorrelating manifold approach independent component analysis ica finds a linear transformation to variables that are maximally statistically independent we examine ica and algorithms for finding the best transformation from the point of view of maximizing the likelihood of the data in particular we discuss the way in which scaling of the unmixing matrix permits a static nonlinearity to adapt to various marginal densities we demonstrate a new algorithm that uses generalized exponential functions to model the marginal densities and is able to separate densities with light tails we characterize the manifold of decorrelating matrices and show that it lies along the ridges of high likelihood unmixing matrices in the space of all unmixing matrices we show how to find the optimum ica matrix on the manifold of decorrelating matrices and as an example we use the algorithm to find independent component basis vectors for an ensemble of portraits

one bit matching conjecture for independent component analysis the one bit matching conjecture for independent component analysis ica could be understood from different perspectives but is basically stated as all the sources can be separated as long as there is a one to one same sign correspondence between the kurtosis signs of all source probability density functions pdf s and the kurtosis signs of all model pdf s xu cheung amari 1998a this conjecture has been widely believed in the ica community and implicitly supported by many ica studies such as the extended infomax lee girolami sejnowski 1999 and the soft switching algorithm welling weber 2001 however there is no mathematical proof to confirm the conjecture theoretically in this article only skewness and kurtosis are considered and such a mathematical proof is given under the assumption that the skewness of the model densities vanishes moreover empirical experiments are demonstrated on the robustness of the conjecture as the vanishing skewness assumption breaks as a by product we also show that the kurtosis maximization criterion moreau macchi 1996 is actually a special case of the minimum mutual information criterion for ica

a constrained em algorithm for independent component analysis we introduce a novel way of performing independent component analysis using a constrained version of the expectation maximization em algorithm the source distributions are modeled as d one dimensional mixtures of gaussians the observed data are modeled as linear mixtures of the sources with additive isotropic noise this generative model is fit to the data using constrained em the simpler soft switching approach is introduced which uses only one parameter to decide on the sub or supergaussian nature of the sources we explain how our approach relates to independent factor analysis

stochastic proximal gradient consensus over random networks we consider solving a convex optimization problem with possibly stochastic gradient and over a randomly time varying multiagent network each agent has access to some local objective function and it only has unbiased estimates of the gradients of the smooth component we develop a dynamic stochastic proximal gradient consensus algorithm with the following key features 1 it works for both the static and certain randomly time varying networks 2 it allows the agents to utilize either the exact or stochastic gradient information 3 it is convergent with provable rate in particular the proposed algorithm converges to a global optimal solution with a rate of mathcal o 1 r resp mathcal o 1 sqrt r when the exact resp stochastic gradient is available where r is the iteration counter interestingly the developed algorithm establishes a close connection among a number of seemingly unrelated distributed algorithms such as the extra the pg extra the ic idc admm the dlm and the classical distributed subgradient method

fast nonlinear autocorrelation algorithm for source separation independent component analysis ica and blind source separation bss methods have been used for pattern recognition problems it is well known that ica and bss depend on the statistical properties of original sources or components such as non gaussianity in the paper using a statistical property nonlinear autocorrelation and maximizing the nonlinear autocorrelation of source signals we propose a fast fixed point algorithm for bss we study its convergence property and show that its convergence speed is at least quadratic simulations by the artificial signals and the real world applications verify the efficient implementation of the proposed method

waveform preserving blind estimation of multiple independent sources the problem of blind estimation of source signals is to estimate the source signals without knowing the characteristics of the transmission channel it is shown that the minimum variance unbiased estimates can be obtained if and only if the transmission channel can be identified blindly it is shown that the channel can be blindly identified if and only if there is not more than one gaussian source this condition suggests that waveform preserving blind estimation can be achieved over a wide range of signal processing applications including those cases in which the source signals have identical nongaussian distributions the constructive proof of the necessary and sufficient condition serves as a foundation for the development of waveform preserving blind signal estimation algorithms examples are presented to demonstrate the applications of the theoretical results

a further result on the ica one bit matching conjecture the one bit matching conjecture for independent component analysis ica has been widely believed in the ica community theoretically it has been proved that under the assumption of zero skewness for the model probability density functions the global maximum of a cost function derived from the typical objective function on the ica problem with the one bit matching condition corresponds to a feasible solution of the ica problem in this note we further prove that all the local maximums of the cost function correspond to the feasible solutions of the ica problem in the two source case under the same assumption that is as long as the one bit matching condition is satisfied the two source ica problem can be successfully solved using any local descent algorithm of the typical objective function with the assumption of zero skewness for all the model probability density functions

dlm decentralized linearized alternating direction method of multipliers this paper develops the decentralized linearized alternating direction method of multipliers dlm that minimizes a sum of local cost functions in a multiagent network the algorithm mimics operation of the decentralized alternating direction method of multipliers dadmm except that it linearizes the optimization objective at each iteration this results in iterations that instead of successive minimizations implement steps whose cost is akin to the much lower cost of the gradient descent step used in the distributed gradient method dgm the algorithm is proven to converge to the optimal solution when the local cost functions have lipschitz continuous gradients its rate of convergence is shown to be linear if the local cost functions are further assumed to be strongly convex numerical experiments in least squares and logistic regression problems show that the number of iterations to achieve equivalent optimality gaps are similar for dlm and admm and both much smaller than those of dgm in that sense dlm combines the rapid convergence of admm with the low computational burden of dgm

collision superoperator in the subdynamics approach of inhomogeneous systems abstract in order to obtain the dynamical description of irreversible processes in large inhomogeneous systems the decomposition of the dynamics into various subdynamics is used each subdynamics is governed by an irreversible equation the formal solution corresponding to the contributions of the diagonal diagrams is entirely expressed by means of the collision superoperator a general compact formula for the collision superoperator is obtained

linear convergence rate of a class of distributed augmented lagrangian algorithms we study distributed optimization where nodes cooperatively minimize the sum of their individual locally known convex costs f i x s x in bbr d is global distributed augmented lagrangian al methods have good empirical performance on several signal processing and learning applications but there is limited understanding of their convergence rates and how it depends on the underlying network this paper establishes globally linear geometric convergence rates of a class of deterministic and randomized distributed al methods when the f i s are twice continuously differentiable and have a bounded hessian we give explicit dependence of the convergence rates on the underlying network parameters simulations illustrate our analytical findings

the fastica algorithm revisited convergence analysis the fast independent component analysis fastica algorithm is one of the most popular methods to solve problems in ica and blind source separation it has been shown experimentally that it outperforms most of the commonly used ica algorithms in convergence speed a rigorous local convergence analysis has been presented only for the so called one unit case in which just one of the rows of the separating matrix is considered however in the fastica algorithm there is also an explicit normalization step and it may be questioned whether the extra rotation caused by the normalization will affect the convergence speed the purpose of this paper is to show that this is not the case and the good convergence properties of the one unit case are also shared by the full algorithm with symmetrical normalization a local convergence analysis is given for the general case and the global behavior is illustrated numerically for two sources and two mixtures in several typical cases

fast consensus by the alternating direction multipliers method the alternating direction multipliers method admm has been recently proposed as a practical and efficient algorithm for distributed computing we discuss its applicability to the average consensus problem in this paper by carefully relaxing admm augmentation coefficients we are able to analytically investigate its properties and to propose simple and strict analytical bounds these provide a clear indication on how to choose system parameters for optimized performance we prove both analytically and via simulations that the proposed approach exhibits convergence speed between the best in the literature classical and optimized solutions while providing the most powerful resilience to noise

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

extra an exact first order algorithm for decentralized consensus optimization recently there has been growing interest in solving consensus optimization problems in a multiagent network in this paper we develop a decentralized algorithm for the consensus optimization problem mathrm minimize x in mathbb r p bar f x frac 1 n sum i 1 n f i x which is defined over a connected network of n agents where each function f i is held privately by agent i and encodes the agent s data and objective all the agents shall collaboratively find the minimizer while each agent can only communicate with its neighbors such a computation scheme avoids a data fusion center or long distance communication and offers better load balance to the network this paper proposes a novel decentralized exact first order algorithm abbreviated as extra to solve the consensus optimization problem exact means that it can converge to the exact solution extra uses a fixed large step size which can be determined independently of the network size or topology the local variable of every a

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

consensus in ad hoc wsns with noisy links part i distributed estimation of deterministic signals we deal with distributed estimation of deterministic vector parameters using ad hoc wireless sensor networks wsns we cast the decentralized estimation problem as the solution of multiple constrained convex optimization subproblems using the method of multipliers in conjunction with a block coordinate descent approach we demonstrate how the resultant algorithm can be decomposed into a set of simpler tasks suitable for distributed implementation different from existing alternatives our approach does not require the centralized estimator to be expressible in a separable closed form in terms of averages thus allowing for decentralized computation even of nonlinear estimators including maximum likelihood estimators mle in nonlinear and non gaussian data models we prove that these algorithms have guaranteed convergence to the desired estimator when the sensor links are assumed ideal furthermore our decentralized algorithms exhibit resilience in the presence of receiver and or quantization noise in particular we introduce a decentralized scheme for least squares and best linear unbiased estimation blue and establish its convergence in the presence of communication noise our algorithms also exhibit potential for higher convergence rate with respect to existing schemes corroborating simulations demonstrate the merits of the novel distributed estimation algorithms

high order contrasts for independent component analysis this article considers high order measures of independence for the independent component analysis problem and discusses the class of jacobi algorithms for their optimization several implementations are discussed we compare the proposed approaches with gradient based techniques from the algorithmic point of view and also on a set of biomedical data