nils-herrmann/clean_scirepeval huggingface dataset:
Self-Organising Neural Networks: Independent Component Analysis and Blind Source Separation

From the Publisher: 
This volume presents the theory and applications of self-organising neural network models which perform the Independent Component Analysis (ICA) transformation and Blind Source Separation (BSS). It is largely self-contained, covering the fundamental concepts of information theory, higher order statistics and information geometry. Neural models for instantaneous and temporal BSS and their adaptation algorithms are presented and studied in detail.. "This volume will be of interest to postgraduate students and researchers in connections AI, signal processing and neural networks, as well as research and development workers, technology development engineers and research engineers.
Blind source separation of more sources than mixtures using overcomplete representations

Empirical results were obtained for the blind source separation of more sources than mixtures using a previously proposed framework for learning overcomplete representations. This technique assumes a linear mixing model with additive noise and involves two steps: (1) learning an overcomplete representation for the observed data and (2) inferring sources given a sparse prior on the coefficients. We demonstrate that three speech signals can be separated with good fidelity given only two mixtures of the three signals. Similar results were obtained with mixtures of two speech signals and one music signal.
Self-Organising Neural Networks

This chapter focuses on self-organising neural networks with linear and non-linear activation functions. Many researchers have studied the properties of linear networks and their ability to extract or transfer information regarding the statistics of the observed data. The natural progression to non-linear networks requires alternative analysis tools due to their additional complexity. The emergent behaviour of non-linear networks is much richer than their linear counterparts. A brief review of linear and non-linear self-organising networks within the context of source separation is presented here.
A Unifying Information-theoretic Framework forIndependent Component

We show that diierent theories recently proposed for Independent Component Analysis (ICA) lead to the same iterative learning algorithm for blind separation of mixed independent sources. We review those theories and suggest that information theory can be used to unify several lines of research. Pearlmutter and Parra (1996) and Cardoso (1997) showed that the infomax approach of Bell and Sejnowski (1995) and the maximum likelihood estimation approach are equivalent. We show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a xed nonlinearity. Girolami and Fyfe (1997a) have shown that the nonlinear Principal Component Analysis (PCA) algorithm of Karhunen and Joutsensalo (1994) and Oja (1997) can also be viewed from information-theoretic principles since it minimizes the sum of squares of the fourth-order marginal cumulants and therefore approximately minimizes the mutual information (Comon, 1994). Lambert (1996) has proposed diierent Bussgang cost functions for multichannel blind deconvolution. We show how the Bussgang property relates to the infomax principle. Finally, we discuss convergence and stability as well as future research issues in blind source separation.
The Pearson Mixture Model for Cluster Analysis and Data Visualisation

General learning algorithms based on finite mixture density models have recently been developed and applied to independent component analysis and the blind source separation of linear mixtures. This paper proposes the finite mixture density originally proposed by Pearson as a simple parametric model to be used within these algorithms for clustering and visualising data. The various forms of the resulting algorithms, which utilise this mixture model, bring fresh insights to the nature of the robust principal component analysis and extended infomax independent component analysis algorithms.
A common neural-network model for unsupervised exploratory data analysis and independent component analysis

This paper presents the derivation of an unsupervised learning algorithm, which enables the identification and visualization of latent structure within ensembles of high-dimensional data. This provides a linear projection of the data onto a lower dimensional subspace to identify the characteristic structure of the observations independent latent causes. The algorithm is shown to be a very promising tool for unsupervised exploratory data analysis and data visualization. Experimental results confirm the attractiveness of this technique for exploratory data analysis and an empirical comparison is made with the recently proposed generative topographic mapping (GTM) and standard principal component analysis (PCA). Based on standard probability density models a generic nonlinearity is developed which allows both 1) identification and visualization of dichotomised clusters inherent in the observed data and 2) separation of sources with arbitrary distributions from mixtures, whose dimensionality may be greater than that of number of sources. The resulting algorithm is therefore also a generalized neural approach to independent component analysis (ICA) and it is considered to be a promising method for analysis of real-world data that will consist of sub- and super-Gaussian components such as biomedical signals.
Noise reduction and speech enhancement via temporal anti-Hebbian learning

Temporal extensions of both linear and nonlinear anti-Hebbian learning have been shown to be suited to the problem of blind separation of sources from their convolved mixtures. This paper presents a generalized form of anti-Hebbian learning for a partially connected recurrent network based on the maximum likelihood estimation principle. Inspired by features of the binaural unmasking effect the network and associated online adaptation are applied to the enhancement of speech, which is corrupted by interfering noise, competing speech and reverberation. Graded simulations based on speech corrupted with increasingly complex levels of reverberation are reported. It is shown that for high levels of reverberation the proposed method compares favorably with classical adaptive filter approaches to speech enhancement in real acoustic environments.
An Alternative Perspective on Adaptive Independent Component Analysis Algorithms

This article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources. The gaussian mixture model of Pearson is employed in deriving a closed-form generic score function for strictly subgaussian sources. This is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources.
Generalised independent component analysis through unsupervised learning with emergent Bussgang properties

We utilise an information theoretic criterion for exploratory projection pursuit (EPP) and have shown that maximisation by natural gradient ascent of the divergence of a multivariate distribution from normality, using the negentropy as a distance measure, yields a generalised independent component analysis (ICA). By considering a Gram-Charlier approximation of the latent probability density functions (PDF) we develop a generalised neuron nonlinearity which can be considered as a conditional mean estimator of the underlying independent components. The unsupervised learning rule developed is shown to asymptotically exhibit the Bussgang property and as such produces output data with independent components, irrespective of whether the independent latent variables are sub-gaussian or super-gaussian. Improved convergence speeds are reported when momentum terms are introduced into the learning.
Kurtosis extrema and identification of independent components: a neural network approach

We propose a nonlinear self-organising network which solely employs computationally simple Hebbian and anti-Hebbian learning in approximating a linear independent component analysis (ICA). Current neural architectures and algorithms which perform parallel ICA are either restricted to positively kurtotic data distributions or data which exhibits one sign of kurtosis . We show that the proposed network is capable of separating mixtures of speech, noise and signals with both platykurtic (positive kurtosis) and leptokurtic (negative kurtosis) distributions in a blind manner. A simulation is reported which successfully separates a mixture of twenty sources of music, speech, noise and fundamental frequencies.
Principal Components Identify MLP Hidden Layer Size for Optimal Generalisation Performance

One of the major concerns when implementing a supervised artificial neural network solution to a classification or prediction problem, is the network’s performance on unseen data. The phenomenon of the network overfitting the training data, is understood and reported in the literature. Most researchers recommend a ‘trial and error’ approach to selecting the optimal number of weights for the network, which is time consuming, or start with a large network and prune to an optimal size. Current pruning techniques based on approximations of the Hessian matrix of the error surface are computationally intensive and prone to severe approximation errors if a suitable minimal training error has not been achieved. We propose a novel and simple design heuristic for a three layer multi-layer perceptron (MLP) based on an eigenvalue decomposition of the covariance matrix of the middle layer output. This technique identifies the neurons which are contributing to the redundancy of data through the network and as such are additional effective network parameters which have a deleterious effect on the classifier surface smoothness. This technique identifies redundancy in the network data and so is not dependant on the network training having reached a minimal error value making the Levenberg-Marquardt approximation valid. We report on simulations using the double-convex benchmark which show the utility of the proposed method.
Fahlman-Type Activation Functions Applied to Nonlinear PCA Networks Provide a Generalised Independent Component Analysis

It has been shown experimentally that Oja’s nonlinear principal component analysis (PCA) algorithm is capable of performing an independent component analysis (ICA) on a specific data set . However, the dynamic stability requirements of the nonlinear PCA algorithm restrict its use to data which has sub-gaussian probability densities [6]. The restriction is particularly severe as this precludes the application of the algorithm from performing ICA on naturally occurring data such as speech, music and certain visual images. We have shown that the nonlinear PCA algorithm can be considered as minimising an information theoretic contrast function and develop a more direct link between ICA and the algorithm function . To remove the sub-gaussian restriction and enable a generalised ICA which will span the full range of possible data kurtosis, we propose the use of Fahlman type activation functions  in the nonlinear PCA algorithm. We show that variants of these functions satisfy all the dynamic and asymptotic stability requirements of the algorithm and successfully remove the sub-gaussian restriction. We also report on simulations which demonstrate the blind separating ability of the nonlinear PCA algorithm with the Fahlman type functions on mixtures of super-Gaussian data (natural speech).
Stochastic ICA Contrast Maximisation Using Oja's Nonlinear PCA Algorithm

Independent Component Analysis (ICA) is an important extension of linear Principal Component Analysis (PCA). PCA performs a data transformation to provide independence to second order, that is, decorrelation. ICA transforms data to provide approximate independence up to and beyond second order yielding transformed data with fully factorable probability densities. The linear ICA transformation has been applied to the classical statistical signal-processing problem of Blind Separation of Sources (BSS), that is, separating unknown original source signals from a mixture whose mode of mixing is undetermined. In this paper it is shown that Oja's Nonlinear PCA algorithm performs a general stochastic online adaptive ICA. This analysis is corroborated with three simulations. The first separates unknown mixtures of original natural images, which have sub-Gaussian densities, the second separates linear mixtures of natural speech whose densities are super-Gaussian. Finally unknown mixtures of original images, which have both sub- and super-Gaussian densities are separated.
Tracking Independent Sources

In this paper, we review an Artiicial Neural Network we have previously used for extraction of independent signals from a mixture of signals. The network, called the Extended Exploratory Projection Pursuit Network, is shown to be superior to Exploratory Projection Pursuit because of its ability to track time-dependence in its inputs. We give results on artiicial and real data. One of the most interesting problems to which Arti-cial Neural Networks is being applied is the blind separation of independent sources from a (linear) mixture of sources 2, 9]. The problem is blind in that we assume no prior information about either the mixing matrix or the sources other than that the sources should be independent and at most one should be drawn from a Gaussian distribution. have been mixed linearly to get a set of values (in practice, liable to be sensor readings), x 1 ; x 2 ; :::; x n using a mixing matrix A. x = As (1) We assume no prior knowledge of the input signals nor of the mixing matrix A. However, the unknown signals, s i must be independent from each other and at most one of them may be drawn from a Gaussian distribution. The task is to nd an un-mixing matrix W such that y = Wx (2) is such that the individual elements of y are the original signals (up to a permutation and scaling factor). Some of the methods used to solve this problem have been based on the statistical method of Exploratory Projection Pursuit (EPP). We have previously used an EPP network ((5]) to search for lters which identify the independent sources but have found better results with an extension of the network which we have called an Extended Exploratory Projection Pursuit (EEPP) network((8]). We have, however, continued to justify our method by referring to the theoretical results of the EPP network, though this leaves open the question as to why the EEPP network should perform better than the EPP network which has been analytically derived for this exact purpose. In this paper, we show that the EEPP network has superior capabilities in tracking the locally time-dependent statistics of the data. We will use artiicial data to make an empirical investigation of the separation properties. Our arti-cial data will be drawn from a number of diierent time-dependent distributions and we will conclude with results from real speech data. 2 …
Extraction of independent signal sources using a deflationary exploratory projection pursuit network

A nonlinear self-organising neural network is proposed, which employs hierarchic linear negative feedback, and this network is applied to the blind separation of independent source signals from their mixtures. Blind separation of sources has become an important area of research, with significant contributions recently being made from both the statistical signal processing and artificial neural network research communities. A nonlinear extension of a negative feedback network is developed and it is shown that hierarchic linear feedback provides a deflation of the network residuals, which are employed in the Hebbian learning of the network. As each of the output neuron weights converge to a separating vector, then the weighted feedback will remove the contribution of the extracted source from the remaining residual mixture. It is shown that the data driven self-organisation of the proposed network using only Hebbian and anti-Hebbian learning will extract the underlying source signals from the received mixture. The results of a simulation are reported, which demonstrates the ability of the network in restoring images after degradation with noise and interfering images.
Negentropy and Kurtosis as Projection Pursuit Indices Provide Generalised ICA Algorithms

We develop a generalised form of the independent component analysis (ICA) algorithm introduced by Bell and Sejnowski , Amari et al  and lately by Pearlmutter and Parra  and also MacKay . Motivated by information theoretic indices for exploratory projection pursuit (EPP) we show that maximisation by natural gradient ascent of the divergence of a multivariate distribution from normality, using the negentropy as a distance measure, yields a generalised ICA. We introduce a form of nonlinearity which has an inherently simple form and exhibits the Bussgang property [30] within the algorithm. We show that this is sufficient to perform ICA on data which has latent variables exhibiting either unimodal or bimodal probability density functions (PDF) or both. Kurtosis has been used as a moment based projection pursuit index and as a contrast for ICA [5, 6, 7]. We introduce a simple adaptive nonlinearity which is formed by on-line estimation of the latent variable kurtosis and demonstrate the removal of the standard ICA constraint of latent variable pdf modality uniformity.
STABILITY RESULTS IN LEARNING THEORY

The problem of proving generalization bounds for the performance of learning algorithms can be formulated as a problem of bounding the bias and variance of estimators of the expected error. We show how various stability assumptions can be employed for this purpose. We provide a necessary and sufficient stability condition for bounding the bias and variance for the Empirical Risk Minimization algorithm, and various sufficient conditions for bounding bias and variance of estimators for general algorithms. We discuss settings in which it is possible to obtain exponential bounds, and we prove an extension of the bounded-difference inequality for "almost always" stable algorithms.
Risk bounds for mixture density estimation

In this paper we focus on the problem of estimating a bounded density using a finite combination of densities from a given class. We consider the Maximum Likelihood Estimator (MLE) and the greedy procedure described by Li and Barron (1999) under the additional assumption of boundedness of densities. We prove an O( 1 √ n ) bound on the estimation error which does not depend on the number of densities in the estimated combination. Under the boundedness assumption, this improves the bound of Li and Barron by removing the logn factor and also generalizes it to the base classes with converging Dudley integral.
Some Properties of Empirical Risk Minimization Over Donsker Classes

We study properties of algorithms which minimize (or almost-minimise) empirical error over a Donsker class of functions. we show that the L2-diameter of the set of almost-minimizers is converging to zero in probability. Therefore, as the number of samples grows, it is becoming unlikely that adding a point (or a number of points) to the training set will result in a large jump (in L2 distance) to a new hypothesis. We also show that under some conditions the expected errors of the almost-minimizers are becoming close with a rate faster than pi(exp -1/2).
Applications of Empirical Processes in Learning Theory: Algorithmic Stability and Performance Guarantees

We study two key properties of learning algorithms: their generalization ability and their stability with respect to perturbations. To analyze these properties, we focus on concentration inequalities and tools from empirical process theory. We obtain theoretical results and demonstrate their applications to machine learning. First, we demonstrate how the Contraction Lemma for Rademacher averages can be used to obtain tight performance guarantees for learning methods . In particular, we derive risk bounds for a greedy mixture density estimation procedure. We prove that, unlike what is suggested in the literature, the number of terms in the mixture is not a bias-variance trade-off for the performance. Our upper bound asserts that the estimation error decreases as the number of terms in the mixture increases. Next, we study stability properties of general learning algorithms . We show how various stabilities upper- and lower-bound the bias and variance of several estimators of the expected performance. Furthermore, we prove a generalization of the bounded-difference concentration inequality of McDiarmid for almost-everywhere-smooth functions. Experiments are performed to ascertain the utility of stability measures in practical applications. We provide a solution  to an open problem regarding the stability of a specific learning algorithm called Empirical Risk Minimization (ERM). This algorithm is of central importance in Learning Theory. By studying the suprema of the empirical process, we prove that ERM over Donsker classes of functions is stable in the L1 norm. Hence, as the number of samples grows, it becomes less and less likely that a perturbation of o( p n) samples will result in a very different empirical minimizer. Asymptotic rates of this stability are proved under metric entropy assumptions on the function class. Through the use of a ratio limit inequality, we also prove stability of expected errors of empirical minimizers. Finally, we investigate applications of the above stability result. In particular, we focus on procedures that optimize an objective function, such as k-means and other clustering methods. We demonstrate that stability of clustering is closely related to the notion of “realizability” in Learning Theory. Furthermore, our result on stability of ERM delineates a phase transition between stability and instability of clustering methods. Stability is the most widely used empirical quality measure for clustering, and our results make progress towards building a theoretical foundation for clustering methods.
Risk Bounds for Density Estimation

Li and Barron [6, 7] give a greedy method for estimating densities by a finite mixture. The trade-off between approximation and estimation error is present in their risk bound in terms of the optimal number of components used for density estimation. Using techniques from Empirical Process theory [11] and results from geometry in Banach Spaces , we improve the bounds and show that, in fact, the estimation bound does not depend on the number of components.
Bagging Regularizes

Intuitively, we expect that averaging — or bagging — different regressors with low correlation should smooth their behavior and be somewhat similar to regularization. In this note we make this intuition precise. Using an almost classical definition of stability, we prove that a certain form of averaging provides generalization bounds with a rate of convergence of the same order as Tikhonov regularization — similar to fashionable RKHSbased learning algorithms. This report describes research done within the Center for Biological & Computational Learning which is part of the McGovern Institute, the Department of Brain & Cognitive Sciences and the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. This research was sponsored by grants from: Office of Naval Research (DARPA) under contract No. N00014-001-0907, National Science Foundation (ITR) under contract No. IIS-0085836, National Science Foundation (KDI) under contract No. DMS-9872936, and National Science Foundation under contract No. IIS-9800032 Additional support was provided by: Central Research Institute of Electric Power Industry, Eastman Kodak Company, DaimlerChrysler AG, Compaq, Honda R&D Co., Ltd., Komatsu Ltd., NEC Fund, Siemens Corporate Research, Inc., and The Whitaker Foundation.