a quantitative study of fault tolerance noise immunity and generalization ability of mlps an analysis of the influence of weight and input perturbations in a multilayer perceptron mlp is made in this article quantitative measurements of fault tolerance noise immunity and generalization ability are provided from the expressions obtained it is possible to justify some previously reported conjectures and experimentally obtained results e g the influence of weight magnitudes the relation between training with noise and the generalization ability the relation between fault tolerance and the generalization ability the measurements introduced here are explicitly related to the mean squared error degradation in the presence of perturbations thus constituting a selection criterion between different alternatives of weight configurations moreover they allow us to predict the degradation of the learning performance of an mlp when its weights or inputs are deviated from their nominal values and thus the behavior of a physical implementation can be evaluated before the weights are mapped on it according to its accuracy

asymptotic statistical theory of overtraining and cross validation a statistical theory for overtraining is proposed the analysis treats general realizable stochastic neural networks trained with kullback leibler divergence in the asymptotic case of a large number of training examples it is shown that the asymptotic gain in the generalization error is small if we perform early stopping even if we have access to the optimal stopping time based on the cross validation stopping we consider the ratio the examples should be divided into training and cross validation sets in order to obtain the optimum performance although cross validated early stopping is useless in the asymptotic region it surely decreases the generalization error in the nonasymptotic region our large scale simulations done on a cm5 are in good agreement with our analytical findings

extending stochastic resonance for neuron models to general levy noise a recent paper by patel and kosko 2008 demonstrated stochastic resonance sr for general feedback continuous and spiking neuron models using additive levy noise constrained to have finite second moments in this brief we drop this constraint and show that their result extends to general levy noise models we achieve this by showing that large jump discontinuities in the noise can be controlled so as to allow the stochastic model to tend to a deterministic one as the noise dissipates to zero sr then follows by a forbidden intervals theorem as in patel and kosko s paper

convergence analyses on on line weight noise injection based training algorithms for mlps injecting weight noise during training is a simple technique that has been proposed for almost two decades however little is known about its convergence behavior this paper studies the convergence of two weight noise injection based training algorithms multiplicative weight noise injection with weight decay and additive weight noise injection with weight decay we consider that they are applied to multilayer perceptrons either with linear or sigmoid output nodes let w t be the weight vector let v w be the corresponding objective function of the training algorithm let 0 be the weight decay constant and let t be the step size we show that if t 0 then with probability one e w t 22 is bound and limt w t 2 exists based on these two properties we show that if t 0 t t and t t 2 then with probability one these algorithms converge moreover w t converges with probability one to a point where wv w 0

on objective function regularizer and prediction error of a learning algorithm for dealing with multiplicative weight noise in this paper an objective function for training a functional link network to tolerate multiplicative weight noise is presented basically the objective function is similar in form to other regularizer based functions that consist of a mean square training error term and a regularizer term our study shows that under some mild conditions the derived regularizer is essentially the same as a weight decay regularizer this explains why applying weight decay can also improve the fault tolerant ability of a radial basis function rbf with multiplicative weight noise in accordance with the objective function a simple learning algorithm for a functional link network with multiplicative weight noise is derived finally the mean prediction error of the trained network is analyzed simulated experiments on two artificial data sets and a real world application are performed to verify theoretical result

efficient algorithm for training interpolation rbf networks with equally spaced nodes this brief paper proposes a new algorithm to train interpolation gaussian radial basis function rbf networks in order to solve the problem of interpolating multivariate functions with equally spaced nodes based on an efficient two phase algorithm recently proposed by the authors euclidean norm associated to gaussian rbf is now replaced by a conveniently chosen mahalanobis norm that allows for directly computing the width parameters of gaussian radial basis functions the weighting parameters are then determined by a simple iterative method the original two phase algorithm becomes a one phase one simulation results show that the generality of networks trained by this new algorithm is sensibly improved and the running time significantly reduced especially when the number of nodes is large

lambertian reflectance and linear subspaces we prove that the set of all lambertian reflectance functions the mapping from surface normals to intensities obtained with arbitrary distant light sources lies close to a 9d linear subspace this implies that in general the set of images of a convex lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low dimensional linear subspace explaining prior empirical results we also provide a simple analytic characterization of this linear space we obtain these results by representing lighting using spherical harmonics and describing the effects of lambertian materials as the analog of a convolution these results allow us to construct algorithms for object recognition based on linear methods as well as algorithms that use convex optimization to enforce nonnegative lighting functions we also show a simple way to enforce nonnegative lighting when the images of an object lie near a 4d linear space we apply these algorithms to perform face recognition by finding the 3d model that best matches a 2d query image

robust subspace clustering via thresholding the problem of clustering noisy and incompletely observed high dimensional data points into a union of low dimensional subspaces and a set of outliers is considered the number of subspaces their dimensions and their orientations are assumed unknown we propose a simple low complexity subspace clustering algorithm which applies spectral clustering to an adjacency matrix obtained by thresholding the correlations between data points in other words the adjacency matrix is constructed from the nearest neighbors of each data point in spherical distance a statistical performance analysis shows that the algorithm exhibits robustness to additive noise and succeeds even when the subspaces intersect specifically our results reveal an explicit tradeoff between the affinity of the subspaces and the tolerable noise level we furthermore prove that the algorithm succeeds even when the data points are incompletely observed with the number of missing entries allowed to be up to a log factor linear in the ambient dimension we also propose a simple scheme that provably detects outliers and we present numerical results on real and synthetic data

a geometric analysis of subspace clustering with outliers this paper considers the problem of clustering a collection of unlabeled data points assumed to lie near a union of lower dimensional planes as is common in computer vision or unsupervised learning applications we do not know in advance how many subspaces there are nor do we have any information about their dimensions we develop a novel geometric analysis of an algorithm named sparse subspace clustering ssc 11 which signicantly broadens the range of problems where it is provably eective for instance we show that ssc can recover multiple subspaces each of dimension comparable to the ambient dimension we also prove that ssc can correctly cluster data points even when the subspaces of interest intersect further we develop an extension of ssc that succeeds when the data set is corrupted with possibly overwhelmingly many outliers underlying our analysis are clear geometric insights which may bear on other sparse recovery problems a numerical study complements our theoretical analysis and demonstrates the eectiveness of these methods

estimations of error bounds for neural network function approximators neural networks are being increasingly used for problems involving function approximation however a key limitation of neural methods is the lack of a measure of how much confidence can be placed in output estimates in the last few years many authors have addressed this shortcoming from various angles focusing primarily on predicting output bounds as a function of the trained network s characteristics typically as defined by the hessian matrix in this paper the problem of the effect of errors or noise in the presented input vector is examined and a method based on perturbation analysis of determining output bounds from the error in the input vector and the imperfections in the weight values after training is also presented and demonstrated

robust recovery of multiple subspaces by geometric lp minimization we assume i i d data sampled from a mixture distribution with k components along fixed d dimensional linear subspaces and an additional outlier component for p 0 we study the simultaneous recovery of the k fixed subspaces by minimizing the lp averaged distances of the sampled data points from any k subspaces under some conditions we show that if 0 1 and p 1 then the underlying subspaces cannot be recovered or even nearly recovered by lp minimization the results of this paper partially explain the successes and failures of the basic approach of lp energy minimization for modeling data by multiple subspaces

coherence pursuit fast simple and robust principal component analysis this paper presents a remarkably simple yet powerful algorithm termed coherence pursuit cop to robust principal component analysis pca as inliers lie in a low dimensional subspace and are mostly correlated an inlier is likely to have strong mutual coherence with a large number of data points by contrast outliers either do not admit low dimensional structures or form small clusters in either case an outlier is unlikely to bear strong resemblance to a large number of data points given that cop sets an outlier apart from an inlier by comparing their coherence with the rest of the data points the mutual coherences are computed by forming the gram matrix of the normalized data points subsequently the sought subspace is recovered from the span of the subset of the data points that exhibit strong coherence with the rest of the data as cop only involves one simple matrix multiplication it is significantly faster than the state of the art robust pca algorithms we derive analytical performance guarantees for cop under different models for the distributions of inliers and outliers in both noise free and noisy settings cop is the first robust pca algorithm that is simultaneously non iterative provably robust to both unstructured and structured outliers and can tolerate a large number of unstructured outliers

regularization in the selection of radial basis function centers subset selection and regularization are two well known techniques that can improve the generalization performance of nonparametric linear regression estimators such as radial basis function networks this paper examines regularized forward selection rfs a combination of forward subset selection and zero order regularization an efficient implementation of rfs into which either delete 1 or generalized cross validation can be incorporated and a reestimation formula for the regularization parameter are also discussed simulation studies are presented that demonstrate improved generalization performance due to regularization in the forward selection of radial basis function centers

multiscale hybrid linear models for lossy image representation in this paper we introduce a simple and efficient representation for natural images we view an image in either the spatial domain or the wavelet domain as a collection of vectors in a high dimensional space we then fit a piece wise linear model i e a union of affine subspaces to the vectors at each downsampling scale we call this a multiscale hybrid linear model for the image the model can be effectively estimated via a new algebraic method known as generalized principal component analysis gpca the hybrid and hierarchical structure of this model allows us to effectively extract and exploit multimodal correlations among the imagery data at different scales it conceptually and computationally remedies limitations of many existing image representation methods that are based on either a fixed linear transformation e g dct wavelets or an adaptive uni modal linear transformation e g pca or a multimodal model that uses only cluster means e g vq we will justify both quantitatively and experimentally why and how such a simple multiscale hybrid model is able to reduce simultaneously the model complexity and computational cost despite a small overhead of the model our careful and extensive experimental results show that this new model gives more compact representations for a wide variety of natural images under a wide range of signal to noise ratios than many existing methods including wavelets we also briefly address how the same hybrid linear modeling paradigm can be extended to be potentially useful for other applications such as image segmentation

efficient homophonic coding homophonic coding or homophonic substitution is referred to as a technique that contributes to reliability of the secret key cipher systems its main goal is to convert the plaintext into a sequence of completely random equiprobable and independent code letters in solving this problem three characteristics are to be considered i redundancy defined as the difference between the mean codeword length and the source entropy ii an average number of random bits used in encoding and iii complexity of the encoder and decoder measured by memory size in bits and computation time in bit operations a class of homophonic codes is suggested for which both the redundancy and the average number of random bits can be made as small as required with nonexponential growth of memory size and roughly logarithmic growth of computation time

generalized principal component analysis gpca this paper presents an algebro geometric solution to the problem of segmenting an unknown number of subspaces of unknown and varying dimensions from sample data points we represent the subspaces with a set of homogeneous polynomials whose degree is the number of subspaces and whose derivatives at a data point give normal vectors to the subspace passing through the point when the number of subspaces is known we show that these polynomials can be estimated linearly from data hence subspace segmentation is reduced to classifying one point per subspace we select these points optimally from the data set by minimizing certain distance function thus dealing automatically with moderate noise in the data a basis for the complement of each subspace is then recovered by applying standard pca to the collection of derivatives normal vectors extensions of gpca that deal with data in a high dimensional space and with an unknown number of subspaces are also presented our experiments on low dimensional data show that gpca outperforms existing algebraic algorithms based on polynomial factorization and provides a good initialization to iterative techniques such as k subspaces and expectation maximization we also present applications of gpca to computer vision problems such as face clustering temporal video segmentation and 3d motion segmentation from point correspondences in multiple affine views

sensitivity analysis of multilayer perceptron with differentiable activation functions in a neural network many different sets of connection weights can approximately realize an input output mapping the sensitivity of the neural network varies depending on the set of weights for the selection of weights with lower sensitivity or for estimating output perturbations in the implementation it is important to measure the sensitivity for the weights a sensitivity depending on the weight set in a single output multilayer perceptron mlp with differentiable activation functions is proposed formulas are derived to compute the sensitivity arising from additive multiplicative weight perturbations or input perturbations for a specific input pattern the concept of sensitivity is extended so that it can be applied to any input patterns a few sensitivity measures for the multiple output mlp are suggested for the verification of the validity of the proposed sensitivities computer simulations have been performed resulting in good agreement between theoretical and simulation outcomes for small weight perturbations

spectral curvature clustering scc this paper presents novel techniques for improving the performance of a multi way spectral clustering framework govindu in proceedings of the 2005 ieee computer society conference on computer vision and pattern recognition cvpr 05 vol 1 pp 1150 1157 2005 chen and lerman 2007 preprint in the supplementary webpage for segmenting affine subspaces specifically it suggests an iterative sampling procedure to improve the uniform sampling strategy an automatic scheme of inferring the tuning parameter from the data a precise initialization procedure for k means as well as a simple strategy for isolating outliers the resulting algorithm spectral curvature clustering scc requires only linear storage and takes linear running time in the size of the data it is supported by theory which both justifies its successful performance and guides our practical choices we compare it with other existing methods on a few artificial instances of affine subspaces application of the algorithm to several real world problems is also discussed

robust nonparametric nearest neighbor random process clustering we consider the problem of clustering noisy finite length observations of stationary ergodic random processes according to their generative models without prior knowledge of the model statistics and the number of generative models two algorithms both using the l 1 distance between estimated power spectral densities psds as a measure of dissimilarity are analyzed the first one termed nearest neighbor process clustering nnpc relies on partitioning the nearest neighbor graph of the observations via spectral clustering the second algorithm simply referred to as k means consists of a single k means iteration with farthest point initialization and was considered before in the literature albeit with a different dissimilarity measure we prove that both algorithms succeed with high probability in the presence of noise and missing entries and even when the generative process psds overlap significantly all provided that the observation length is sufficiently large our results quantify the tradeoff between the overlap of the generative process psds the observation length the fraction of missing entries and the noise variance finally we provide extensive numerical results for synthetic and real data and find that nnpc outperforms state of the art algorithms in human motion sequence clustering

greedy feature selection for subspace clustering unions of subspaces provide a powerful generalization of single subspace models for collections of high dimensional data however learning multiple subspaces from data is challenging due to the fact that segmentation the identification of points that live in the same subspace and subspace estimation must be performed simultaneously recently sparse recovery methods were shown to provide a provable and robust strategy for exact feature selection efs recovering subsets of points from the ensemble that live in the same subspace in parallel with recent studies of efs with l1 minimization in this paper we develop sufficient conditions for efs with a greedy method for sparse signal recovery known as orthogonal matching pursuit omp following our analysis we provide an empirical study of feature selection strategies for signals living on unions of subspaces and characterize the gap between sparse recovery methods and nearest neighbor nn based approaches in particular we demonstrate that sparse recovery methods provide significant advantages over nn methods and that the gap between the two approaches is particularly pronounced when the sampling of subspaces in the data set is sparse our results suggest that omp may be employed to reliably recover exact feature sets in a number of regimes where nn approaches fail to reveal the subspace membership of points in the ensemble

hybrid linear modeling via local best fit flats we present a simple and fast geometric method for modeling data by a union of affine subspaces the method begins by forming a collection of local best fit affine subspaces i e subspaces approximating the data in local neighborhoods the correct sizes of the local neighborhoods are determined automatically by the jones 2 numbers we prove under certain geometric conditions that our method finds the optimal local neighborhoods the collection of subspaces is further processed by a greedy selection procedure or a spectral method to generate the final model we discuss applications to tracking based motion segmentation and clustering of faces under different illuminating conditions we give extensive experimental evidence demonstrating the state of the art accuracy and speed of the suggested algorithms on these problems and also on synthetic hybrid linear data as well as the mnist handwritten digits data and we demonstrate how to use our algorithms for fast determination of the number of affine subspaces

sensitivity analysis of the split complex valued multilayer perceptron due to the errors of the i i d inputs and weights in this paper we analyze the sensitivity of a split complex multilayer perceptron split cmlp due to the errors of the inputs and the connection weights between neurons for simplicity all the inputs and weights studied here are independent and identically distributed i i d to develop an algorithm to estimate the sensitivity of the entire split cmlp we compute statistically the sensitivity by using the central limit theorem clt the results show that the sensitivity is affected by the number of the layers and the number of the neurons adopted in each layer we derive a theoretical estimation of the sensitivity several numerical results of the sensitivity for the split cmlp are presented and they match the theoretical ones the agreement between the theoretical results and experimental results verifies the feasibility of the proposed algorithm thus we not only analyze the sensitivity of the split cmlp due to the errors of the i i d inputs and weights but also develop an efficient algorithm to estimate the sensitivity

efficient relighting of rbf based illumination adjustable images an illumination adjustable image iai contains a large number of prerecorded images under various light directions relighting a scene under complicated lighting conditions can be achieved from the iai using the radial basis function rbf approach to represent an iai is proven to be more efficient than using the spherical harmonic approach however to represent high frequency lighting effects we need to use many rbfs hence the relighting speed could be very slow this brief investigates a partial reconstruction scheme for relighting an iai based on the locality of rbfs compared with the conventional rbf and spherical harmonics sh approaches the proposed scheme has a much faster relighting speed under the similar distortion performance

acquiring linear subspaces for face recognition under variable lighting previous work has demonstrated that the image variation of many objects human faces in particular under variable lighting can be effectively modeled by low dimensional linear spaces even when there are multiple light sources and shadowing basis images spanning this space are usually obtained in one of three ways a large set of images of the object under different lighting conditions is acquired and principal component analysis pca is used to estimate a subspace alternatively synthetic images are rendered from a 3d model perhaps reconstructed from images under point sources and again pca is used to estimate a subspace finally images rendered from a 3d model under diffuse lighting based on spherical harmonics are directly used as basis images in this paper we show how to arrange physical lighting so that the acquired images of each object can be directly used as the basis vectors of a low dimensional linear space and that this subspace is close to those acquired by the other methods more specifically there exist configurations of k point light source directions with k typically ranging from 5 to 9 such that by taking k images of an object under these single sources the resulting subspace is an effective representation for recognition under a wide range of lighting conditions since the subspace is generated directly from real images potentially complex and or brittle intermediate steps such as 3d reconstruction can be completely avoided nor is it necessary to acquire large numbers of training images or to physically construct complex diffuse harmonic light fields we validate the use of subspaces constructed in this fashion within the context of face recognition

learning the nonlinear geometry of high dimensional data models and algorithms modern information processing relies on the axiom that high dimensional data lie near low dimensional geometric structures this paper revisits the problem of data driven learning of these geometric structures and puts forth two new nonlinear geometric models for data describing related objects phenomena the first one of these models straddles the two extremes of the subspace model and the union of subspaces model and is termed the metric constrained union of subspaces mc uos model the second one of these models suited for data drawn from a mixture of nonlinear manifolds generalizes the kernel subspace model and is termed the metric constrained kernel union of subspaces mc kuos model the main contributions of this paper in this regard include the following first it motivates and formalizes the problems of mc uos and mc kuos learning second it presents algorithms that efficiently learn an mc uos or an mc kuos underlying data of interest third it extends these algorithms to the case when parts of the data are missing last but not least it reports the outcomes of a series of numerical experiments involving both synthetic and real data that demonstrate the superiority of the proposed geometric models and learning algorithms over existing approaches in the literature these experiments also help clarify the connections between this work and the literature on subspace and kernel k means clustering

sparse subspace clustering algorithm theory and applications many real world problems deal with collections of high dimensional data such as images videos text and web documents dna microarray data and more often such high dimensional data lie close to low dimensional structures corresponding to several classes or categories to which the data belong in this paper we propose and study an algorithm called sparse subspace clustering to cluster data points that lie in a union of low dimensional subspaces the key idea is that among the infinitely many possible representations of a data point in terms of other points a sparse representation corresponds to selecting a few points from the same subspace this motivates solving a sparse optimization program whose solution is used in a spectral clustering framework to infer the clustering of the data into subspaces since solving the sparse optimization program is in general np hard we consider a convex relaxation and show that under appropriate conditions on the arrangement of the subspaces and the distribution of the data the proposed minimization program succeeds in recovering the desired sparse representations the proposed algorithm is efficient and can handle data points near the intersections of subspaces another key advantage of the proposed algorithm with respect to the state of the art is that it can deal directly with data nuisances such as noise sparse outlying entries and missing entries by incorporating the model of the data into the sparse optimization program we demonstrate the effectiveness of the proposed algorithm through experiments on synthetic data as well as the two real world problems of motion segmentation and face clustering

algebraic clustering of affine subspaces subspace clustering is an important problem in machine learning with many applications in computer vision and pattern recognition prior work has studied this problem using algebraic iterative statistical low rank and sparse representation techniques while these methods have been applied to both linear and affine subspaces theoretical results have only been established in the case of linear subspaces for example algebraic subspace clustering asc is guaranteed to provide the correct clustering when the data points are in general position and the union of subspaces is transversal in this paper we study in a rigorous fashion the properties of asc in the case of affine subspaces using notions from algebraic geometry we prove that the homogenization trick which embeds points in a union of affine subspaces into points in a union of linear subspaces preserves the general position of the points and the transversality of the union of subspaces in the embedded space thus establishing the correctness of asc for affine subspaces

restricted isometry property of gaussian random projection for finite set of subspaces dimension reduction plays an essential role when decreasing the complexity of solving large scale problems the well known johnson lindenstrauss jl lemma and restricted isometry property rip admit the use of random projection to reduce the dimension while keeping the euclidean distance which leads to the boom of compressed sensing and the field of sparsity related signal processing recently successful applications of sparse models in computer vision and machine learning have increasingly hinted that the underlying structure of high dimensional data looks more like a union of subspaces in this paper motivated by jl lemma and an emerging field of compressed subspace clustering we study for the first time the rip of gaussian random matrices for the compression of two subspaces based on the generalized projection f norm distance we theoretically prove that with high probability the affinity or distance between two projected subspaces are concentrated around their estimates when the ambient dimension after projection is sufficiently large the affinity and distance between two subspaces almost remain unchanged after random projection numerical experiments verify the theoretical work

robust subspace clustering subspace clustering refers to the task of nding a multi subspace representation that best ts a collection of points taken from a high dimensional space this paper introduces an algorithm inspired by sparse subspace clustering ssc 25 to cluster noisy data and develops some novel theory demonstrating its correctness in particular the theory uses ideas from geometric functional analysis to show that the algorithm can accurately recover the underlying subspaces under minimal requirements on their orientation and on the number of samples per subspace synthetic as well as real data experiments complement our theoretical study illustrating our approach and demonstrating its eectiveness