efficient estimation of class a noise parameters via the em algorithm the class a middleton noise model is a commonly used statistical physical parametric model for non gaussian interference superimposed on a gaussian background in this study the problem of efficient estimation of the class a parameters for small sample sizes is considered the proposed estimator is based on the em algorithm a two step iterative estimation technique that is ideally suited for the class a estimation problem since the observations can be readily treated as incomplete data for the single parameter estimation problem a closed form expression for the estimator is obtained furthermore for the single parameter estimation problem it is shown that the sequence of estimates obtained via the em algorithm converges and a characterization of the point to which the sequence converges is given in particular it is shown that if the limit point of this convergent sequence is an interior point of the parameter set of interest then it must be a stationary point of the traditional likelihood function in addition for both the single parameter and two parameter estimation problems the small sample size performance of the proposed em algorithm is examined via an extensive simulation study

large scale multiple kernel learning while classical kernel based learning algorithms are based on a single kernel in practice it is often desirable to use multiple kernels lanckriet et al 2004 considered conic combinations of kernel matrices for classification leading to a convex quadratically constrained quadratic program we show that it can be rewritten as a semi infinite linear program that can be efficiently solved by recycling the standard svm implementations moreover we generalize the formulation and our method to a larger class of problems including regression and one class classification experimental results show that the proposed algorithm works for hundred thousands of examples or hundreds of kernels to be combined and helps for automatic model selection improving the interpretability of the learning result in a second part we discuss general speed up mechanism for svms especially when used with sparse feature maps as appear for string kernels allowing us to train a string kernel svm on a 10 million real world splice data set from computational biology we integrated multiple kernel learning in our machine learning toolbox shogun for which the source code is publicly available at http www fml tuebingen mpg de raetsch projects shogun

generalizing discriminant analysis using the generalized singular value decomposition discriminant analysis has been used for decades to extract features that preserve class separability it is commonly defined as an optimization problem involving covariance matrices that represent the scatter within and between clusters the requirement that one of these matrices be nonsingular limits its application to data sets with certain relative dimensions we examine a number of optimization criteria and extend their applicability by using the generalized singular value decomposition to circumvent the nonsingularity requirement the result is a generalization of discriminant analysis that can be applied even when the sample size is smaller than the dimension of the sample data we use classification results from the reduced representation to compare the effectiveness of this approach with some alternatives and conclude with a discussion of their relative merits

semi infinite programming theory methods and applications starting from a number of motivating and abundant applications in 2 including control of robots eigenvalue computations mechanical stress of materials and statistical design the authors describe a class of optimization problems which are referred to as semi infinite because their constraints bound functions of a finite number of variables on a whole region in 3 5 first and second order optimality conditions are derived for general nonlinear problems as well as a procedure for reducing the problem locally to one with only finitely many constraints another main effort for achieving simplification is through duality in 6 there algebraic properties of finite linear programming are brought to bear on duality theory in semi infinite programming section 7 treats numerical methods based on either discretization or local reduction with the emphasis on the design of superlinearly convergent sqp type methods taking this differentiable point of view this paper can be considered to be complementar

blind detection of synchronous cdma in non gaussian channels most blind source separation algorithms assume the channel noise to be gaussian this paper considers the problem of noncooperative blind detection of synchronous direct sequence code division multiple access communications no knowledge of the spreading sequences or training data in non gaussian channels three iterative algorithms with different performance and complexity tradeoffs are proposed simulation results show that they significantly outperform gaussian optimal blind source separation algorithms in non gaussian channels the cramer rao lower bound for this problem is computed and the performance of the proposed algorithms is shown to approach this bound for moderate signal to noise ratios

higher order godunov methods for general systems of hyperbolic conservation laws abstract we describe an extension of higher order godunov methods to general systems of hyperbolic conservation laws this extension allow the method to be applied to problems that are not strictly hyperbolic and exhibit local linear degeneracies in the wave fields the method constructs an approximation of the riemann problem from local wave information a generalization of the engquist osher flux for systems is then used to compute a numerical flux based on this approximation this numerical flux replaces the godunov numerical flux in the algorithm thereby eliminating the need for a global riemann problem solution the additional modifications to the godunov methodology that are needed to treat loss of strict hyperbolicity are described in detail the method is applied to some simple model problems for which the global analytic structure is known the method is also applied to the black oil model for multiphase flow in petroleum reservoirs

finding community structure in networks using the eigenvectors of matrices we consider the problem of detecting communities or modules in networks groups of vertices with a higher than average density of edges connecting them previous work indicates that a robust approach to this problem is the maximization of the benefit function known as modularity over possible divisions of a network here we show that this maximization process can be written in terms of the eigenspectrum of a matrix we call the modularity matrix which plays a role in community detection similar to that played by the graph laplacian in graph partitioning calculations this result leads us to a number of possible algorithms for detecting community structure as well as several other results including a spectral measure of bipartite structure in networks and a centrality measure that identifies vertices that occupy central positions within the communities to which they belong the algorithms and measures proposed are illustrated with applications to a variety of real world complex networks

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

multidimensional upwind methods for hyperbolic conservation laws abstract we present a class of second order conservative finite difference algorithms for solving numerically time dependent problems for hyperbolic conservation laws in several space variables these methods are upwind and multidimensional in that the numerical fluxes are obtained by solving the characteristic form of the full multidimensional equations at the zone edge and that all fluxes are evaluated and differenced at the same time in particular operator splitting is not used correct behavior at discontinuities is obtained by the use of solutions to the riemann problem and by limiting some of the second order terms numerical results are presented which show that the methods described here yield the same high resolution as the corresponding operator split methods

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

structural minimax probability machine minimax probability machine mpm is an interesting discriminative classifier based on generative prior knowledge it can directly estimate the probabilistic accuracy bound by minimizing the maximum probability of misclassification the structural information of data is an effective way to represent prior knowledge and has been found to be vital for designing classifiers in real world problems however mpm only considers the prior probability distribution of each class with a given mean and covariance matrix which does not efficiently exploit the structural information of data in this paper we use two finite mixture models to capture the structural information of the data from binary classification for each subdistribution in a finite mixture model only its mean and covariance matrix are assumed to be known based on the finite mixture models we propose a structural mpm smpm smpm can be solved effectively by a sequence of the second order cone programming problems moreover we extend a linear model of smpm to a nonlinear model by exploiting kernelization techniques we also show that the smpm can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi min margin machine under certain special conditions experimental results on both synthetic and real world data sets demonstrate the effectiveness of smpm

cumulative voting consensus method for partitions with variable number of clusters over the past few years there has been a renewed interest in the consensus clustering problem several new methods have been proposed for finding a consensus partition for a set of n data objects that optimally summarizes an ensemble in this paper we propose new consensus clustering algorithms with linear computational complexity in n we consider clusterings generated with a random number of clusters which we describe by categorical random variables we introduce the idea of cumulative voting as a solution for the problem of cluster label alignment where unlike the common one to one voting scheme a probabilistic mapping is computed we seek a first summary of the ensemble that minimizes the average squared distance between the mapped partitions and the optimal representation of the ensemble where the selection criterion of the reference clustering is defined based on maximizing the information content as measured by the entropy we describe cumulative vote weighting schemes and corresponding algorithms to compute an empirical probability distribution summarizing the ensemble given the arbitrary number of clusters of the input partitions we formulate the problem of extracting the optimal consensus as that of finding a compressed summary of the estimated distribution that preserves the maximum relevant information an efficient solution is obtained using an agglomerative algorithm that minimizes the average generalized jensen shannon divergence within the cluster the empirical study demonstrates significant gains in accuracy and superior performance compared to several recent consensus clustering algorithms

multiple kernel clustering based on centered kernel alignment multiple kernel clustering mkc which performs kernel based data fusion for data clustering is an emerging topic it aims at solving clustering problems with multiple cues most mkc methods usually extend existing clustering methods with a multiple kernel learning mkl setting in this paper we propose a novel mkc method that is different from those popular approaches centered kernel alignment an effective kernel evaluation measure is employed in order to unify the two tasks of clustering and mkl into a single optimization framework to solve the formulated optimization problem an efficient two step iterative algorithm is developed experiments on several uci datasets and face image datasets validate the effectiveness and efficiency of our mkc algorithm

multi class discriminant kernel learning via convex programming regularized kernel discriminant analysis rkda performs linear discriminant analysis in the feature space via the kernel trick its performance depends on the selection of kernels in this paper we consider the problem of multiple kernel learning mkl for rkda in which the optimal kernel matrix is obtained as a linear combination of pre specified kernel matrices we show that the kernel learning problem in rkda can be formulated as convex programs first we show that this problem can be formulated as a semidefinite program sdp based on the equivalence relationship between rkda and least square problems in the binary class case we propose a convex quadratically constrained quadratic programming qcqp formulation for kernel learning in rkda a semi infinite linear programming silp formulation is derived to further improve the efficiency we extend these formulations to the multi class case based on a key result established in this paper that is the multi class rkda kernel learning problem can be decomposed into a set of binary class kernel learning problems which are constrained to share a common kernel based on this decomposition property sdp formulations are proposed for the multi class case furthermore it leads naturally to qcqp and silp formulations as the performance of rkda depends on the regularization parameter we show that this parameter can also be optimized in a joint framework with the kernel extensive experiments have been conducted and analyzed and connections to other algorithms are discussed

information criteria for support vector machines this paper presents kernel regularization information criterion kric which is a new criterion for tuning regularization parameters in kernel logistic regression klr and support vector machines svms the main idea of the kric is based on the regularization information criterion ric we derive an eigenvalue equation to calculate the kric and solve the problem the computational cost for parameter tuning by the kric is reduced drastically by using the nystroumlm approximation the test error rate of svms or klr with the regularization parameter tuned by the kric is comparable with the one by the cross validation or evaluation of the evidence the computational cost of the kric is significantly lower than the one of the other criteria

a fast and accurate matrix completion method based on qr decomposition and l_ 2 1 norm minimization low rank matrix completion aims to recover matrices with missing entries and has attracted considerable attention from machine learning researchers most of the existing methods such as weighted nuclear norm minimization based methods and qatar riyal qr decomposition based methods cannot provide both convergence accuracy and convergence speed to investigate a fast and accurate completion method an iterative qr decomposition based method is proposed for computing an approximate singular value decomposition this method can compute the largest r r 0 singular values of a matrix by iterative qr decomposition then under the framework of matrix trifactorization a method for computing an approximate svd based on qr decomposition csvd qr based l 2 1 norm minimization method lnm qr is proposed for fast matrix completion theoretical analysis shows that this qr decomposition based method can obtain the same optimal solution as a nuclear norm minimization method i e the l 2 1 norm of a submatrix can converge to its nuclear norm consequently an lnm qr based iteratively reweighted l 2 1 norm minimization method irlnm qr is proposed to improve the accuracy of lnm qr theoretical analysis shows that irlnm qr is as accurate as an iteratively reweighted nuclear norm minimization method which is much more accurate than the traditional qr decomposition based matrix completion methods experimental results obtained on both synthetic and real world visual data sets show that our methods are much faster and more accurate than the state of the art methods

sampling methods for the nystrom method the nystrom method is an efficient technique to generate low rank matrix approximations and is used in several large scale learning applications a key aspect of this method is the procedure according to which columns are sampled from the original matrix in this work we explore the efficacy of a variety of fixed and adaptive sampling schemes we also propose a family of ensemble based sampling algorithms for the nystrom method we report results of extensive experiments that provide a detailed comparison of various fixed and adaptive sampling techniques and demonstrate the performance improvement associated with the ensemble nystrom method when used in conjunction with either fixed or adaptive sampling schemes corroborating these empirical findings we present a theoretical analysis of the nystrom method providing novel error bounds guaranteeing a better convergence rate of the ensemble nystrom method in comparison to the standard nystrom method

large scale nystrom kernel matrix approximation using randomized svd the nystrom method is an efficient technique for the eigenvalue decomposition of large kernel matrices however to ensure an accurate approximation a sufficient number of columns have to be sampled on very large data sets the singular value decomposition svd step on the resultant data submatrix can quickly dominate the computations and become prohibitive in this paper we propose an accurate and scalable nystrom scheme that first samples a large column subset from the input matrix but then only performs an approximate svd on the inner submatrix using the recent randomized low rank matrix approximation algorithms theoretical analysis shows that the proposed algorithm is as accurate as the standard nystrom method that directly performs a large svd on the inner submatrix on the other hand its time complexity is only as low as performing a small svd encouraging results are obtained on a number of large scale data sets for low rank approximation moreover as the most computational expensive steps can be easily distributed and there is minimal data transfer among the processors significant speedup can be further obtained with the use of multiprocessor and multi gpu systems

finding structure with randomness probabilistic algorithms for constructing approximate matrix decompositions low rank matrix approximations such as the truncated singular value decomposition and the rank revealing qr decomposition play a central role in data analysis and scientific computing this work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low rank matrix approximation these techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets this paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions these methods use random sampling to identify a subspace that captures most of the action of a matrix the input matrix is then compressed either explicitly or implicitly to this subspace and the reduced matrix is manipulated deterministically to obtain the desired low rank factorization in many cases this approach beats its classical competitors in terms of accuracy robustness and or speed these claims are supported by extensive numerical experiments and a detailed error analysis the specific benefits of randomized techniques depend on the computational environment consider the model problem of finding the k dominant components of the singular value decomposition of an m times n matrix i for a dense input matrix randomized algorithms require bigo mn log k floating point operations flops in contrast to bigo mnk for classical algorithms ii for a sparse input matrix the flop count matches classical krylov subspace methods but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures iii for a matrix that is too large to fit in fast memory the randomized techniques require only a constant number of passes over the data as opposed to bigo k passes for classical algorithms in fact it is sometimes possible to perform matrix approximation with a single pass over the data

mercer kernel based clustering in feature space the article presents a method for both the unsupervised partitioning of a sample of data and the estimation of the possible number of inherent clusters which generate the data this work exploits the notion that performing a nonlinear data transformation into some high dimensional feature space increases the probability of the linear separability of the patterns within the transformed space and therefore simplifies the associated data structure it is shown that the eigenvectors of a kernel matrix which defines the implicit mapping provides a means to estimate the number of clusters inherent within the data and a computationally simple iterative procedure is presented for the subsequent feature space partitioning of the data

uncorrelated multilinear discriminant analysis with regularization and aggregation for tensor object recognition this paper proposes an uncorrelated multilinear discriminant analysis umlda framework for the recognition of multidimensional objects known as tensor objects uncorrelated features are desirable in recognition tasks since they contain minimum redundancy and ensure independence of features the umlda aims to extract uncorrelated discriminative features directly from tensorial data through solving a tensor to vector projection the solution consists of sequential iterative processes based on the alternating projection method and an adaptive regularization procedure is incorporated to enhance the performance in the small sample size sss scenario a simple nearest neighbor classifier is employed for classification furthermore exploiting the complementary information from differently initialized and regularized umlda recognizers an aggregation scheme is adopted to combine them at the matching score level resulting in enhanced generalization performance while alleviating the regularization parameter selection problem the umlda based recognition algorithm is then empirically shown on face and gait recognition tasks to outperform four multilinear subspace solutions mpca dater gtda tr1da and four linear subspace solutions bayesian lda ulda r jd lda

uncorrelated linear discriminant analysis based on weighted pairwise fisher criterion in this paper we propose a novel uncorrelated weighted linear discriminant analysis uwlda method for feature extraction and recognition the uwlda first introduces a weighting function to restrain the dominant role of the classes with larger distance and then searches the optimal discriminant vectors under the conjugative orthogonal constrains in the null space of the within class scatter matrix and its conjugative orthogonal complement space respectively as a result the proposed technique not only derive the optimal and lossless discriminative information but also guarantee that all extracted features are statistically uncorrelated experiments on feret face database and ar face database are performed to test and evaluate the proposed algorithm the results demonstrate the effectiveness of uwlda

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

spectral and dynamical properties in classes of sparse networks with mesoscopic inhomogeneities we study structure eigenvalue spectra and random walk dynamics in a wide class of networks with subgraphs modules at mesoscopic scale the networks are grown within the model with three parameters controlling the number of modules their internal structure as scale free and correlated subgraphs and the topology of connecting network within the exhaustive spectral analysis for both the adjacency matrix and the normalized laplacian matrix we identify the spectral properties which characterize the mesoscopic structure of sparse cyclic graphs and trees the minimally connected nodes the clustering and the average connectivity affect the central part of the spectrum the number of distinct modules leads to an extra peak at the lower part of the laplacian spectrum in cyclic graphs such a peak does not occur in the case of topologically distinct tree subgraphs connected on a tree whereas the associated eigenvectors remain localized on the subgraphs both in trees and cyclic graphs we also find a characteristic pattern of periodic localization along the chains on the tree for the eigenvector components associated with the largest eigenvalue lambda l 2 of the laplacian further differences between the cyclic modular graphs and trees are found by the statistics of random walks return times and hitting patterns at nodes on these graphs the distribution of first return times averaged over all nodes exhibits a stretched exponential tail with the exponent sigma approximately 1 3 for trees and sigma approximately 2 3 for cyclic graphs which is independent of their mesoscopic and global structure

fast approximation of matrix coherence and statistical leverage the statistical leverage scores of a matrix a are the squared row norms of the matrix containing its top left singular vectors and the coherence is the largest leverage score these quantities are of interest in recently popular problems such as matrix completion and nystrom based low rank matrix approximation as well as in large scale statistical data analysis applications more generally moreover they are of interest since they define the key structural nonuniformity that must be dealt with in developing fast randomized matrix algorithms our main result is a randomized algorithm that takes as input an arbitrary n d matrix a with n d and that returns as output relative error approximations to all n of the statistical leverage scores the proposed algorithm runs under assumptions on the precise values of n and d in o nd logn time as opposed to the o nd2 time required by the naive algorithm that involves computing an orthogonal basis for the range of a our analysis may be viewed in terms of computing a relative error approximation to an underconstrained least squares approximation problem or relatedly it may be viewed as an application of johnson lindenstrauss type ideas several practically important extensions of our basic result are also described including the approximation of so called cross leverage scores the extension of these ideas to matrices with n d and the extension to streaming environments

combining multiple clusterings using evidence accumulation we explore the idea of evidence accumulation eac for combining the results of multiple clusterings first a clustering ensemble a set of object partitions is produced given a data set n objects or patterns in d dimensions different ways of producing data partitions are 1 applying different clustering algorithms and 2 applying the same clustering algorithm with different values of parameters or initializations further combinations of different data representations feature spaces and clustering algorithms can also provide a multitude of significantly different data partitionings we propose a simple framework for extracting a consistent clustering given the various partitions in a clustering ensemble according to the eac concept each partition is viewed as an independent evidence of data organization individual data partitions being combined based on a voting mechanism to generate a new n times n similarity matrix between the n patterns the final data partition of the n patterns is obtained by applying a hierarchical agglomerative clustering algorithm on this matrix we have developed a theoretical framework for the analysis of the proposed clustering combination strategy and its evaluation based on the concept of mutual information between data partitions stability of the results is evaluated using bootstrapping techniques a detailed discussion of an evidence accumulation based clustering algorithm using a split and merge strategy based on the k means clustering algorithm is presented experimental results of the proposed method on several synthetic and real data sets are compared with other combination strategies and with individual clustering results produced by well known clustering algorithms

simple graphs in granular computing given a graph we interpret its adjacency matrix as an information table we study this correspondence in two directions firstly on the side of graphs by applying to it standard techniques from granular computing in this way we are able to connect automorphisms on graphs to the so called indiscernibility relation and a particular hypergraph built from the starting graph to core and reducts on the other hand new concepts are introduced on graphs that have an interesting correspondence on information tables in particular some new topological interpretations of the graph and the concept of extended core are given

noise level estimation for model selection in kernel pca denoising one of the main challenges in unsupervised learning is to find suitable values for the model parameters in kernel principal component analysis kpca for example these are the number of components the kernel and its parameters this paper presents a model selection criterion based on distance distributions mdds this criterion can be used to find the number of components and the sigma 2 parameter of radial basis function kernels by means of spectral comparison between information and noise the noise content is estimated from the statistical moments of the distribution of distances in the original dataset this allows for a type of randomization of the dataset without actually having to permute the data points or generate artificial datasets after comparing the eigenvalues computed from the estimated noise with the ones from the input dataset information is retained and maximized by a set of model parameters in addition to the model selection criterion this paper proposes a modification to the fixed size method and uses the incomplete cholesky factorization both of which are used to solve kpca in large scale applications these two approaches together with the model selection mdd were tested in toy examples and real life applications and it is shown that they outperform other known algorithms

dimensionality reduction of clustered data sets we present a novel probabilistic latent variable model to perform linear dimensionality reduction on data sets which contain clusters we prove that the maximum likelihood solution of the model is an unsupervised generalization of linear discriminant analysis this provides a completely new approach to one of the most established and widely used classification algorithms the performance of the model is then demonstrated on a number of real and artificial data sets

clustered nystrom method for large scale manifold learning and dimension reduction kernel or similarity matrix plays a key role in many machine learning algorithms such as kernel methods manifold learning and dimension reduction however the cost of storing and manipulating the complete kernel matrix makes it infeasible for large problems the nystrom method is a popular sampling based low rank approximation scheme for reducing the computational burdens in handling large kernel matrices in this paper we analyze how the approximating quality of the nystrom method depends on the choice of landmark points and in particular the encoding powers of the landmark points in summarizing the data our non probabilistic error analysis justifies a clustered nystrom method that uses the k means clustering centers as landmark points our algorithm can be applied to scale up a wide variety of algorithms that depend on the eigenvalue decomposition of kernel matrix or its variant such as kernel principal component analysis laplacian eigenmap spectral clustering as well as those involving kernel matrix inverse such as least squares support vector machine and gaussian process regression extensive experiments demonstrate the competitive performance of our algorithm in both accuracy and efficiency

optimized data fusion for kernel k means clustering this paper presents a novel optimized kernel k means algorithm okkc to combine multiple data sources for clustering analysis the algorithm uses an alternating minimization framework to optimize the cluster membership and kernel coefficients as a nonconvex problem in the proposed algorithm the problem to optimize the cluster membership and the problem to optimize the kernel coefficients are all based on the same rayleigh quotient objective therefore the proposed algorithm converges locally okkc has a simpler procedure and lower complexity than other algorithms proposed in the literature simulated and real life data fusion applications are experimentally studied and the results validate that the proposed algorithm has comparable performance moreover it is more efficient on large scale data sets the matlab implementation of okkc algorithm is downloadable from http homes esat kuleuven be sistawww bio syu okkc html

graph regularized nonnegative matrix factorization for data representation matrix factorization techniques have been frequently applied in information retrieval computer vision and pattern recognition among them nonnegative matrix factorization nmf has received considerable attention due to its psychological and physiological interpretation of naturally occurring data whose representation may be parts based in the human brain on the other hand from the geometric perspective the data is usually sampled from a low dimensional manifold embedded in a high dimensional ambient space one then hopes to find a compact representation which uncovers the hidden semantics and simultaneously respects the intrinsic geometric structure in this paper we propose a novel algorithm called graph regularized nonnegative matrix factorization gnmf for this purpose in gnmf an affinity graph is constructed to encode the geometrical information and we seek a matrix factorization which respects the graph structure our empirical study shows encouraging results of the proposed algorithm in comparison to the state of the art algorithms on real world problems

cross flow tensors and finite volume approximation with by deferred correction abstract a general framework for full tensor finite volume schemes is presented along with properties of the schemes that in turn provide motivation for a deferred correction scheme the approach is illustrated by writing a general nine node discretization as a five node scheme plus a residual term involving the extra nodes a possible solution procedure involves solving the standard five node matrix problem to yield an initial solution then re solving the problem with the residual determined by the first iterate of the pressure field relaxation using the pressure from the previous impes time step is also examined stability of the inner iteration is considered and strategies are discussed this method is coupled with a higher order godunov scheme for integrating the convective components of the flow equations results presented include two phase flow examples with full tensor fields and serve to demonstrate the potential of the method

a semi supervised approximate spectral clustering algorithm based on hmrf model abstract before clustering we usually have some background knowledge about the data structure pairwise constraints are commonly used background knowledge for graph partition problems pairwise constraints can be naturally added to the graph edge this paper integrates pairwise constraints into the objective function of graph cuts and derive the semi supervised approximate spectral clustering based on hidden markov random fields hmrf this algorithm utilize the mathematical connection between hmrf semi supervised clustering and approximate weighted kernel k means the approximate weighted kernel k means is used to calculate the optimal clustering results of hmrf spectral clustering the effectiveness of the proposed algorithm is verified on several benchmark data sets experiments show that adding more pairwise constraints will help improve the clustering performance our method has advantages for the challenging clustering tasks of large scale nonlinear data because of the high efficiency and less memory consumption

analytical method for blind binary signal separation the blind separation of multiple co channel binary digital signals using an antenna array involves finding a factorization of a data matrix x into x as where all entries of s are 1 or 1 it is shown that this problem can be solved exactly and noniteratively via a certain generalized eigenvalue decomposition as indicated by simulations the algorithm is robust in the presence of noise an interesting implication is that certain cluster segmentation problems can be solved using eigenvalue techniques

approximate normalized cuts without eigen decomposition most traditional weighted graph clustering algorithms are solved by spectral method which is only suitable for small scale datasets because of the high space and time complexity how to reduce the computational complexity of graph cut clustering to process the massive complex data has become a challenging problem to overcome this problem we design an approximate normalized cuts algorithm without eigen decomposition for large scale clustering on the one hand the space requirement of normalized cut is decreased by sampling a few data points to infer the global features of dataset instead of using the whole affinity matrix on the other hand the graph cut clustering procedure is accelerated in an iterative way that using the approximate weighted kernel k means to optimize the objective function of normalized cut which avoids the direct eigen decomposition of laplacian matrix we also analyze the approximation error of the proposed algorithm and compare it with other state of the arts clustering algorithms on several benchmark datasets the experimental results demonstrate that our method can efficiently do the clustering when the number of data objects exceeds tens of thousands

fuzzy nonlinear regression analysis using a random weight network modeling a fuzzy in fuzzy out system where both inputs and outputs are uncertain is of practical and theoretical importance fuzzy nonlinear regression fnr is one of the approaches used most widely to model such systems in this study we propose the use of a random weight network rwn to develop a fnr model called fnrrwn where both the inputs and outputs are triangular fuzzy numbers unlike existing fnr models based on back propagation bp and radial basis function rbf networks fnrrwn does not require iterative adjustment of the network weights and biases instead the input layer weights and hidden layer biases of fnrrwn are selected randomly the output layer weights for fnrrwn are calculated analytically based on a derived updating rule which aims to minimize the integrated squared error between cut sets that correspond to the predicted fuzzy outputs and target fuzzy outputs respectively in fnrrwn the integrated squared error is solved approximately by riemann integral theory the experimental results show that the proposed fnrrwn method can effectively approximate a fuzzy in fuzzy out system fnrrwn obtains better prediction accuracy in a lower computational time compared with existing fnr models based on bp and rbf networks

space alternating generalized expectation maximization algorithm the expectation maximization em method can facilitate maximizing likelihood functions that arise in statistical estimation problems in the classical em paradigm one iteratively maximizes the conditional log likelihood of a single unobservable complete data space rather than maximizing the intractable likelihood function for the measured or incomplete data em algorithms update all parameters simultaneously which has two drawbacks 1 slow convergence and 2 difficult maximization steps due to coupling when smoothness penalties are used the paper describes the space alternating generalized em sage method which updates the parameters sequentially by alternating between several small hidden data spaces defined by the algorithm designer the authors prove that the sequence of estimates monotonically increases the penalized likelihood objective derive asymptotic convergence rates and provide sufficient conditions for monotone convergence in norm two signal processing applications illustrate the method estimation of superimposed signals in gaussian noise and image reconstruction from poisson measurements in both applications the sage algorithms easily accommodate smoothness penalties and converge faster than the em algorithms

multidimensional upwind convection schemes for flow in porous media on structured and unstructured quadrilateral grids standard reservoir simulation schemes employ first order upwind schemes for approximation of the convective fluxes when multiple phases or components are present these convective flux schemes rely upon upwind information that is determined according to grid geometry as a consequence directional diffusion is introduced into the solution that is grid dependent the effect can be particularly important for cases where the flow is across grid coordinate lines and is known as cross wind diffusion truly higher dimensional upwind schemes that minimize cross wind diffusion are presented for convective flow approximation on quadrilateral unstructured grids the schemes are locally conservative and yield improved results that are essentially free of spurious oscillations the higher dimensional schemes are coupled with full tensor darcy flux approximations the benefits of the resulting schemes are demonstrated for classical test problems in reservoir simulation including cases with full tensor permeability fields the test cases involve a range of structured and unstructured grids with variations in orientation and permeability that lead to flow fields that are poorly resolved by standard simulation methods the higher dimensional formulations are shown to effectively reduce the numerical cross wind diffusion effect leading to improved resolution of concentration and saturation fronts

weighted graph cuts without eigenvectors a multilevel approach a variety of clustering algorithms have recently been proposed to handle data that is not linearly separable spectral clustering and kernel k means are two of the main methods in this paper we discuss an equivalence between the objective functions used in these seemingly different methods in particular a general weighted kernel k means objective is mathematically equivalent to a weighted graph clustering objective we exploit this equivalence to develop a fast high quality multilevel algorithm that directly optimizes various weighted graph clustering objectives such as the popular ratio cut normalized cut and ratio association criteria this eliminates the need for any eigenvector computation for graph clustering problems which can be prohibitive for very large graphs previous multilevel graph partitioning methods such as metis have suffered from the restriction of equal sized clusters our multilevel algorithm removes this restriction by using kernel k means to optimize weighted graph cuts experimental results show that our multilevel algorithm outperforms a state of the art spectral clustering algorithm in terms of speed memory usage and quality we demonstrate that our algorithm is applicable to large scale clustering tasks such as image segmentation social network analysis and gene network analysis

maximum likelihood estimation of constellation vectors for blind separation of co channel bpsk signals and its performance analysis in this paper we present a method for blind separation of co channel bpsk signals arriving at an antenna array this method consists of two parts the maximum likelihood constellation estimation and assignment we show that at high snr the maximum likelihood constellation estimation is well approximated by the smallest distance clustering algorithm which we proposed earlier on heuristic grounds we observe that both these methods for estimating the constellation vectors perform very well at high snr and nearly attain cramer rao bounds using this fact and noting that the assignment algorithm causes negligible error at high snr we derive upper bounds on the probability of bit error for the above method at high snr these upper bounds fall very rapidly with increasing snr showing that our constellation estimation assignment approach is very efficient simulation results are given to demonstrate the usefulness of the bounds

greedy discrete particle swarm optimization for large scale social network clustering social computing is a new paradigm for information and communication technology social network analysis is one of the theoretical underpinnings of social computing community structure detection is believed to be an effective tool for social network analysis uncovering community structures in social networks can be regarded as clustering optimization problems because social networks are characterized by dynamics and huge volumes of data conventional nature inspired optimization algorithms encounter serious challenges when applied to solving large scale social network clustering optimization problems in this study we put forward a novel particle swarm optimization algorithm to reveal community structures in social networks the particle statuses are redefined under a discrete scenario the status updating rules are reconsidered based on the network topology a greedy strategy is designed to drive particles to a promising region to this end a greedy discrete particle swarm optimization framework for large scale social network clustering is suggested to determine the performance of the algorithm extensive experiments on both synthetic and real world social networks are carried out we also compare the proposed algorithm with several state of the art network community clustering methods the experimental results demonstrate that the proposed method is effective and promising for social network clustering

generalized discriminant analysis using a kernel approach we present a new method that we call generalized discriminant analysis gda to deal with nonlinear discriminant analysis using kernel function operator the underlying theory is close to the support vector machines svm insofar as the gda method provides a mapping of the input vectors into high dimensional feature space in the transformed space linear properties make it easy to extend and generalize the classical linear discriminant analysis lda to nonlinear discriminant analysis the formulation is expressed as an eigenvalue problem resolution using a different kernel one can cover a wide class of nonlinearities for both simulated data and alternate kernels we give classification results as well as the shape of the decision function the results are confirmed using real data to perform seed classification

blind separation of synchronous co channel digital signals using an antenna array i algorithms we propose a maximum likelihood ml approach for separating and estimating multiple synchronous digital signals arriving at an antenna array at a cell site the spatial response of the array is assumed to be known imprecisely or unknown we exploit the finite alphabet property of digital signals to simultaneously estimate the array response and the symbol sequence for each signal uniqueness of the estimates is established for bpsk signals we introduce a signal detection technique based on the finite alphabet property that is different from a standard linear combiner computationally efficient algorithms for both block and recursive estimation of the signals are presented this new approach is applicable to an unknown array geometry and propagation environment which is particularly useful in wireless communication systems simulation results demonstrate its promising performance

fast monte carlo algorithms for matrices i approximating matrix multiplication motivated by applications in which the data may be formulated as a matrix we consider algorithms for several common linear algebra problems these algorithms make more efficient use of computational resources such as the computation time random access memory ram and the number of passes over the data than do previously known algorithms for these problems in this paper we devise two algorithms for the matrix multiplication problem suppose a and b which are m times n and n times p respectively are the two input matrices in our main algorithm we perform c independent trials where in each trial we randomly sample an element of 1 2 ldots n with an appropriate probability distribution cal p on 1 2 ldots n we form an m times c matrix c consisting of the sampled columns of a each scaled appropriately and we form a c times n matrix r using the corresponding rows of b again scaled appropriately the choice of cal p and the column and row scaling are crucial features of the algorithm when these are chosen judiciously we show that cr is a good approximation to ab more precisely we show that left ab cr right f o left a right f left b right f sqrt c where cdot f denotes the frobenius norm i e a 2 f sum i j a ij 2 this algorithm can be implemented without storing the matrices a and b in ram provided it can make two passes over the matrices stored in external memory and use o c m n p additional ram to construct c and r we then present a second matrix multiplication algorithm which is similar in spirit to our main algorithm in addition we present a model the pass efficient model in which the efficiency of these and other approximate matrix algorithms may be studied and which we argue is well suited to many applications involving massive data sets in this model the scarce computational resources are the number of passes over the data and the additional space and time required by the algorithm the input matrices may be presented in any order of the entries and not just row or column order as is the case in many applications where e g the data has been written in by multiple agents in addition the input matrices may be presented in a sparse representation where only the nonzero entries are written

clustering ensembles models of consensus and weak partitions clustering ensembles have emerged as a powerful method for improving both the robustness as well as the stability of unsupervised classification solutions however finding a consensus clustering from multiple partitions is a difficult problem that can be approached from graph based combinatorial or statistical perspectives this study extends previous research on clustering ensembles in several respects first we introduce a unified representation for multiple clusterings and formulate the corresponding categorical clustering problem second we propose a probabilistic model of consensus using a finite mixture of multinomial distributions in a space of clusterings a combined partition is found as a solution to the corresponding maximum likelihood problem using the em algorithm third we define a new consensus function that is related to the classical intraclass variance criterion using the generalized mutual information definition finally we demonstrate the efficacy of combining partitions generated by weak clustering algorithms that use data projections and random data splits a simple explanatory model is offered for the behavior of combinations of such weak clustering components combination accuracy is analyzed as a function of several parameters that control the power and resolution of component partitions as well as the number of partitions we also analyze clustering ensembles with incomplete information and the effect of missing cluster labels on the quality of overall consensus experimental results demonstrate the effectiveness of the proposed methods on several real world data sets

blind digital signal separation using successive interference cancellation iterative least squares blind separation of instantaneous linear mixtures of digital signals is a basic problem in communications when little or nothing can be assumed about the mixing matrix signal separation may be achieved by exploiting structural properties of the transmitted signals e g finite alphabet or coding constraints we propose a monotonically convergent and computationally efficient iterative least squares ils blind separation algorithm based on an optimal scaling lemma the signal estimation step of the proposed algorithm is reminiscent of successive interference cancellation sic ideas for well conditioned data and moderate snr the proposed sic ils algorithm provides a better performance complexity tradeoff than competing ils algorithms coupled with blind algebraic digital signal separation methods sic ils offers a computationally inexpensive true least squares refinement option we also point out that a widely used ils finite alphabet blind separation algorithm can exhibit limit cycle behavior

uniform projection for multi view learning multi view learning aims to integrate multiple data information from different views to improve the learning performance the key problem is to handle the unconformities or distortions among view specific samples or measurements of similarity or dissimilarity this paper models the view specific samples as a nonlinear mapping of uniform but latent intact samples for all the views and the view specific dissimilarity matrices or similarity matrices are estimated in terms of the uniform latent one two methods are then developed for multi view clustering one makes use of uniform multidimensional scaling umds on multi view dissimilarities or kernels the other one uses a uniform class assignment uca procedure that optimally extracts the cluster components contained in the view specific similarity matrices these two methods result in the same optimization model subjected to some slightly different constraints a first order condition of solutions is given as a nonlinear eigenvalue problem and a second order condition guarantees local optimality the nonlinear eigenvalue problem is solved by an iterative algorithm via eigen space updating and its convergence is proven furthermore a fast implementation of the algorithm is discussed which adopts the strategy of restarting subspace extension numerical experiments on some real world data sets provide good support to the proposed methods

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

on the nystrom method for approximating a gram matrix for improved kernel based learning a problem for many kernel based methods is that the amount of computation required to find the solution scales as o n3 where n is the number of training examples we develop and analyze an algorithm to compute an easily interpretable low rank approximation to an n n gram matrix g such that computations of interest may be performed more rapidly the approximation is of the form gk cwk ct where c is a matrix consisting of a small number c of columns of g and wk is the best rank k approximation to w the matrix formed by the intersection between those c columns of g and the corresponding c rows of g an important aspect of the algorithm is the probability distribution used to randomly sample the columns we will use a judiciously chosen and data dependent nonuniform probability distribution let 2 and f denote the spectral norm and the frobenius norm respectively of a matrix and let gk be the best rank k approximation to g we prove that by choosing o k e4 columns g cwk ct g gk e i 1n gii2 both in expectation and with high probability for both 2 f and for all k 0 k rank w this approximation can be computed using o n additional space and time after making two passes over the data from external storage the relationships between this algorithm other related matrix decompositions and the nystrom method from integral equation theory are discussed

fast monte carlo algorithms for matrices ii computing a low rank approximation to a matrix in many applications the data consist of or may be naturally formulated as an m times n matrix a it is often of interest to find a low rank approximation to a i e an approximation d to the matrix a of rank not greater than a specified rank k where k is much smaller than m and n methods such as the singular value decomposition svd may be used to find an approximation to a which is the best in a well defined sense these methods require memory and time which are superlinear in m and n for many applications in which the data sets are very large this is prohibitive two simple and intuitive algorithms are presented which when given an m times n matrix a compute a description of a low rank approximation d to a and which are qualitatively faster than the svd both algorithms have provable bounds for the error matrix a d for any matrix x let x f and x 2 denote its frobenius norm and its spectral norm respectively in the first algorithm c columns of a are randomly chosen if the m times c matrix c consists of those c columns of a after appropriate rescaling then it is shown that from c tc approximations to the top singular values and corresponding singular vectors may be computed from the computed singular vectors a description d of the matrix a may be computed such that mathrm rank d le k and such that left a d right xi 2 le min d mathrm rank d le k left a d right xi 2 poly k 1 c left a right 2 f holds with high probability for both xi 2 f this algorithm may be implemented without storing the matrix a in random access memory ram provided it can make two passes over the matrix stored in external memory and use o cm c 2 additional ram the second algorithm is similar except that it further approximates the matrix c by randomly sampling r rows of c to form a r times c matrix w thus it has additional error but it can be implemented in three passes over the matrix using only constant additional ram to achieve an additional error beyond the best rank k approximation that is at most epsilon a 2 f both algorithms take time which is polynomial in k 1 epsilon and log 1 delta where delta 0 is a failure probability the first takes time linear in mbox max m n and the second takes time independent of m and n our bounds improve previously published results with respect to the rank parameter k for both the frobenius and spectral norms in addition the proofs for the error bounds use a novel method that makes important use of matrix perturbation theory the probability distribution over columns of a and the rescaling are crucial features of the algorithms which must be chosen judiciously

cluster ensembles a knowledge reuse framework for combining multiple partitions this paper introduces the problem of combining multiple partitionings of a set of objects into a single consolidated clustering without accessing the features or algorithms that determined these partitionings we first identify several application scenarios for the resultant knowledge reuse framework that we call cluster ensembles the cluster ensemble problem is then formalized as a combinatorial optimization problem in terms of shared mutual information in addition to a direct maximization approach we propose three effective and efficient techniques for obtaining high quality combiners consensus functions the first combiner induces a similarity measure from the partitionings and then reclusters the objects the second combiner is based on hypergraph partitioning the third one collapses groups of clusters into meta clusters which then compete for each object to determine the combined clustering due to the low computational costs of our techniques it is quite feasible to use a supra consensus function that evaluates all three approaches against the objective function and picks the best solution for a given situation we evaluate the effectiveness of cluster ensembles in three qualitatively different application scenarios i where the original clusters were formed based on non identical sets of features ii where the original clustering algorithms worked on non identical sets of objects and iii where a common data set is used and the main purpose of combining multiple clusterings is to improve the quality and robustness of the solution promising results are obtained in all three situations for synthetic as well as real data sets

revisiting the nystrom method for improved large scale machine learning we reconsider randomized algorithms for the low rank approximation of symmetric positive semi definite spsd matrices such as laplacian and kernel matrices that arise in data analysis and machine learning applications our main results consist of an empirical evaluation of the performance quality and running time of sampling and projection methods on a diverse suite of spsd matrices our results highlight complementary aspects of sampling versus projection methods they characterize the effects of common data preprocessing steps on the performance of these algorithms and they point to important differences between uniform sampling and nonuniform sampling methods based on leverage scores in addition our empirical results illustrate that existing theory is so weak that it does not provide even a qualitative guide to practice thus we complement our empirical results with a suite of worst case theoretical bounds for both random sampling and random projection methods these bounds are qualitatively superior to existing bounds e g improved additive error bounds for spectral and frobenius norm error and relative error bounds for trace norm error and they point to future directions to make these algorithms useful in even larger scale machine learning applications