a gibbs sampler for learning dags we propose a gibbs sampler for structure learning in directed acyclic graph dag models the standard markov chain monte carlo algorithms used for learning dags are random walk metropolis hastings samplers these samplers are guaranteed to converge asymptotically but often mix slowly when exploring the large graph spaces that arise in structure learning in each step the sampler we propose draws entire sets of parents for multiple nodes from the appropriate conditional distribution this provides an efficient way to make large moves in graph space permitting faster mixing whilst retaining asymptotic guarantees of convergence the conditional distribution is related to variable selection with candidate parents playing the role of covariates or inputs we empirically examine the performance of the sampler using several simulated and real data examples the proposed method gives robust results in diverse settings outperforming several existing bayesian and frequentist methods in addition our empirical results shed some light on the relative merits of bayesian and constraint based methods for structure learning

large sample learning of bayesian networks is np hard in this paper we provide new complexity results for algorithms that learn discrete variable bayesian networks from data our results apply whenever the learning algorithm uses a scoring criterion that favors the simplest structure for which the model is able to represent the generative distribution exactly our results therefore hold whenever the learning algorithm uses a consistent scoring criterion and is applied to a sufficiently large dataset we show that identifying high scoring structures is np hard even when any combination of one or more of the following hold the generative distribution is perfect with respect to some dag containing hidden variables we are given an independence oracle we are given an inference oracle we are given an information oracle we restrict potential solutions to structures in which each node has at most k parents for all k 3 our proof relies on a new technical result that we establish in the appendices in particular we provide a method for constructing the local distributions in a bayesian network such that the resulting joint distribution is provably perfect with respect to the structure of the network

asymptotics for the minimum covariance determinant estimator consistency is shown for the minimum covariance determinant mcd estimators of multivariate location and scale and asymptotic normality is shown for the former the proofs are made possible by showing a separating ellipsoid property for the mcd subset of observations an analogous property is shown for the mcd subset computed from the population distribution

rank revealing qr factorizations abstract an algorithm is presented for computing a column permutation pi and a qr factorization a pi qr of an m by n m or n matrix a such that a possible rank deficiency of a will be revealed in the triangular factor r having a small lower right block for low rank deficient matrices the algorithm is guaranteed to reveal the rank of a and the cost is only slightly more than the cost of one regular qr factorization a posteriori upper and lower bounds on the singular values of a are derived and can be used to infer the numerical rank of a keywords qr factorization rank deficient matrices least squares computation subset selection rank singular values

nonparametric interval and point prediction using data trimmed by a grubbs type outlier rule for a fixed probability 0 y 1 the most outlying 100 1 y subset of the data from a location model may be located with a grubbs outlier subset test statistic this subset is essentially located in terms of its complement which is the connected looy span of the data which supports the smallest sample variance we show that this range of the data may be characterized approximately as the looy span such that its midpoint is equal to the trimmed mean averaged over the span such a range forms a tolerance interval for predicting a future observation from the location mqdel and the asymptotic laws for its location coverage and center are presented

the complexity of counting cuts and of computing the probability that a graph is connected several enumeration and reliability problems are shown to be p complete and hence at least as hard as np complete problems included are important problems in network reliability analysis namely computing the probability that a graph is connected and counting the number of minimum cardinality s t cuts or directed network cuts also shown to be p complete are counting vertex covers in a bipartite graph counting antichains in a partial order and approximating the probability that a graph is connected and the probability that a pair of vertices is connected

a cross product approach for low rank approximations of large matrices abstract in this paper we present a new type of restarted krylov method for calculating low rank approximations of large matrices in contrast to former krylov methods our approach does not apply the lanczos algorithm or lanczos bidiagonalization this simplifies the basic iteration and allows the introduction of several innovations an advantage of the proposed method is that it requires a minimal amount of computer storage the krylov matrix which is built at each iteration is using an improved starting vector and a new three term recurrence relation these modifications lead to fast rate of convergence numerical experiments illustrate the usefulness of the proposed approach

sampling from large matrices an approach through geometric functional analysis we study random submatrices of a large matrix a we show how to approximately compute a from its random submatrix of the smallest possible size o rlog r with a small error in the spectral norm where r e vav2f vav22 is the numerical rank of a the numerical rank is always bounded by and is a stable relaxation of the rank of a this yields an asymptotically optimal guarantee in an algorithm for computing low rank approximations of a we also prove asymptotically optimal estimates on the spectral norm and the cut norm of random submatrices of a the result for the cut norm yields a slight improvement on the best known sample complexity for an approximation algorithm for max 2csp problems we use methods of probability in banach spaces in particular the law of large numbers for operator valued random variables

a deim induced cur factorization we derive a cur approximate matrix factorization based on the discrete empirical interpolation method deim for a given matrix bf a such a factorization provides a low rank approximate decomposition of the form bf a approx bf c bf u bf r where bf c and bf r are subsets of the columns and rows of bf a and bf u is constructed to make bf c bf u bf r a good approximation given a low rank singular value decomposition bf a approx bf v bf s bf w t the deim procedure uses bf v and bf w to select the columns and rows of bf a that form bf c and bf r through an error analysis applicable to a general class of cur factorizations we show that the accuracy tracks the optimal approximation error within a factor that depends on the conditioning of submatrices of bf v and bf w for very large problems bf v and bf w can be approximated well using an incremental qr algorithm that makes only one pass through bf a numer

gait analysis and the bootstrap this paper is about random coefficient trigonometric regression models and their use in gait analysis here gait analysis means free speed walking on a level surface our study focuses on bootstrap based prediction regions for the angular rotation curves of test children when the relevant training data are gathered from normal children of comparable ages considerations that led to our choice of model and use of the bootstrap are given prediction regions and empirical bootstrap distributions are displayed as is their application to several test cases also included is a study of the almost sure asymptotic behavior of the theoretical bootstrap probability of the prediction regions

a theory of pseudoskeleton approximations abstract let an m n matrix a be approximated by a rank r matrix with an accuracy e we prove that it is possible to choose r columns and r rows of a forming a so called pseudoskeleton component which approximates a with o e r m n accuracy in the sense of the 2 norm on the way to this estimate we study the interconnection between the volume i e the determinant in the absolute value and the minimal singular value r of r r submatrices of an n r matrix with orthogonal columns we propose a lower bound better than one given by chandrasekaran and ipsen and by hong and pan for the maximum of r over all these submatrices and formulate a hypothesis on a tighter bound

fast monte carlo algorithms for matrices iii computing a compressed approximate matrix decomposition in many applications the data consist of or may be naturally formulated as an m times n matrix a which may be stored on disk but which is too large to be read into random access memory ram or to practically perform superlinear polynomial time computations on it two algorithms are presented which when given an m times n matrix a compute approximations to a which are the product of three smaller matrices c u and r each of which may be computed rapidly let a cur be the computed approximate decomposition both algorithms have provable bounds for the error matrix a a in the first algorithm c columns of a and r rows of a are randomly chosen if the m times c matrix c consists of those c columns of a after appropriate rescaling and the r times n matrix r consists of those r rows of a also after appropriate rescaling then the c times r matrix u may be calculated from c and r for any matrix x let x f and x 2 denote its frobenius norm and its spectral norm respectively it is proven that left a a right xi le min d mathrm rank d le k left a d right xi poly k 1 c left a right f holds in expectation and with high probability for both xi 2 f and for all k 1 ldots mbox rank a thus by appropriate choice of k left a a right 2 le epsilon left a right f also holds in expectation and with high probability this algorithm may be implemented without storing the matrix a in ram provided it can make two passes over the matrix stored in external memory and use o m n additional ram assuming that c and r are constants independent of the size of the input the second algorithm is similar except that it approximates the matrix c by randomly sampling a constant number of rows of c thus it has additional error but it can be implemented in three passes over the matrix using only constant additional ram to achieve an additional error beyond the best rank k approximation that is at most epsilon a f both algorithms take time which is a low degree polynomial in k 1 epsilon and 1 delta where delta 0 is a failure probability the first takes time linear in mbox max m n and the second takes time independent of m and n the proofs for the error bounds make important use of matrix perturbation theory and previous work on approximating matrix multiplication and computing low rank approximations to a matrix the probability distribution over columns and rows and the rescaling are crucial features of the algorithms and must be chosen judiciously

fast monte carlo algorithms for finding low rank approximations we consider the problem of approximating a given m n matrix a by another matrix of specified rank k which is smaller than m and n the singular value decomposition svd can be used to find the best such approximation however it takes time polynomial in m n which is prohibitive for some modern applications in this article we develop an algorithm that is qualitatively faster provided we may sample the entries of the matrix in accordance with a natural probability distribution in many applications such sampling can be done efficiently our main result is a randomized algorithm to find the description of a matrix d of rank at most k so that holds with probability at least 1 where v vf is the frobenius norm the algorithm takes time polynomial in k 1 e log 1 only and is independent of m and n in particular this implies that in constant time it can be determined if a given matrix of arbitrary size has a good low rank approximation

near optimal column based matrix reconstruction we consider low rank reconstruction of a matrix using a subset of its columns and present asymptotically optimal algorithms for both spectral norm and frobenius norm reconstruction the main tools

expected computation time for hamiltonian path problem one way to cope with an np hard problem is to find an algorithm that is fact on average with respect to a natural probability distribution on inputs we consider from that point of view the hamilto

sampling methods for the nystrom method the nystrom method is an efficient technique to generate low rank matrix approximations and is used in several large scale learning applications a key aspect of this method is the procedure according to which columns are sampled from the original matrix in this work we explore the efficacy of a variety of fixed and adaptive sampling schemes we also propose a family of ensemble based sampling algorithms for the nystrom method we report results of extensive experiments that provide a detailed comparison of various fixed and adaptive sampling techniques and demonstrate the performance improvement associated with the ensemble nystrom method when used in conjunction with either fixed or adaptive sampling schemes corroborating these empirical findings we present a theoretical analysis of the nystrom method providing novel error bounds guaranteeing a better convergence rate of the ensemble nystrom method in comparison to the standard nystrom method

finding structure with randomness probabilistic algorithms for constructing approximate matrix decompositions low rank matrix approximations such as the truncated singular value decomposition and the rank revealing qr decomposition play a central role in data analysis and scientific computing this work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low rank matrix approximation these techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets this paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions these methods use random sampling to identify a subspace that captures most of the action of a matrix the input matrix is then compressed either explicitly or implicitly to this subspace and the reduced matrix is manipulated deterministically to obtain the desired low rank factorization in many cases this approach beats its classical competitors in terms of accuracy robustness and or speed these claims are supported by extensive numerical experiments and a detailed error analysis the specific benefits of randomized techniques depend on the computational environment consider the model problem of finding the k dominant components of the singular value decomposition of an m times n matrix i for a dense input matrix randomized algorithms require bigo mn log k floating point operations flops in contrast to bigo mnk for classical algorithms ii for a sparse input matrix the flop count matches classical krylov subspace methods but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures iii for a matrix that is too large to fit in fast memory the randomized techniques require only a constant number of passes over the data as opposed to bigo k passes for classical algorithms in fact it is sometimes possible to perform matrix approximation with a single pass over the data

towards more efficient spsd matrix approximation and cur matrix decomposition symmetric positive semi definite spsd matrix approximation methods have been extensively used to speed up large scale eigenvalue computation and kernel learning methods the standard sketch based method which we call the prototype model produces relatively accurate approximations but is inefficient on large square matrices the nystrom method is highly efficient but can only achieve low accuracy in this paper we propose a novel model that we call the fast spsd matrix approximation model the fast model is nearly as efficient as the nystrom method and as accurate as the prototype model we show that the fast model can potentially solve eigenvalue problems and kernel learning problems in linear time with respect to the matrix size n to achieve 1 e relative error whereas both the prototype model and the nystrom method cost at least quadratic time to attain comparable error bound empirical comparisons among the prototype model the nystrom method and our fast model demonstrate the superiority of the fast model we also contribute new understandings of the nystrom method the nystrom method is a special instance of our fast model and is approximation to the prototype model our technique can be straightforwardly applied to make the cur matrix decomposition more efficiently computed without much affecting the accuracy

fast computation of low rank matrix approximations given a matrix a it is often desirable to find a good approximation to a that has low rank we introduce a simple technique for accelerating the computation of such approximations when a has strong spectral features that is when the singular values of interest are significantly greater than those of a random matrix with size and entries similar to a our technique amounts to independently sampling and or quantizing the entries of a thus speeding up computation by reducing the number of nonzero entries and or the length of their representation our analysis is based on observing that the acts of sampling and quantization can be viewed as adding a random matrix n to a whose entries are independent random variables with zero mean and bounded variance since with high probability n has very weak spectral features we can prove that the effect of sampling and quantization nearly vanishes when a low rank approximation to a p n is computed we give high probability bounds on the quality of our approximation both in the frobenius and the 2 norm

forecasting using random subspace methods textabstractrandom subspace methods are a novel approach to obtain accurate forecasts in high dimensional regression settings we provide a theoretical justification of the use of random subspace methods and show their usefulness when forecasting monthly macroeconomic variables we focus on two approaches the first is random subset regression where random subsets of predictors are used to construct a forecast second we discuss random projection regression where artificial predictors are formed by randomly weighting the original predictors using recent results from random matrix theory we obtain a tight bound on the mean squared forecast error for both randomized methods we identify settings in which one randomized method results in more precise forecasts than the other and than alternative regularization strategies such as principal component regression partial least squares lasso and ridge regression the predictive accuracy on the high dimensional macroeconomic fred md data set increases substantially when using the randomized methods with random subset regression outperforming any one of the above mentioned competing methods for at least 66 of the series

probability centered prediction regions consider the problem of constructing a prediction region dn for a potentially observable variable x on the basis of a learning sample of size n usually the requirement that dn contain x with probability a conditionally on the learning sample does not uniquely determine dn this paper develops a general probability centering concept for prediction regions that extends to vector valued or function valued x the classical notion of an equal tailed prediction interval the dual requirements of probability centering and specified coverage probability determine dn uniquely several examples illustrate the scope and consequences of the proposed centering concept

generalized low rank approximations of matrices revisited compared to singular value decomposition svd generalized low rank approximations of matrices glram can consume less computation time obtain higher compression ratio and yield competitive classification performance glram has been successfully applied to applications such as image compression and retrieval and quite a few extensions have been successively proposed however in literature some basic properties and crucial problems with regard to glram have not been explored or solved yet for this sake we revisit glram in this paper first we reveal such a close relationship between glram and svd that glram s objective function is identical to svd s objective function except the imposed constraints second we derive a lower bound of glram s objective function and discuss when the lower bound can be touched moreover from the viewpoint of minimizing the lower bound we answer one open problem raised by ye machine learning 2005 i e a theoretical justification of the experimental phenomenon that under given number of reduced dimension the lowest reconstruction error is obtained when the left and right transformations have equal number of columns third we explore when and why glram can perform well in terms of compression which is a fundamental problem concerning the usability of glram

efficient low rank approximation of matrices based on randomized pivoted decomposition given a matrix bf a with numerical rank k the two sided orthogonal decomposition tsod computes a factorization bf a bf udv t where bf u and bf v are orthogonal and bf d is upper lower triangular tsod is rank revealing as the middle factor bf d reveals the rank of bf a the computation of tsod however is demanding in this paper we present an algorithm called randomized pivoted tsod rp tsod where the middle factor is lower triangular key in our work is the exploitation of randomization and rp tsod is primarily devised to efficiently construct an approximation to a low rank matrix we provide three different types of bounds for rp tsod i we furnish upper bounds on the error of the low rank approximation ii we bound the k approximate principal singular values and iii we derive bounds for the canonical angles between the approximate and the exact singular subspaces our bounds describe the characteristics and behavior of our proposed algorithm through numerical tests we show the effectiveness of the devised bounds as well as our proposed algorithm

approximating the minimum spanning tree weight in sublinear time we present a probabilistic algorithm that given a connected graph g represented by adjacency lists of average degree d with edge weights in the set 1 w and given a parameter 0 eps 1 2 estimates in time o dw varepsilon 2 log frac dw varepsilon the weight of the minimum spanning tree mst of g with a relative error of at most eps note that the running time does not depend on the number of vertices in g we also prove a nearly matching lower bound of omega dw varepsilon 2 on the probe and time complexity of any approximation algorithm for mst weight the essential component of our algorithm is a procedure for estimating in time o d eps 2 log frac d varepsilon the number of connected components of an unweighted graph to within an additive error of varepsilon n this becomes o eps 2 log frac 1 varepsilon for d o 1 the time bound is shown to be tight up to within the log frac d varepsilon factor our connected components algorithm picks o 1 varepsilon 2 vertices in the graph and then grows local spanning trees whose sizes are specified by a stochastic process from the local information collected in this way the algorithm is able to infer with high confidence an estimate of the number of connected components we then show how estimates on the number of components in various subgraphs of g can be used to estimate the weight of its mst

matrix completion from noisy entries given a matrix m of low rank we consider the problem of reconstructing it from noisy observations of a small random subset of its entries the problem arises in a variety of applications from collaborative filtering the netflix problem to structure from motion and positioning we study a low complexity algorithm introduced by keshavan montanari and oh 2010 based on a combination of spectral techniques and manifold optimization that we call here optspace we prove performance guarantees that are order optimal in a number of circumstances

territorial design optimization for business sales plan abstract a well designed territory enhances customer coverage increases sales fosters fair performance and rewards systems and lower travel cost this paper considers a real life case study to design the sales territory for a business sales plan the business plan consist of assigning the optimal quantity of sellers to a territory including the scheduling and routing plans for each seller the problem is formulated as a combination of assignment scheduling and routing optimization models the solution approach considers a meta heuristic using stochastic iterative projection method for large systems several real life instances of different sizes were tested with stochastic data to represent raise fall in the customers demand as well as the appearance loss of customers

approximating the radii of point sets we consider the problem of computing the outer radii of point sets in this problem we are given integers n d and k where k le d and a set p of n points in re d the goal is to compute the outer k radius of p denoted by cal r k p which is the minimum over all d k dimensional flats f of max p in p d p f where d p f is the euclidean distance between the point p and flat f computing the radii of point sets is a fundamental problem in computational convexity with many significant applications the problem admits a polynomial time algorithm when the dimension d is constant u faigle w kern and m streng math program 73 1996 pp 1 5 here we are interested in the general case in which the dimension d is not fixed and can be as large as n where the problem becomes np hard even for k 1 it is known that r k p can be approximated in polynomial time by a factor of 1 varepsilon for any varepsilon 0 when d k is a fixed constant m babdoiu s har peled and p indyk in proceedings of the acm symposium on the theory of computing 2002 s har peled and k varadarajan in proceedings of the acm symposium on computing geometry 2002 a polynomial time algorithm that guarantees a factor of o sqrt log n approximation for r 1 p the width of the point set p is implied by the results of nemirovski roos and terlaky math program 86 1999 pp 463 473 and nesterov handbook of semidefinite programming theory algorithms kluwer academic publishers norwell ma 2000 in this paper we show that r k p can be approximated by a ratio of o sqrt log n for any 1 leq k leq d thus matching the previously best known ratio for approximating the special case r 1 p the width of point set p our algorithm is based on semidefinite programming relaxation with a new mixed deterministic and randomized rounding procedure we also prove an inapproximability result that gives evidence that our approximation algorithm is doing well for a large range of k we show that there exists a constant delta 0 such that the following holds for any 0 eps 1 there is no polynomial time algorithm that approximates r k p within log n delta for all k such that k leq d d varepsilon unless np subseteq dtime 2 log m o 1 our inapproximability result for r k p extends a previously known hardness result of brieden discrete comput geom 28 2002 pp 201 209 and is proved by modifying brieden s construction using basic ideas from probabilistically checkable proofs pcp theory

matrix completion from a few entries let m be an n n matrix of rank r and assume that a uniformly random subset e of its entries is observed we describe an efficient algorithm which we call optspace that reconstructs m from e o rn observed entries with relative root mean square error 1 2 rmse c nr e 1 2 with probability larger than 1 1 n3 further if r o 1 and m is sufficiently unstructured then optspace reconstructs it exactly from e o n log n entries with probability larger than 1 1 n3 this settles in the case of bounded rank a question left open by candes and recht and improves over the guarantees for their reconstruction algorithm the complexity of our algorithm is o e r log n which opens the way to its use for massive data sets in the process of proving these statements we obtain a generalization of a celebrated result by friedman kahn szemeredi and feige ofek on the spectrum of sparse random matrices

clustered nystrom method for large scale manifold learning and dimension reduction kernel or similarity matrix plays a key role in many machine learning algorithms such as kernel methods manifold learning and dimension reduction however the cost of storing and manipulating the complete kernel matrix makes it infeasible for large problems the nystrom method is a popular sampling based low rank approximation scheme for reducing the computational burdens in handling large kernel matrices in this paper we analyze how the approximating quality of the nystrom method depends on the choice of landmark points and in particular the encoding powers of the landmark points in summarizing the data our non probabilistic error analysis justifies a clustered nystrom method that uses the k means clustering centers as landmark points our algorithm can be applied to scale up a wide variety of algorithms that depend on the eigenvalue decomposition of kernel matrix or its variant such as kernel principal component analysis laplacian eigenmap spectral clustering as well as those involving kernel matrix inverse such as least squares support vector machine and gaussian process regression extensive experiments demonstrate the competitive performance of our algorithm in both accuracy and efficiency

calibrating prediction regions abstract suppose that the variable x to be predicted and the learning sample yn that was observed have a joint distribution which depends on an unknown parameter the parameter can be finite or infinite dimensional a prediction region dn for x is a random set depending on yn that contains x with prescribed probability this article studies methods for controlling simultaneously the conditional coverage proability of dn given yn and the overall unconditional coverage probability of dn the basic construction yields a prediction region dn which has the following properties in regular models both the conditional and overall coverage probabilities of dn converge to as the size n of the learning sample increases the convergence of the former is in probability moreover the asymptotic distribution of the conditional coverage probability about is typically normal and the overall coverage probability tends to at rate n 1 can one reduce the dispersion of the conditional coverage pr

a scoring function for learning bayesian networks based on mutual information and conditional independence tests we propose a new scoring function for learning bayesian networks from data using score search algorithms this is based on the concept of mutual information and exploits some well known properties of this measure in a novel way essentially a statistical independence test based on the chi square distribution associated with the mutual information measure together with a property of additive decomposition of this measure are combined in order to measure the degree of interaction between each variable and its parent variables in the network the result is a non bayesian scoring function called mit mutual information tests which belongs to the family of scores based on information theory the mit score also represents a penalization of the kullback leibler divergence between the joint probability distributions associated with a candidate network and with the available data set detailed results of a complete experimental evaluation of the proposed scoring function and its comparison with the well known k2 bdeu and bic mdl scores are also presented

efficient structure learning of bayesian networks using constraints this paper addresses the problem of learning bayesian network structures from data based on score functions that are decomposable it describes properties that strongly reduce the time and memory costs of many known methods without losing global optimality guarantees these properties are derived for different score criteria such as minimum description length or bayesian information criterion akaike information criterion and bayesian dirichlet criterion then a branch and bound algorithm is presented that integrates structural constraints with data in a way to guarantee global optimality as an example structural constraints are used to map the problem of structure learning in dynamic bayesian networks into a corresponding augmented bayesian network finally we show empirically the benefits of using the properties with state of the art methods and with the new algorithm which is able to handle larger data sets than before

exact bayesian structure discovery in bayesian networks learning a bayesian network structure from data is a well motivated but computationally hard task we present an algorithm that computes the exact posterior probability of a subnetwork e g a directed edge a modified version of the algorithm finds one of the most probable network structures this algorithm runs in time o n 2n nk 1c m where n is the number of network variables k is a constant maximum in degree and c m is the cost of computing a single local marginal conditional likelihood for m data instances this is the first algorithm with less than super exponential complexity with respect to n exact computation allows us to tackle complex cases where existing monte carlo methods and local search procedures potentially fail we show that also in domains with a large number of variables exact computation is feasible given suitable a priori restrictions on the structures combining exact and inexact methods is also possible we demonstrate the applicability of the presented algorithm on four synthetic data sets with 17 22 37 and 100 variables

bootstrap prediction intervals for regression abstract bootstrap prediction intervals provide a nonparametric measure of the probable error of forecasts from a standard linear regression model these intervals approximate the nominal probability content in small samples without requiring specific assumptions about the sampling distribution empirical measures of the prediction error rate motivate the choice of these intervals which are calculated by an application of the bootstrap the intervals are contrasted to other nonparametric procedures in several monte carlo experiments asymptotic invariance properties are also investigated

low rank approximation and regression in input sparsity time we design a new distribution over m n matrices s so that for any fixed n d matrix a of rank r with probability at least 9 10 psaxp2 e 1 e paxp2 simultaneously for all x rd here m is bounded by a polynomial in re 1 and the parameter e 0 1 such a matrix s is called a subspace embedding furthermore sa can be computed in o nnz a time where nnz a is the number of nonzero entries of a this improves over all previous subspace embeddings for which computing sa required at least ndlog d time we call these s sparse embedding matrices using our sparse embedding matrices we obtain the fastest known algorithms for overconstrained least squares regression low rank approximation approximating all leverage scores and ep regression more specifically let b be an n 1 vector e 0 a small enough value and integers k p g 1 our results include the following regression the regression problem is to find d 1 vector x for which pax bpp l 1 e min xpax bpp for the euclidean case p e 2 we obtain an algorithm running in o nnz a o d3e 2 time and another in o nnz a log 1 e o d3 log 1 e time here o f e f log o 1 f for p 1 more generally we obtain an algorithm running in o nnz a log n o r e 1 c time for a fixed c low rank approximation we give an algorithm to obtain a rank k matrix k such that pa kpf 1 e p a akpf where ak is the best rank k approximation to a that is ak is the output of principal components analysis produced by a truncated singular value decomposition useful for latent semantic indexing and many other statistical problems our algorithm runs in o nnz a o nk2e 4 k3e 5 time leverage scores we give an algorithm to estimate the leverage scores of a up to a constant factor in o nnz a log n o r3 time

balanced simultaneous confidence sets abstract suppose that t tu is a family of parametric functions indexed by the variable u for each u an approximate confidence set cn u for tu may be obtained by referring a function of tu and the sample to an estimated quantile dn u of that function s sampling distribution this approach is sometimes called the pivotal method for constructing a confidence set even when it is not based on a true pivot a simultaneous confidence set cn for t is then obtained by simultaneously asserting the individual confidence sets cn u the problem is to choose the critical values for the cn u in such a way that the overall coverage probability of cn is correct and the coverage probabilities of the individual confidence sets cn u are equal the second property is termed balance it means that the simultaneous confidence set cn treats each constituent confidence statement cn u fairly aside from a few special cases the problem just described is too difficult for analytical approaches whe

efficient algorithms for computing a strong rank revealing qr factorization given anm n matrixm withm n it is shown that there exists a permutation fi and an integer k such that the qr factorization myi q ak ckbk reveals the numerical rank of m the k k upper triangular matrix ak is well conditioned ilckll2 is small and bk is linearly dependent on ak with coefficients bounded by a low degree polynomial in n existing rank revealing qr rrqr algorithms are related to such factorizations and two algorithms are presented for computing them the new algorithms are nearly as efficient as qr with column pivoting for most problems and take o ran2 floating point operations in the worst case

spsd matrix approximation vis column selection theories algorithms and extensions symmetric positive semidefinite spsd matrix approximation is an important problem with applications in kernel methods however existing spsd matrix approximation methods such as the nystrom method only have weak error bounds in this paper we conduct in depth studies of an spsd matrix approximation model and establish strong relative error bounds we call it the prototype model for it has more efficient and effective extensions and some of its extensions have high scalability though the prototype model itself is not suitable for large scale data it is still useful to study its properties on which the analysis of its extensions relies this paper offers novel theoretical analysis efficient algorithms and a highly accurate extension first we establish a lower error bound for the prototype model and improve the error bound of an existing column selection algorithm to match the lower bound in this way we obtain the first optimal column selection algorithm for the prototype model we also prove that the prototype model is exact under certain conditions second we develop a simple column selection algorithm with a provable error bound third we propose a so called spectral shifting model to make the approximation more accurate when the eigenvalues of the matrix decay slowly and the improvement is theoretically quantified the spectral shifting method can also be applied to improve other spsd matrix approximation models

improving cur matrix decomposition and the nystrom approximation via adaptive sampling the cur matrix decomposition and the nystrom approximation are two important low rank matrix approximation techniques the nystrom method approximates a symmetric positive semidefinite matrix in terms of a small number of its columns while cur approximates an arbitrary data matrix by a small number of its columns and rows thus cur decomposition can be regarded as an extension of the nystrom approximation in this paper we establish a more general error bound for the adaptive column row sampling algorithm based on which we propose more accurate cur and nystrom algorithms with expected relative error bounds the proposed cur and nystrom algorithms also have low time complexity and can avoid maintaining the whole data matrix in ram in addition we give theoretical analysis for the lower error bounds of the standard nystrom method and the ensemble nystrom method the main theoretical results established in this paper are novel and our analysis makes no special assumption on the data matrices

efficient denoising algorithms for large experimental datasets and their applications in fourier transform ion cyclotron resonance mass spectrometry modern scientific research produces datasets of increasing size and complexity that require dedicated numerical methods to be processed in many cases the analysis of spectroscopic data involves the denoising of raw data before any further processing current efficient denoising algorithms require the singular value decomposition of a matrix with a size that scales up as the square of the data length preventing their use on very large datasets taking advantage of recent progress on random projection and probabilistic algorithms we developed a simple and efficient method for the denoising of very large datasets based on the qr decomposition of a matrix randomly sampled from the data this approach allows a gain of nearly three orders of magnitude in processing time compared with classical singular value decomposition denoising this procedure called urqrd uncoiled random qr denoising strongly reduces the computer memory footprint and allows the denoising algorithm to be applied to virtually unlimited data size the efficiency of these numerical tools is demonstrated on experimental data from high resolution broadband fourier transform ion cyclotron resonance mass spectrometry which has applications in proteomics and metabolomics we show that robust denoising is achieved in 2d spectra whose interpretation is severely impaired by scintillation noise these denoising procedures can be adapted to many other data analysis domains where the size and or the processing time are crucial

finding optimal bayesian networks using precedence constraints we consider the problem of finding a directed acyclic graph dag that optimizes a decomposable bayesian network score while in a favorable case an optimal dag can be found in polynomial time in the worst case the fastest known algorithms rely on dynamic programming across the node subsets taking time and space 2n to within a factor polynomial in the number of nodes n in practice these algorithms are feasible to networks of at most around 30 nodes mainly due to the large space requirement here we generalize the dynamic programming approach to enhance its feasibility in three dimensions first the user may trade space against time second the proposed algorithms easily and efficiently parallelize onto thousands of processors third the algorithms can exploit any prior knowledge about the precedence relation on the nodes underlying all these results is the key observation that given a partial order p on the nodes an optimal dag compatible with p can be found in time and space roughly proportional to the number of ideals of p which can be significantly less than 2n considering sufficiently many carefully chosen partial orders guarantees that a globally optimal dag will be found aside from the generic scheme we present and analyze concrete tradeoff schemes based on parallel bucket orders

orthogonal optimal reverse prediction for semi supervised learning optimal reverse prediction orp has recently been proposed as a semi supervised framework to unify supervised and unsupervised training methods such as supervised least square principal component analysis pca k means clustering and normalized graph cut orp has an ability to deal with classification tasks in which the labeled data are insufficient but the performance of orp and its kernelized version is still not satisfactory for classification applications to further improve performance of orp motivated by recently proposed orthogonal k means clustering in this paper we propose an orthogonal optimal reverse prediction oorp together with its kernelized and laplacian regularized extensions with only limited additional computations our algorithms can greatly enhance the classification performance compared to the original orp extensive experiments on synthetic and benchmark data collections consistently prove the effectiveness and efficiency of our oorp in comparison with several competing approaches highlightswe imposed the column orthogonal constraints to the mapping operator and propose the orthogonal optimal reverse prediction algorithm which shows a better performance than the original algorithm we imposed the orthogonal constraints method to the kernelized optimal reverse prediction algorithm and proposed the kernelized orthogonal optimal reverse prediction an optimization algorithm is designed to solve the problem and the model s classification performance is encouraging we also extend the idea to the laplacian orthogonal optimal reverse prediction and get a better classification performance

subspace iteration randomization and singular value problems a classical problem in matrix computations is the efficient and reliable approximation of a given matrix by a matrix of lower rank the truncated singular value decomposition svd is known to provide the best such approximation for any given fixed rank however the svd is also known to be very costly to compute among the different approaches in the literature for computing low rank approximations randomized algorithms have attracted researchers attention recently due to their surprising reliability and computational efficiency in different application areas typically such algorithms are shown to compute with very high probability low rank approximations that are within a constant factor from optimal and are known to perform even better in many practical situations in this paper we present a novel error analysis that considers randomized algorithms within the subspace iteration framework and show with very high probability that highly accurate low rank approximations as well as singular values ca

importance sampling for a monte carlo matrix multiplication algorithm with application to information retrieval we perform importance sampling for a randomized matrix multiplication algorithm by drineas kannan and mahoney and derive probabilities that minimize the expected value with regard to the distributions of the matrix elements of the variance we compare these optimized probabilities with uniform probabilities and derive conditions under which the actual variance of the optimized probabilities is lower numerical experiments with query matching in information retrieval applications illustrate that the optimized probabilities produce more accurate matchings than the uniform probabilities and that they can also be computed efficiently

a randomized algorithm for a tensor based generalization of the singular value decomposition an algorithm is presented and analyzed that when given as input a d mode tensor a computes an approximation a the approximation a is computed by performing the following for each of the d modes first form implicitly a matrix by unfolding the tensor along that mode then choose columns from the matrices thus generated and finally project the tensor along that mode onto the span of those columns an important issue affecting the quality of the approximation is the choice of the columns from the matrices formed by unfolding the tensor along each of its modes in order to address this issue two algorithms of independent interest are presented that given an input matrix a and a target rank k select columns that span a space close to the best rank k subspace of the matrix for example in one of the algorithms a number c that depends on k an error parameter and a failure probability of columns are chosen in c independent random trials according to a nonuniform probability distribution depending on the euclidean lengths of the columns when this algorithm for choosing columns is used in the tensor approximation then under appropriate assumptions bounds of the form

randomized interpolative decomposition of separated representations we introduce an algorithm to compute tensor interpolative decomposition dubbed ctd id for the reduction of the separation rank of canonical tensor decompositions ctds tensor id selects for a user defined accuracy a near optimal subset of terms of a ctd to represent the remaining terms via a linear combination of the selected terms ctd id can be used as an alternative to or in combination with the alternating least squares als algorithm we present examples of its use within a convergent iteration to compute inverse operators in high dimensions we also briefly discuss the spectral norm as a computational alternative to the frobenius norm in estimating approximation errors of tensor id we reduce the problem of finding tensor ids to that of constructing interpolative decompositions of certain matrices these matrices are generated via randomized projection of the terms of the given tensor we provide cost estimates and several examples of the new approach to the reduction of separation rank

structure discovery in bayesian networks by sampling partial orders we present methods based on metropolis coupled markov chain monte carlo mc3 and annealed importance sampling ais for estimating the posterior distribution of bayesian networks the methods draw samples from an appropriate distribution of partial orders on the nodes continued by sampling directed acyclic graphs dags conditionally on the sampled partial orders we show that the computations needed for the sampling algorithms are feasible as long as the encountered partial orders have relatively few down sets while the algorithms assume suitable modularity properties of the priors arbitrary priors can be handled by dividing the importance weight of each sampled dag by the number of topological sorts it has we give a practical dynamic programming algorithm to compute these numbers our empirical results demonstrate that the presented partial order based samplers are superior to previous markov chain monte carlo methods which sample dags either directly or via linear orders on the nodes the results also suggest that the convergence rate of the estimators based on ais are competitive to those of mc3 thus ais is the preferred method as it enables easier large scale parallelization and in addition supplies good probabilistic lower bound guarantees for the marginal likelihood of the model

rank and null space calculations using matrix decomposition without column interchanges abstract the most widely used stable methods for numerical determination of the rank of a matrix a are the singular value decomposition and the qr algorithm with column interchanges here two algorithms are presented which determine rank and nullity in a numerically stable manner without using column interchanges one algorithm makes use of the condition estimator of cline moler stewart and wilkinson and relative to alternative stable algorithms is particularly efficient for sparse matrices the second algorithm is important in the case that one wishes to test for rank and nullity while sequentially adding columns to a matrix

four algorithms for the the efficient computation of truncated pivoted qr approximations to a sparse matrix in this paper we propose four algorithms to compute truncated pivoted qr approximations to a sparse matrix three are based on the gram schmidt algorithm and the other on householder triangularization all four algorithms leave the original matrix unchanged and the only additional storage requirements are arrays to contain the factorization itself thus the algorithms are particularly suited to determining low rank approximations to a sparse matrix

on the nystrom method for approximating a gram matrix for improved kernel based learning a problem for many kernel based methods is that the amount of computation required to find the solution scales as o n3 where n is the number of training examples we develop and analyze an algorithm to compute an easily interpretable low rank approximation to an n n gram matrix g such that computations of interest may be performed more rapidly the approximation is of the form gk cwk ct where c is a matrix consisting of a small number c of columns of g and wk is the best rank k approximation to w the matrix formed by the intersection between those c columns of g and the corresponding c rows of g an important aspect of the algorithm is the probability distribution used to randomly sample the columns we will use a judiciously chosen and data dependent nonuniform probability distribution let 2 and f denote the spectral norm and the frobenius norm respectively of a matrix and let gk be the best rank k approximation to g we prove that by choosing o k e4 columns g cwk ct g gk e i 1n gii2 both in expectation and with high probability for both 2 f and for all k 0 k rank w this approximation can be computed using o n additional space and time after making two passes over the data from external storage the relationships between this algorithm other related matrix decompositions and the nystrom method from integral equation theory are discussed

subspace orbit randomized decomposition for low rank matrix approximations an efficient accurate and reliable approximation of a matrix by one of lower rank is a fundamental task in numerical linear algebra and signal processing applications in this paper we introduce a new matrix decomposition approach termed subspace orbit randomized singular value decomposition sor svd which makes use of random sampling techniques to give a low rank approximation to an input matrix given a large and dense data matrix of size m times n the algorithm requires a few passes through data to compute a rank k approximation in o mnk floating point operations moreover the sor svd algorithm can utilize advanced computer architectures and as a result it can be optimized for maximum efficiency the sor svd algorithm is simple accurate and provably correct and outperforms previously reported techniques in terms of accuracy and efficiency our numerical experiments support these claims

revisiting the nystrom method for improved large scale machine learning we reconsider randomized algorithms for the low rank approximation of symmetric positive semi definite spsd matrices such as laplacian and kernel matrices that arise in data analysis and machine learning applications our main results consist of an empirical evaluation of the performance quality and running time of sampling and projection methods on a diverse suite of spsd matrices our results highlight complementary aspects of sampling versus projection methods they characterize the effects of common data preprocessing steps on the performance of these algorithms and they point to important differences between uniform sampling and nonuniform sampling methods based on leverage scores in addition our empirical results illustrate that existing theory is so weak that it does not provide even a qualitative guide to practice thus we complement our empirical results with a suite of worst case theoretical bounds for both random sampling and random projection methods these bounds are qualitatively superior to existing bounds e g improved additive error bounds for spectral and frobenius norm error and relative error bounds for trace norm error and they point to future directions to make these algorithms useful in even larger scale machine learning applications

fast direct methods for gaussian processes a number of problems in probability and statistics can be addressed using the multivariate normal gaussian distribution in the one dimensional case computing the probability for a given mean and variance simply requires the evaluation of the corresponding gaussian density in the n dimensional setting however it requires the inversion of an n times n covariance matrix c as well as the evaluation of its determinant det c in many cases such as regression using gaussian processes the covariance matrix is of the form c sigma 2 i k where k is computed using a specified covariance kernel which depends on the data and additional parameters hyperparameters the matrix c is typically dense causing standard direct methods for inversion and determinant evaluation to require mathcal o n 3 work this cost is prohibitive for large scale modeling here we show that for the most commonly used covariance functions the matrix c can be hierarchically factored into a product of block low rank updates of the identity matrix yielding an mathcal o n log 2 n algorithm for inversion more importantly we show that this factorization enables the evaluation of the determinant det c permitting the direct calculation of probabilities in high dimensions under fairly broad assumptions on the kernel defining k our fast algorithm brings many problems in marginalization and the adaptation of hyperparameters within practical reach using a single cpu core the combination of nearly optimal scaling in terms of problem size with high performance computing resources will permit the modeling of previously intractable problems we illustrate the performance of the scheme on standard covariance kernels

rank revealing qr factorizations and the singular value decomposition t chan has noted that even when the singular value decomposition of a matrix a is known it is still not obvious how to find a rank revealing qr factorization rrqr of a if a has numerical rank deficiency this paper offers a constructive proof of the existence of the rrqr factorization of any matrix a of size m x n with numerical rank r the bounds derived in this paper that guarantee the existence of rrqr are all of order f i in comparison with chan s 0 2 it has been known for some time that if a is only numerically rank one deficient then the column permutation l7 of a that guarantees a small rnn in the qr factorization of a n can be obtained by inspecting the size of the elements of the right singular vector of a corresponding to the smallest singular value of a to some extent our paper generalizes this well known result we consider the interplay between two important matrix decompositions the singular value decomposition and the qr factorization of a matrix a in particular we are interested in the case when a is singular or nearly singular it is well known that for any a e r m x n a real matrix with rn rows and n columns where without loss of generality we assume rn n there are orthogonal matrices u and v such that where c is a diagonal matrix with nonnegative diagonal elements we assume that a 2 a2 2 2 on 2 0 the decomposition 0 1 is the singular value decomposition svd of a and the ai are the singular values of a the columns of v are the right singular vectors of a and the columns of u are the left singular vectors of a mathematically in terms of the singular values received december 1 1990 revised february 8 199 1 199 1 mathematics subject classification primary 65f30 15a23 15a42 15a15