adaptive importance sampling monte carlo simulation for general multivariate probability laws we establish a parametric adaptive importance sampling variance reduction method for general multivariate probability laws employing the principle of bypass distributions makes it possible to develop adaptive algorithms without relying on particular properties of the target and proposal laws both of which in the proposed framework are as general as the uniform law on the unit hypercube without changing the sampling distribution at each iteration we establish the asymptotic normality of the estimator of the desired mean and of the importance sampling parameter as the number of observations tends to infinity although implementation of the proposed methodology requires a small amount of initial work it has the potential to yield substantial improvements in estimator efficiency in various general problem settings to illustrate the applicability and effectiveness we provide numerical results throughout in which we apply exponential and normal bypass distributions as well as demonstrate that well known adaptive importance sampling formulations in the literature can be easily rewritten in the proposed framework

entropy based importance measure for uncertain model inputs in the information theory entropy is a measure of the uncertainty associated with a random variable to identify the important variables of a system subjected to many random variables a novel entropy based importance measure for random variables is proposed in this paper the method evaluates the effect of a random variable on the output responses by calculating the change of the entropy value this technique focuses on the influence of input uncertainty on the entropy of the output responses and can be easily extended to the case taking the correlations among the random variables into consideration besides the effect of a random variable within any partial region of interest can also be evaluated by the proposed global sensitivity indicator the mathematical properties of the proposed importance measure are investigated and proven in detail three simple numerical examples are employed to demonstrate the applicability of the new importance measure and then a probabilistic risk assessment model is us

nonparametric importance sampling abstract importance sampling is a widely used variance reduction simulation technique for the evaluation of high dimensional integrals a key step in the implementation of importance sampling is to choose a proper distribution function from which pseudorandom numbers are generated parametric sampling distributions if available at all are often inadequate for high dimensional integrals over irregular regions one possible remedy is to use a nonparametric method to estimate the unknown optimal sampling function we show that the nonparametric approach yields integral estimates that converge faster than estimates obtained from parametric approaches we also demonstrate that an adaptive method which has been used successfully in parametric settings does not yield better results than simple one step methods in the nonparametric setting

tabulation based 5 independent hashing with applications to linear probing and second moment estimation in the framework of wegman and carter a k independent hash function maps any k keys independently it is known that 5 independent hashing provides good expected performance in applications such as linear probing and second moment estimation for data streams the classic 5 independent hash function evaluates a degree 4 polynomial over a prime field containing the key domain n 0 ldots n 1 here we present an efficient 5 independent hash function that uses no multiplications instead for any parameter c we make 2c 1 lookups in tables of size o n 1 c in experiments on different computers our scheme gained factors of 1 8 to 10 in speed over the polynomial method we also conducted experiments on the performance of hash functions inside the above applications in particular we give realistic examples of inputs that make the most popular 2 independent hash function perform quite poorly this illustrates the advantage of using schemes with provably good expected performance for all inputs

importance sampling for stochastic simulations importance sampling is one of the classical variance reduction techniques for increasing the efficiency of monte carlo algorithms for estimating integrals the basic idea is to replace the original random mechanism in the simulation by a new one and at the same time modify the function being integrated in this paper the idea is extended to problems arising in the simulation of stochastic systems discrete time markov chains continuous time markov chains and generalized semi markov processes are covered applications are given to a gi g 1 queueing problem and response surface estimation computation of the theoretical moments arising in importance sampling is discussed and some numerical examples given

near optimal column based matrix reconstruction we consider low rank reconstruction of a matrix using a subset of its columns and present asymptotically optimal algorithms for both spectral norm and frobenius norm reconstruction the main tools

towards more efficient spsd matrix approximation and cur matrix decomposition symmetric positive semi definite spsd matrix approximation methods have been extensively used to speed up large scale eigenvalue computation and kernel learning methods the standard sketch based method which we call the prototype model produces relatively accurate approximations but is inefficient on large square matrices the nystrom method is highly efficient but can only achieve low accuracy in this paper we propose a novel model that we call the fast spsd matrix approximation model the fast model is nearly as efficient as the nystrom method and as accurate as the prototype model we show that the fast model can potentially solve eigenvalue problems and kernel learning problems in linear time with respect to the matrix size n to achieve 1 e relative error whereas both the prototype model and the nystrom method cost at least quadratic time to attain comparable error bound empirical comparisons among the prototype model the nystrom method and our fast model demonstrate the superiority of the fast model we also contribute new understandings of the nystrom method the nystrom method is a special instance of our fast model and is approximation to the prototype model our technique can be straightforwardly applied to make the cur matrix decomposition more efficiently computed without much affecting the accuracy

on large deviations theory and asymptotically efficient monte carlo estimation it is shown how large deviations theory can be applied to construct an asymptotically efficient simulation distribution for monte carlo simulation using the importance sampling technique a sufficient and necessary condition is given for the asymptotic efficiency of the candidate simulation distributions this is done in the multidimensional setting that is required by many practical simulation problems the result obtained is applied primarily in two areas first the generalization of previous work dealing with functionals of markov chains is discussed a second area of application is the simulation of nonlinear systems with gaussian inputs as an example of a system with gaussian inputs the simulation of a digital communication channel with nonlinear isi intersymbol interference characteristic of satellite data links is considered in the case of linear isi the asymptotically efficient exponentially twisted distribution turns out to agree with a method previously proposed by d lu and k yao 1988 the large deviations point of view provides some useful insight on how to extent the lu and yao method to nonlinear channels

finding structure with randomness probabilistic algorithms for constructing approximate matrix decompositions low rank matrix approximations such as the truncated singular value decomposition and the rank revealing qr decomposition play a central role in data analysis and scientific computing this work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low rank matrix approximation these techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets this paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions these methods use random sampling to identify a subspace that captures most of the action of a matrix the input matrix is then compressed either explicitly or implicitly to this subspace and the reduced matrix is manipulated deterministically to obtain the desired low rank factorization in many cases this approach beats its classical competitors in terms of accuracy robustness and or speed these claims are supported by extensive numerical experiments and a detailed error analysis the specific benefits of randomized techniques depend on the computational environment consider the model problem of finding the k dominant components of the singular value decomposition of an m times n matrix i for a dense input matrix randomized algorithms require bigo mn log k floating point operations flops in contrast to bigo mnk for classical algorithms ii for a sparse input matrix the flop count matches classical krylov subspace methods but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures iii for a matrix that is too large to fit in fast memory the randomized techniques require only a constant number of passes over the data as opposed to bigo k passes for classical algorithms in fact it is sometimes possible to perform matrix approximation with a single pass over the data

sampling methods for the nystrom method the nystrom method is an efficient technique to generate low rank matrix approximations and is used in several large scale learning applications a key aspect of this method is the procedure according to which columns are sampled from the original matrix in this work we explore the efficacy of a variety of fixed and adaptive sampling schemes we also propose a family of ensemble based sampling algorithms for the nystrom method we report results of extensive experiments that provide a detailed comparison of various fixed and adaptive sampling techniques and demonstrate the performance improvement associated with the ensemble nystrom method when used in conjunction with either fixed or adaptive sampling schemes corroborating these empirical findings we present a theoretical analysis of the nystrom method providing novel error bounds guaranteeing a better convergence rate of the ensemble nystrom method in comparison to the standard nystrom method

approximating posterior distributions by mixtures kernel density estimation techniques are used to smooth simulated samples from importance sampling function approximations to posterior distributions resulting in revised approximations that are mixtures of standard parametric forms usually multivariate normal or t distributions adaptive refinement of such mixture approximations involves repeating this process to home in successively on the posterior in fairly low dimensional problems this provides a general and automatic method of approximating posteriors by mixtures so that marginal densities and other summaries may be easily computed

computationally efficient nonparametric importance sampling the variance reduction established by importance sampling strongly depends on the choice of the importance sampling distribution a good choice is often hard to achieve especially for high dimensional integration problems nonparametric estimation of the optimal importance sampling distribution known as nonparametric importance sampling is a reasonable alternative to parametric approaches in this article nonparametric variants of both the self normalized and the unnormalized importance sampling estimator are proposed and investigated a common critique of nonparametric importance sampling is the increased computational burden compared with parametric methods we solve this problem to a large degree by utilizing the linear blend frequency polygon estimator instead of a kernel estimator mean square error convergence properties are investigated leading to recommendations for the efficient application of nonparametric importance sampling particularly we show that nonparametric importance sampling as

clustered nystrom method for large scale manifold learning and dimension reduction kernel or similarity matrix plays a key role in many machine learning algorithms such as kernel methods manifold learning and dimension reduction however the cost of storing and manipulating the complete kernel matrix makes it infeasible for large problems the nystrom method is a popular sampling based low rank approximation scheme for reducing the computational burdens in handling large kernel matrices in this paper we analyze how the approximating quality of the nystrom method depends on the choice of landmark points and in particular the encoding powers of the landmark points in summarizing the data our non probabilistic error analysis justifies a clustered nystrom method that uses the k means clustering centers as landmark points our algorithm can be applied to scale up a wide variety of algorithms that depend on the eigenvalue decomposition of kernel matrix or its variant such as kernel principal component analysis laplacian eigenmap spectral clustering as well as those involving kernel matrix inverse such as least squares support vector machine and gaussian process regression extensive experiments demonstrate the competitive performance of our algorithm in both accuracy and efficiency

memory efficient kernel approximation scaling kernel machines to massive data sets is a major challenge due to storage and computation issues in handling large kernel matrices that are usually dense recently many papers have suggested tackling this problem by using a low rank approximation of the kernel matrix in this paper we first make the observation that the structure of shift invariant kernels changes from low rank to block diagonal without any low rank structure when varying the scale parameter based on this observation we propose a new kernel approximation framework memory efficient kernel approximation meka which considers both low rank and clustering structure of the kernel matrix we show that the resulting algorithm outperforms state of the art low rank kernel approximation methods in terms of speed approximation error and memory usage as an example on the covtype dataset with half a million samples meka takes around 70 seconds and uses less than 80 mb memory on a single machine to achieve 10 relative approximation error while standard nystrom approximation is about 6 times slower and uses more than 400mb memory to achieve similar approximation we also present extensive experiments on applying meka to speed up kernel ridge regression

local adaptive importance sampling for multivariate densities with strong nonlinear relationships abstract we consider adaptive importance sampling techniques that use kernel density estimates at each iteration as importance sampling functions these can provide more nearly constant importance weights and more precise estimates of quantities of interest than the sampling importance resampling algorithm when the initial importance sampling function is diffuse relative to the target we propose a new method that adapts to the varying local structure of the target when the target has unusual structure such as strong nonlinear relationships between variables this method provides estimates with smaller mean squared error than alternative methods

integration of multimodal functions by monte carlo importance sampling abstract numerical integration of a multimodal integrand f is approached by monte carlo integration via importance sampling a mixture of multivariate t density functions is suggested as an importance function g for its easy random variate generation thick tails and high flexibility the number of components in the mixture is determined by the number of modes of f and the mixing weights and location and scale parameters of the component distributions are determined by numerical minimization of a monte carlo estimate of the squared variation coefficient of the weight function f g stratified importance sampling and control variates are shown to be particularly effective variance reduction techniques in this case the algorithm is applied to a 10 dimensional example and shown to yield significant improvement over usual integration schemes

quantum surface of section method demonstration of semiclassical berry robnik energy level spacing distribution in a generic two dimensional hamiltonian system the recently developed quantum surface of section method is applied to a search for extremely high lying energy levels in a simple but generic hamiltonian system between integrability and chaos namely the semiseparable two dimensional oscillator using the stretch of 13445 consecutive levels with the sequential number around 1 8 107 eighteen million we have clearly demonstrated the validity of the semiclassical berry robnik level spacing distribution while at 1000 times smaller sequential quantum numbers we find the very persistent quasi universal phenomenon of power law level repulsion which is globally very well described by the phenomenological brody distribution this is the first statistically significant numerical confirmation of the berry robnik surmise in an autonomous two dimensional hamiltonian system

spsd matrix approximation vis column selection theories algorithms and extensions symmetric positive semidefinite spsd matrix approximation is an important problem with applications in kernel methods however existing spsd matrix approximation methods such as the nystrom method only have weak error bounds in this paper we conduct in depth studies of an spsd matrix approximation model and establish strong relative error bounds we call it the prototype model for it has more efficient and effective extensions and some of its extensions have high scalability though the prototype model itself is not suitable for large scale data it is still useful to study its properties on which the analysis of its extensions relies this paper offers novel theoretical analysis efficient algorithms and a highly accurate extension first we establish a lower error bound for the prototype model and improve the error bound of an existing column selection algorithm to match the lower bound in this way we obtain the first optimal column selection algorithm for the prototype model we also prove that the prototype model is exact under certain conditions second we develop a simple column selection algorithm with a provable error bound third we propose a so called spectral shifting model to make the approximation more accurate when the eigenvalues of the matrix decay slowly and the improvement is theoretically quantified the spectral shifting method can also be applied to improve other spsd matrix approximation models

improving cur matrix decomposition and the nystrom approximation via adaptive sampling the cur matrix decomposition and the nystrom approximation are two important low rank matrix approximation techniques the nystrom method approximates a symmetric positive semidefinite matrix in terms of a small number of its columns while cur approximates an arbitrary data matrix by a small number of its columns and rows thus cur decomposition can be regarded as an extension of the nystrom approximation in this paper we establish a more general error bound for the adaptive column row sampling algorithm based on which we propose more accurate cur and nystrom algorithms with expected relative error bounds the proposed cur and nystrom algorithms also have low time complexity and can avoid maintaining the whole data matrix in ram in addition we give theoretical analysis for the lower error bounds of the standard nystrom method and the ensemble nystrom method the main theoretical results established in this paper are novel and our analysis makes no special assumption on the data matrices

faster least squares approximation least squares approximation is a technique to find an approximate solution to a system of linear equations that has no exact solution in a typical setting one lets n be the number of constraints and d be the number of variables with n gg d then existing exact methods find a solution vector in o nd 2 time we present two randomized algorithms that provide accurate relative error approximations to the optimal value and the solution vector of a least squares approximation problem more rapidly than existing exact algorithms both of our algorithms preprocess the data with the randomized hadamard transform one then uniformly randomly samples constraints and solves the smaller problem on those constraints and the other performs a sparse random projection and solves the smaller problem on those projected coordinates in both cases solving the smaller problem provides relative error approximations and if n is sufficiently larger than d the approximate solution can be computed in o nd ln d time

an efficient binning scheme with application to statistical crack mechanics summary a computational scheme is developed for sampling based evaluation of a function whose inputs are statistically variable after a general abstract framework is developed it is applied to initialize and evolve the size and orientation of cracks within a finite domain such as a finite element or similar subdomain the finite element is presumed to be too large to explicitly track each of the potentially thousands or even millions of individual cracks in the domain accordingly a novel binning scheme is developed that maps the crack data to nodes on a reference grid in probability space the scheme which is clearly generalizable to applications involving arbitrary numbers of random variables is illustrated in the scope of planar deformations of a brittle material containing straight cracks assuming two random variables describe each crack the cracks are assigned uniformly random orientations and non uniformly random sizes their data are mapped to a computationally tractable number of nodes on a grid laid out in the unit square of probability space so that gauss points on the grid may be used to define an equivalent subpopulation of the cracks this significantly reduces the computational cost of evaluating ensemble effects of large evolving populations of random variables copyright 2015 john wiley sons ltd

four algorithms for the the efficient computation of truncated pivoted qr approximations to a sparse matrix in this paper we propose four algorithms to compute truncated pivoted qr approximations to a sparse matrix three are based on the gram schmidt algorithm and the other on householder triangularization all four algorithms leave the original matrix unchanged and the only additional storage requirements are arrays to contain the factorization itself thus the algorithms are particularly suited to determining low rank approximations to a sparse matrix

revisiting the nystrom method for improved large scale machine learning we reconsider randomized algorithms for the low rank approximation of symmetric positive semi definite spsd matrices such as laplacian and kernel matrices that arise in data analysis and machine learning applications our main results consist of an empirical evaluation of the performance quality and running time of sampling and projection methods on a diverse suite of spsd matrices our results highlight complementary aspects of sampling versus projection methods they characterize the effects of common data preprocessing steps on the performance of these algorithms and they point to important differences between uniform sampling and nonuniform sampling methods based on leverage scores in addition our empirical results illustrate that existing theory is so weak that it does not provide even a qualitative guide to practice thus we complement our empirical results with a suite of worst case theoretical bounds for both random sampling and random projection methods these bounds are qualitatively superior to existing bounds e g improved additive error bounds for spectral and frobenius norm error and relative error bounds for trace norm error and they point to future directions to make these algorithms useful in even larger scale machine learning applications