a preconditioner for a primal dual newton conjugate gradients method for compressed sensing problems in this paper we are concerned with the solution of compressed sensing cs problems where the signals to be recovered are sparse in coherent and redundant dictionaries we extend the primal dual newton conjugate gradient method pdncg in t f chan g h golub and p mulet siam j sci comput 20 1999 pp 1964 1977 to cs problems we provide an inexpensive and provably effective preconditioning technique for linear systems using pdncg numerical results are presented on cs problems which demonstrate the performance of pdncg with the proposed preconditioner compared to state of the art existing solvers

the limited memory conjugate gradient method in theory the successive gradients generated by the conjugate gradient method applied to a quadratic should be orthogonal however for some ill conditioned problems orthogonality is quickly lost due to rounding errors and convergence is much slower than expected a limited memory version of the nonlinear conjugate gradient method is developed the memory is used to both detect the loss of orthogonality and to restore orthogonality an implementation of the algorithm is presented based on the cg descent nonlinear conjugate gradient method limited memory cg descent l cg descent possesses a global convergence property similar to that of the memoryless algorithm but has much better practical performance numerical comparisons to the limited memory bfgs method l bfgs are given using the cuter test problems

local search for the probabilistic traveling salesman problem correction to the 2 p opt and 1 shift algorithms abstract the probabilistic traveling salesman problem concerns the best way to visit a set of customers located in some metric space where each customer requires a visit only with some known probability a solution to this problem is an a priori tour which visits all customers and the objective is to minimize the expected length of the a priori tour over all customer subsets assuming that customers in any given subset must be visited in the same order as they appear in the a priori tour this problem belongs to the class of stochastic vehicle routing problems a class which has received increasing attention in recent years and which is of major importance in real world applications several heuristics have been proposed and tested for the probabilistic traveling salesman problem many of which are a straightforward adaptation of heuristics for the classical traveling salesman problem in particular two local search algorithms 2 p opt and 1 shift were introduced by bertsimas in a previous report we have shown that the expressions for the cost evaluation of 2 p opt and 1 shift moves as proposed by bertsimas are not correct in this paper we derive the correct versions of these expressions and we show that the local search algorithms based on these expressions perform significantly better than those exploiting the incorrect expressions

construction of a continuous stopping boundary from an alpha spending function lan and demets 1983 biometrika 70 659 663 proposed a flexible method for monitoring accumulating data that does not require the number and times of analyses to be specified in advance yet maintains an overall type i error alpha their method amounts to discretizing a preselected continuous boundary by clumping the density of the boundary crossing time at discrete analysis times and calculating the resultant discrete time boundary values in this framework the cumulative distribution function of the continuous time stopping rule is used as an alpha spending function a key assumption that underlies this method is that future analysis times are not chosen on the basis of the current value of the statistic however clinical trials may be monitored more frequently when they are close to crossing the boundary in this situation the corresponding continuous time boundary should be used here we demonstrate how to construct a continuous stopping boundary from an alpha spending function this capability is useful in the design of clinical trials we use the beta blocker heart attack trial bhat and aids clinical trials group protocol 021 for illustration

a globally convergent version of the polak ribiere conjugate gradient method in this paper we propose a new line search algorithm that ensures global convergence of the polak ribiere conjugate gradient method for the unconstrained minimization of nonconvex differentiable functions in particular we show that with this line search every limit point produced by the polak ribiere iteration is a stationary point of the objective function moreover we define adaptive rules for the choice of the parameters in a way that the first stationary point along a search direction can be eventually accepted when the algorithm is converging to a minimum point with positive definite hessian matrix under strong convexity assumptions the known global convergence results can be reobtained as a special case from a computational point of view we may expect that an algorithm incorporating the step size acceptance rules proposed here will retain the same good features of the polak ribiere method while avoiding pathological situations

further results on the probabilistic traveling salesman problem abstract in 1985 jaillet introduced the probabilistic traveling salesman problem ptsp a variant of the classical tsp in which only a subset of the nodes may be present in any given instance of the problem the goal is to find an a priori tour of minimal expected length with the strategy of visiting the present nodes in a particular instance in the same order as they appear in the a priori tour in this paper we reexamine the ptsp using a variety of theoretical and computational approaches we sharpen the best known bounds for the ptsp derive several asymptotic relations and compare from various veiwpoints the ptsp with the re optimization strategy i e finding an optimal tour in every problem instance when a euclidean metric is used and the nodes are uniformly distributed in the unit square a heuristic for the ptsp is shown to be very close to the re optimization strategy we examine some ptsp heuristics with provable worst case performance and address the question of finding constant guarantee heuristics implementations of various heuristics some based on sorting and some on local optimality permit us to discuss the qualitative and quantitative properties of computational problems with up to 5000 nodes

robust approximate inverse preconditioning for the conjugate gradient method we present a variant of the ainv factorized sparse approximate inverse algorithm which is applicable to any symmetric positive definite matrix the new preconditioner is breakdown free and when used in conjunction with the conjugate gradient method results in a reliable solver for highly ill conditioned linear systems we also investigate an alternative approach to a stable approximate inverse algorithm based on the idea of diagonally compensated reduction of matrix entries the results of numerical tests on challenging linear systems arising from finite element modeling of elasticity and diffusion problems are presented

a new conjugate gradient method with guaranteed descent and an efficient line search a new nonlinear conjugate gradient method and an associated implementation based on an inexact line search are proposed and analyzed with exact line search our method reduces to a nonlinear version of the hestenes stiefel conjugate gradient scheme for any inexact line search our scheme satisfies the descent condition bf g k sf t bf d k le frac 7 8 bf g k 2 moreover a global convergence result is established when the line search fulfills the wolfe conditions a new line search scheme is developed that is efficient and highly accurate efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer high accuracy is achieved by using a convergence criterion which we call the approximate wolfe conditions obtained by replacing the sufficient decrease criterion in the wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion numerical comparisons are given with both l bfgs and conjugate gradient methods using the unconstrained optimization problems in the cute library

a conjugate directions approach to improve the limited memory bfgs method abstract simple modifications of the limited memory bfgs method l bfgs for large scale unconstrained optimization are considered which consist in corrections derived from the idea of conjugate directions of the used difference vectors utilizing information from the preceding iteration for quadratic objective functions the improvement of convergence is the best one in some sense and all stored difference vectors are conjugate for unit stepsizes global convergence of the algorithm is established for convex sufficiently smooth functions numerical experiments indicate that the new method often improves the l bfgs method significantly

on the order of convergence of preconditioned nonlinear conjugate gradient methods an analysis is given of preconditioned nonlinear conjugate gradient methods in which the preconditioning matrix is the exact hessian matrix at each iteration or a nearby matrix it is shown that the order of convergence of certain preconditioned methods is less than that of newton s method when exact line searches are used and an example is given

global convergence properties of conjugate gradient methods for optimization this paper explores the convergence of nonlinear conjugate gradient methods without restarts and with practical line searches the analysis covers two classes of methods that are globally convergent on smooth nonconvex functions some properties of the fletcher reeves method play an important role in the first family whereas the second family shares an important property with the polak ribiere method numerical experiments are presented

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

qn like variable storage conjugate gradients both conjugate gradient and quasi newton methods are quite successful at minimizing smooth nonlinear functions of several variables and each has its advantages in particular conjugate gradient methods require much less storage to implement than a quasi newton code and therefore find application when storage limitations occur they are however slower so there have recently been attempts to combine cg and qn algorithms so as to obtain an algorithm with good convergence properties and low storage requirements one such method is the code conmin due to shanno and phua it has proven quite successful but it has one limitation it has no middle ground in that it either operates as a quasi newton code using o n 2 storage locations or as a conjugate gradient code using 7n locations but it cannot take advantage of the not unusual situation where more than 7n locations are available but a quasi newton code requires an excessive amount of storage

a priori optimization consider a complete graph g v e in which each node is present with probability p we are interested in solving combinatorial optimization problems on subsets of nodes which are present with a certain probability we introduce the idea of a priori optimization as a strategy competitive to the strategy of reoptimization under which the combinatorial optimization problem is solved optimally for every instance we consider four problems the traveling salesman problem tsp the minimum spanning tree vehicle routing and traveling salesman facility location we discuss the applicability of a priori optimization strategies in several areas and show that if the nodes are randomly distributed in the plane the a priori and reoptimization strategies are very close in terms of performance we characterize the complexity of a priori optimization and address the question of approximating the optimal a priori solutions with polynomial time heuristics with provable worst case guarantees finally we use the tsp as an example to find practical solutions based on ideas of local optimality

a matrix free preconditioner for sparse symmetric positive definite systems and least squares problems we analyze and discuss matrix free and limited memory preconditioners for sparse symmetric positive definite systems and normal equations of large and sparse least squares problems the preconditioners are based on a partial cholesky factorization and can be coupled with a deflation strategy the construction of the preconditioners requires only matrix vector products is breakdown free and does not need to form the matrix the memory requirements of the preconditioners are defined in advance and they do not depend on the number of nonzero entries in the matrix when eigenvalue deflation is used the preconditioners turn out to be suitable for solving sequences of slowly changing systems or linear systems with different right hand sides numerical results are provided including the case of sequences arising in nonnegative linear least squares problems solved by interior point methods

adaptive sample size and importance sampling in estimation based local search for the probabilistic traveling salesman problem the probabilistic traveling salesman problem is a paradigmatic example of a stochastic combinatorial optimization problem for this problem recently an estimation based local search algorithm using delta evaluation has been proposed in this paper we adopt two well known variance reduction procedures in the estimation based local search algorithm the first is an adaptive sampling procedure that selects the appropriate size of the sample to be used in monte carlo evaluation the second is a procedure that adopts importance sampling to reduce the variance involved in the cost estimation we investigate several possible strategies for applying these procedures to the given problem and we identify the most effective one experimental results show that a particular heuristic customization of the two procedures increases significantly the effectiveness of the estimation based local search

extension of the 2 p opt and 1 shift algorithms to the heterogeneous probabilistic traveling salesman problem abstract the probabilistic traveling salesman problem is a well known problem that is quite challenging to solve it involves finding the tour with the lowest expected cost for customers that will require a visit with a given probability there are several proposed algorithms for the homogeneous version of the problem where all customers have identical probability of being realized from the literature the most successful approaches involve local search procedures with the most famous being the 2 p opt and 1 shift procedures proposed by bertsimas d j bertsimas l howell further results on the probabilistic traveling salesman problem european journal of operational research 65 1 1993 68 95 recently however evidence has emerged that indicates the equations offered for these procedures are not correct and even when corrected the translation to the heterogeneous version of the problem is not simple in this paper we extend the analysis and correction to the heterogeneous case we derive new expressions for computing the cost of 2 p opt and 1 shift local search moves and we show that the neighborhood of a solution may be explored in o n2 time the same as for the homogeneous case instead of o n3 as first reported in the literature

different initial solution generators in genetic algorithms for solving the probabilistic traveling salesman problem the probabilistic traveling salesman problem ptsp is a topic of theoretical and practical importance in the study of stochastic network problems it provides researchers with a modeling framework for exploring the stochastic effects in routing problems this paper proposed three initial solution generators nn1 nn2 ran under a genetic algorithm ga framework for solving the ptsp a set of numerical experiments based on heterogeneous and homogeneous ptsp instances were conducted to test the effectiveness and efficiency of the proposed algorithms the results from the heterogeneous ptsp show that the average e t values obtained by the three generators under a ga framework are similar to those obtained by the previous best but with an average computation time saving of 50 2 as for the homogeneous ptsp instances nn1 is a relatively better generator among the three examined while ran consistently performs worse than the other two generators in terms of average e t values additionally as compared to previously reported studies no one single algorithm consistently outperformed the others across all homogeneous ptsp instances in terms of the best e t values the fact that no one initial solution generator consistently performs best in terms of the e t value obtained across all instances in heterogeneous cases and that the performance of each examined algorithm is dependent on the number of nodes n and probability p for homogeneous cases suggest the possibility of context dependent phenomenon finally to obtain valid results researchers are advised to include at least a certain amount of test instances with the same combination of n and p when conducting ptsp experiments

diversified local search strategy under scatter search framework for the probabilistic traveling salesman problem this paper focuses on introducing a concept of diversified local search strategy under the scatter search framework for the probabilistic traveling salesman problem ptsp different combinations of three commonly used local search methods in the ptsp i e 1 shift 2 opt and 3 opt were used to investigate its effects a set of numerical experiments were conducted to test the validity of the proposed strategy based on randomly generated test instances the numerical results and the permutation test showed that the diversified local search strategy especially by combining 1 shift and 2 opt algorithms can most effectively solve the homogeneous and heterogeneous ptsp in most of the tested instances in comparison with the single local search strategy under scatter search framework

preconditioned nonlinear conjugate gradient methods based on a modified secant equation this paper includes a twofold result for the nonlinear conjugate gradient ncg method in large scale unconstrained optimization first we consider a theoretical analysis where preconditioning is embedded in a strong convergence framework of an ncg method from the literature mild conditions to be satisfied by the preconditioners are defined in order to preserve ncg convergence as a second task we also detail the use of novel matrixfree preconditioners for ncg our proposals are based on quasinewton updates and either satisfy the secant equation or a secantlike condition at some of the previous iterates we show that in some sense the preconditioners we propose also approximate the inverse of the hessian matrix in particular the structures of our preconditioners depend on lowrank updates used along with different choices of specific parameters the lowrank updates are obtained as byproduct of ncg iterations the results of an extended numerical experience using large scale cutest problems is reported showing that our preconditioners can considerably improve the performance of ncg methods

a priori optimization of the probabilistic traveling salesman problem the probabilistic traveling salesman problem ptsp is defined on a graph g v e where v is the vertex set and e is the edge set each vertex vi has a probability pi of being present with each edge vi vj is associated a distance or cost cij in a first stage an a priori hamiltonian tour on g is designed the list of present vertices is then revealed in a second stage the a priori tour is followed by skipping the absent vertices the ptsp consists of determining a first stage solution that minimizes the expected cost of the second stage tour the problem is formulated as an integer linear stochastic program and solved by means of a branch and cut approach which relaxes some of the constraints and uses lower bounding functionals on the objective function problems involving up to 50 vertices are solved to optimality

maximally selected rank statistics a common statistical problem is the assessment of the predictive power of a quantitative variable for some dependent variable a maximally selected rank statistic regarding the quantitative variable provides a test and implicitly an estimate of a cutpoint as a simple classification rule restricting the selection to an arbitrary given inner part of the support of the quantitative variable we show that the asymptotic null distribution of the maximally selected rank statistic is the distribution of the supremum of the absolute value of a standardized gaussian process on an interval the asymptotic argument holds also in the case of tied or censored observations we compare monte carlo results with an approximation of the asymptotic distribution under the null hypothesis in addition we investigate the behaviour of the test procedure and of the familiar spearman rank test for independence under some alternatives moreover we discuss some aspects of the problem of estimating an underlying cutpoint

on a class of limited memory preconditioners for large scale linear systems with multiple right hand sides this work studies a class of limited memory preconditioners lmps for solving linear positive definite systems of equations with multiple right hand sides we propose a class of lmps whose construction requires a small number of linearly independent vectors after exploring the theoretical properties of the preconditioners we focus on three particular members spectral lmp quasi newton lmp and ritz lmp we show that the first two are well known while the third is new numerical tests indicate that the ritz lmp is efficient on a real life nonlinear optimization problem arising in a data assimilation system for oceanography

a preconditioning framework for sequences of diagonally modified linear systems arising in optimization we propose a framework for building preconditioners for sequences of linear systems of the form a delta k x k b k where a is symmetric positive semidefinite and delta k is diagonal positive semidefinite such sequences arise in several optimization methods e g in affine scaling methods for bound constrained convex quadratic programming and bound constrained linear least squares as well as in trust region and overestimation methods for convex unconstrained optimization problems and nonlinear least squares for all the matrices of a sequence the preconditioners are obtained by updating any preconditioner for a available in the ldl t form the preconditioners in the framework satisfy the natural requirement of being effective on slowly varying sequences furthermore under an additional property they are also able to cluster eigenvalues of the preconditioned matrix when some entries of delta k are sufficiently large we present two low cost preconditioners sharing the above mentioned p

automatic preconditioning by limited memory quasi newton updating this paper proposes a preconditioner for the conjugate gradient method cg that is designed for solving systems of equations ax bi with different right hand side vectors or for solving a sequence of slowly varying systems ak x bk the preconditioner has the form of a limited memory quasi newton matrix and is generated using information from the cg iteration the automatic preconditioner does not require explicit knowledge of the coefficient matrix a and is therefore suitable for problems where only products of a times a vector can be computed numerical experiments indicate that the preconditioner has most to offer when these matrix vector products are expensive to compute and when low accuracy in the solution is required the effectiveness of the preconditioner is tested within a hessian free newton method for optimization and by solving certain linear systems arising in finite element models

generalized maximally selected statistics maximally selected statistics for the estimation of simple cutpoint models are embedded into a generalized conceptual framework based on conditional inference procedures this powerful framework contains most of the published procedures in this area as special cases such as maximally selected chi squared and rank statistics but also allows for direct construction of new test procedures for less standard test problems as an application a novel maximally selected rank statistic is derived from this framework for a censored response partitioned with respect to two ordered categorical covariates and potential interactions this new test is employed to search for a high risk group of rectal cancer patients treated with a neo adjuvant chemoradiotherapy moreover a new efficient algorithm for the evaluation of the asymptotic distribution for a large class of maximally selected statistics is given enabling the fast evaluation of a large number of cutpoints

characterization of the probabilistic traveling salesman problem we show that stochastic annealing can be successfully applied to gain new results on the probabilistic traveling salesman problem the probabilistic traveling salesman must decide on an a priori order in which to visit n cities randomly distributed over a unit square before learning that some cities can be omitted we find the optimized average length of the pruned tour follows e l over bar pruned rootnp 0 872 0 105p f np where p is the probability of a city needing to be visited and f np 1 as np infinity the average length of the a priori tour before omitting any cities is found to follow e l a priori rootn pbeta p where beta p 1 1 25 0 82 ln p is measured for 0 05less than or equal topless than or equal to0 6 scaling arguments and indirect measurements suggest that beta p tends towards a constant for p 0 03 our stochastic annealing algorithm is based on limited sampling of the pruned tour lengths exploiting the sampling error to provide the analog of thermal fluctuations in simulated thermal annealing the method has general application to the optimization of functions whose cost to evaluate rises with the precision required