using sampling and simplex derivatives in pattern search methods in this paper we introduce ways of making a pattern search more efficient by reusing previous evaluations of the objective function based on the computation of simplex derivatives e g simplex gradients at each iteration one can attempt to compute an accurate simplex gradient by identifying a sampling set of previously evaluated points with good geometrical properties this can be done using only past successful iterates or by considering all past function evaluations the simplex gradient can then be used to reorder the evaluations of the objective function associated with the directions used in the poll step or to update the mesh size parameter according to a sufficient decrease criterion neither of which requires new function evaluations we present these procedures in detail and apply them to a set of problems from the cuter collection numerical results show that these procedures can enhance significantly the practical performance of pattern search methods

convergence of the restricted nelder mead algorithm in two dimensions the nelder mead algorithm a longstanding direct search method for unconstrained optimization published in 1965 is designed to minimize a scalar valued function f of n real variables using only function values without any derivative information each nelder mead iteration is asso ciated with a nondegenerate simplex defined by n 1 vertices and their function values a typical iteration produces a new simplex by replacing the worst vertex by a new point despite the method s widespread use theoretical results have been limited for strictly convex objective functions of one variable with bounded level sets the algorithm always converges to the minimizer for such func tions of two variables the diameter of the simplex converges to zero but examples constructed by mckinnon show that the algorithm may converge to a nonminimizing point this paper considers the restricted nelder mead algorithm a variant that does not allow expansion steps in two dimen sions we show that for any nondegenerate starting simplex and any twice continuously differentiable function with positive definite hessian and bounded level sets the algorithm always converges to the minimizer the proof is based on treating the method as a discrete dynamical system and relies on several techniques that are nonstandard in convergence proofs for unconstrained optimization

kriging sparse polynomial dimensional decomposition surrogate model with adaptive refinement uncertainty quantification and global sensitivity analysis problems are made more difficult in the case of applications which involve expensive computer simulations this because a limited amount of simulations is available to build a sufficiently accurate metamodel of the quantities of interest in this work a numerical technique for the construction of a low cost and accurate metamodel is proposed having in mind applications with expensive computer codes two main points are intro duced firstly a technique which couples universal kriging with sparse polynomial dimensional decomposition pdd to build a metamodel with improved accuracy the polynomials selected by the adaptive pdd representation are used as a sparse basis to build an universal kriging surrogate model the second is a strategy derived from anisotropic mesh adaptation to adaptively add a fixed number of new training points to an existing design of experiments the convergence of the proposed algorithm is analyzed and assessed on different test functions with an increasing size of the input space finally the algorithm is used to propagate uncertainties in two high dimensional real problems related to the atmospheric reentry

automatic color constancy algorithm selection and combination in this work we investigate how illuminant estimation techniques can be improved taking into account intrinsic low level properties of the images we show how these properties can be used to drive given a set of illuminant estimation algorithms the selection of the best algorithm for a given image the algorithm selection is made by a decision forest composed of several trees on the basis of the values of a set of heterogeneous features the features represent the image content in terms of low level visual properties the trees are trained to select the algorithm that minimizes the expected error in illuminant estimation we also designed a combination strategy that estimates the illuminant as a weighted sum of the different algorithms estimations experimental results on the widely used ciurea and funt dataset demonstrate the effectiveness of our approach

reducing the number of function evaluations in mesh adaptive direct search algorithms the mesh adaptive direct search mads class of algorithms is designed for nonsmooth optimization where the objective function and constraints are typically computed by launching a time consuming computer simulation each iteration of a mads algorithm attempts to improve the current best known solution by launching the simulation at a finite number of trial points common implementations of mads generate 2n trial points at each iteration where n is the number of variables in the optimization problem the objective of the present work is to dynamically reduce that number we present an algorithmic framework that reduces the number of simulations to exactly n 1 without impacting the theoretical guarantees from the convergence analysis numerical experiments are conducted for several different contexts the results suggest that these strategies allow the new algorithms to reach a better solution with fewer function evaluations

mesh adaptive direct search algorithms for constrained optimization this paper addresses the problem of minimization of a nonsmooth function under general nonsmooth constraints when no derivatives of the objective or constraint functions are available we introduce the mesh adaptive direct search mads class of algorithms which extends the generalized pattern search gps class by allowing local exploration called polling in an asymptotically dense set of directions in the space of optimization variables this means that under certain hypotheses including a weak constraint qualification due to rockafellar mads can treat constraints by the extreme barrier approach of setting the objective to infinity for infeasible points and treating the problem as unconstrained the main gps convergence result is to identify limit points hat x where the clarke generalized derivatives are nonnegative in a finite set of directions called refining directions although in the unconstrained case nonnegative combinations of these directions span the whole space the fact that there can only be finitely many gps refining directions limits rigorous justification of the barrier approach to finitely many linear constraints for gps the main result of this paper is that the general mads framework is flexible enough to allow the generation of an asymptotically dense set of refining directions along which the clarke derivatives are nonnegative we propose an instance of mads for which the refining directions are dense in the hypertangent cone at hat x with probability 1 whenever the iterates associated with the refining directions converge to a single hat x the instance of mads is compared to versions of gps on some test problems we also illustrate the limitation of our results with examples

orthomads a deterministic mads instance with orthogonal directions the purpose of this paper is to introduce a new way of choosing directions for the mesh adaptive direct search mads class of algorithms the advantages of this new orthomads instantiation of mads are that the polling directions are chosen deterministically ensuring that the results of a given run are repeatable and that they are orthogonal to each other which yields convex cones of missed directions at each iteration that are minimal in a reasonable measure convergence results for orthomads follow directly from those already published for mads and they hold deterministically rather than with probability one as is the case for ltmads the first mads instance the initial numerical results are quite good for both smooth and nonsmooth and constrained and unconstrained problems considered here

on the convergence of the multidirectional search algorithm this paper presents the convergence analysis for the multidirectional search algorithm a direct search method for unconstrained minimization the analysis follows the classic lines of proofs of convergence for gradient related methods the novelty of the argument lies in the fact that explicit calculation of the gradient is unnecessary although it is assumed that the function is continuously differentiable over some subset of the domain the proof can be extended to treat most nonsmooth cases of interest the argument breaks down only at points where the derivative exists but is not continuous finally it is shown how a general convergence theory can be developed for an entire class of direct search methods which includes such methods as the factorial design algorithm and the pattern search algorithm that share a key feature of the multidirectional search algorithm

global convergence of trust region algorithms for convex constrained minimization without derivatives in this work we propose a trust region algorithm for the problem of minimizing a function within a convex closed domain we assume that the objective function is differentiable but no derivatives are available the algorithm has a very simple structure and allows a great deal of freedom in the choice of the models under reasonable assumptions for derivative free schemes we prove global convergence for the algorithm that is to say that all accumulation points of the sequence generated by the algorithm are stationary

shape optimization for tumor location in non invasive thermal diagnostics accurate correlations between the thermal image at skin surface and interior human physiology are desired in this work an estimation methodology to determine unknown geometrical parameters of an embedded tumor is proposed we define a functional that represents the mismatch between a measured experimental temperature profile which may be obtained by infrared thermography on the skin surface and the solution of an appropriate boundary problem this functional is related to the geometrical parameters through the solution of the boundary problem in such a way that finding the minimum of this functional form also means finding the unknown geometrical parameters of the embedded tumor sensitivity analysis techniques coupled with the adjoint method were considered to compute the shape derivative of the functional then a nonmonotone spectral projected gradient method was implemented to solve the optimization problem of finding the optimal geometric parameters

superlinear convergence and implicit filtering in this paper we show how the implicit filtering algorithm can be coupled with the bfgs quasi newton update to obtain a superlinearly convergent iteration if the noise in the objective function decays sufficiently rapidly as the optimal point is approached in this way we give insight into the observations of good performance in practice of quasi newton methods when they are coupled with implicit filtering we also report on numerical experiments that show how an implementation of implicit filtering that exploits these new results can improve the performance of the algorithm

frames and grids in unconstrained and linearly constrained optimization a nonsmooth approach this paper describes a class of frame based direct search methods for unconstrained and linearly constrained optimization a template is described and analyzed using clarke s break nonsmooth calculus this provides a unified and simple approach to earlier results for grid and frame based methods and also provides partial convergence results when the objective function is not smooth undefined in some places or both the template also covers many new methods which combine elements of previous ideas using frames and grids these new methods include grid based simple descent algorithms which allow moving to points off the grid at every iteration and can automatically control the grid size provided function values are available the concept of a grid is also generalized to that of an admissible set which allows sets for example with circular symmetries the method is applied to linearly constrained problems using a simple barrier approach

bayesian inference and life testing plan for the weibull distribution in presence of progressive censoring this article deals with the bayesian inference of unknown parameters of the progressively censored weibull distribution it is well known that for a weibull distribution while computing the bayes estimates the continuous conjugate joint prior distribution of the shape and scale parameters does not exist in this article it is assumed that the shape parameter has a log concave prior density function and for the given shape parameter the scale parameter has a conjugate prior distribution as expected when the shape parameter is unknown the closed form expressions of the bayes estimators cannot be obtained we use lindley s approximation to compute the bayes estimates and the gibbs sampling procedure to calculate the credible intervals for given priors we also provide a methodology to compare two different censoring schemes and thus find the optimal bayesian censoring scheme monte carlo simulations are performed to observe the behavior of the proposed methods and a data analysis is onducted for ill

pattern search algorithms for bound constrained minimization we present a convergence theory for pattern search methods for solving bound constrained nonlinear programs the analysis relies on the abstract structure of pattern search methods and an understanding of how the pattern interacts with the bound constraints this analysis makes it possible to develop pattern search methods for bound constrained problems while only slightly restricting the flexibility present in pattern search methods for unconstrained problems we prove global convergence despite the fact that pattern search methods do not have explicit information concerning the gradient and its projection onto the feasible region and consequently are unable to enforce explicitly a notion of sufficient feasible decrease

on trust region methods for unconstrained minimization without derivatives we consider some algorithms for unconstrained minimization without derivatives that form linear or quadratic models by interpolation to values of the objective function then a new vector of variables is calculated by minimizing the current model within a trust region techniques are described for adjusting the trust region radius and for choosing positions of the interpolation points that maintain not only nonsingularity of the interpolation equations but also the adequacy of the model particular attention is given to quadratic models with diagonal second derivative matrices because numerical experiments show that they are often more efficient than full quadratic models for general objective functions finally some recent research on the updating of full quadratic models is described briefly using fewer interpolation equations than before the resultant freedom is taken up by minimizing the frobenius norm of the change to the second derivative matrix of the model a preliminary version of this method provides some very promising numerical results

modeling an augmented lagrangian for blackbox constrained optimization constrained blackbox optimization is a difficult problem with most approaches coming from the mathematical programming literature the statistical literature is sparse especially in addressing problems with nontrivial constraints this situation is unfortunate because statistical methods have many attractive properties global scope handling noisy objectives sensitivity analysis and so forth to narrow that gap we propose a combination of response surface modeling expected improvement and the augmented lagrangian numerical optimization framework this hybrid approach allows the statistical model to think globally and the augmented lagrangian to act locally we focus on problems where the constraints are the primary bottleneck requiring expensive simulation to evaluate and substantial modeling effort to map out in that context our hybridization presents a simple yet effective solution that allows existing objective oriented statistical approaches like those based on gaussian process surrogates

on the convergence of pattern search algorithms we introduce an abstract definition of pattern search methods for solving nonlinear unconstrained optimization problems our definition unifies an important collection of optimization methods that neither compute nor explicitly approximate derivatives we exploit our characterization of pattern search methods to establish a global convergence theory that does not enforce a notion of sufficient decrease our analysis is possible because the iterates of a pattern search method lie on a scaled translated integer lattice this allows us to relax the classical requirements on the acceptance of the step at the expense of stronger conditions on the form of the step and still guarantee global convergence

parallel space decomposition of the mesh adaptive direct search algorithm this paper describes a parallel space decomposition psd technique for the mesh adaptive direct search mads algorithm mads extends a generalized pattern search for constrained nonsmooth optimization problems the objective of the present work is to obtain good solutions to larger problems than the ones typically solved by mads the new method psd mads is an asynchronous parallel algorithm in which the processes solve problems over subsets of variables the convergence analysis based on the clarke calculus is essentially the same as for the mads algorithm a practical implementation is described and some numerical results on problems with up to 500 variables illustrate the advantages and limitations of psd mads

on the convergence of grid based methods for unconstrained optimization the convergence of direct search methods for unconstrained minimization is examined in the case where the underlying method can be interpreted as a grid or pattern search over successively refined meshes an important aspect of the main convergence result is that translation rotation scaling and shearing of the successive grids are allowed

a method for stochastic constrained optimization using derivative free surrogate pattern search and collocation recent advances in coupling novel optimization methods to large scale computing problems have opened the door to tackling a diverse set of physically realistic engineering design problems a large computational overhead is associated with computing the cost function for most practical problems involving complex physical phenomena such problems are also plagued with uncertainties in a diverse set of parameters we present a novel stochastic derivative free optimization approach for tackling such problems our method extends the previously developed surrogate management framework smf to allow for uncertainties in both simulation parameters and design variables the stochastic collocation scheme is employed for stochastic variables whereas kriging based surrogate functions are employed for the cost function this approach is tested on four numerical optimization problems and is shown to have significant improvement in efficiency over traditional monte carlo schemes problems with multiple probabilistic constraints are also discussed

pattern search methods for linearly constrained minimization we extend pattern search methods to linearly constrained minimization we develop a general class of feasible point pattern search algorithms and prove global convergence to a karush kuhn tucker point as in the case of unconstrained minimization pattern search methods for linearly constrained problems accomplish this without explicit recourse to the gradient or the directional derivative of the objective key to the analysis of the algorithms is the way in which the local search patterns conform to the geometry of the boundary of the feasible region

precision control for generalized pattern search algorithms with adaptive precision function evaluations in the literature on generalized pattern search algorithms convergence to a stationary point of a once continuously differentiable cost function is established under the assumption that the cost function can be evaluated exactly however there is a large class of engineering problems where the numerical evaluation of the cost function involves the solution of systems of differential algebraic equations since the termination criteria of the numerical solvers often depend on the design parameters computer code for solving these systems usually defines a numerical approximation to the cost function that is discontinuous with respect to the design parameters standard generalized pattern search algorithms have been applied heuristically to such problems but no convergence properties have been stated in this paper we extend a class of generalized pattern search algorithms to include a subprocedure that adaptively controls the precision of the approximating cost functions the numerical approximations to the cost function need not define a continuous function our algorithms can be used for solving linearly constrained problems with cost functions that are at least locally lipschitz continuous assuming that the cost function is smooth we prove that our algorithms converge to a stationary point under the weaker assumption that the cost function is only locally lipschitz continuous we show that our algorithms converge to points at which the clarke generalized directional derivatives are nonnegative in predefined directions an important feature of our adaptive precision scheme is the use of coarse approximations in the early iterations with the approximation precision controlled by a test we show by numerical experiments that such an approach leads to substantial time savings in minimizing computationally expensive functions

erratum mesh adaptive direct search algorithms for constrained optimization in siam j optim 17 2006 pp 188 217 audet and dennis proposed the class of mesh adaptive direct search mads algorithms for minimization of a nonsmooth function under general nonsmooth constraints the notation used in the paper evolved since the preliminary versions and unfortunately even though the statement of proposition 4 2 is correct it is not compatible with the final notation the purpose of this note is to show that the proposition is valid

pattern search ranking and selection algorithms for mixed variable simulation based optimization the class of generalized pattern search gps algorithms for mixed variable optimization is extended to problems with stochastic objective functions because random noise in the objective function makes it more difficult to compare trial points and ascertain which points are truly better than others replications are needed to generate sufficient statistical power to draw conclusions rather than comparing pairs of points the approach taken here augments pattern search with a ranking and selection r s procedure which allows for comparing many function values simultaneously asymptotic convergence for the algorithm is established numerical issues are discussed and performance of the algorithm is studied on a set of test problems

hybrid pattern search and simulated annealing for fuzzy production planning problems in this paper the hybridization of ps pattern search method and sa simulated annealing are incorporated in the optimization process this is in order to look for the global optimal solution for the fitness function and decision variables as well as minimum computational cpu time the real strength of sa approach has been tested in a case study problem of industrial production planning this is due to the great advantage of sa for being easily escaping from trapped in local minima by accepting up hill move through a probabilistic procedure in the final stages of optimization process in the ph d thesis by vasant 2008 4 16 different techniques were provided of heuristic and meta heuristic approaches in solving industrial production problems with nonlinear cubic objective functions eight decision variables and 29 constraints in this paper fuzzy technological problems have been solved using hybrid techniques of pattern search and simulated annealing hsaps the simulated and computational results are compared to other various evolutionary techniques

optimal progressive censoring plans for the weibull distribution in this article we compute the expected fisher information and the asymptotic variance covariance matrix of the maximum likelihood estimates based on a progressively type ii censored sample from a weibull distribution by direct calculation as well as the missing information principle we then use these values to determine the optimal progressive censoring plans three optimality criteria are considered and some selected optimal progressive censoring plans are presented according to these optimality criteria we also discuss the construction of progressively censored reliability sampling plans for the weibull distribution three illustrative examples are provided with discussion

stationarity results for generating set search for linearly constrained optimization we present a new generating set search gss approach for minimizing functions subject to linear constraints gss is a class of direct search optimization methods that includes generalized pattern search one of our main contributions in this paper is a new condition to define the set of conforming search directions that admits several computational advantages for continuously differentiable functions we also derive a bound relating a measure of stationarity which is equivalent to the norm of the gradient of the objective in the unconstrained case and a parameter used by gss algorithms to control the lengths of the steps with the additional assumption that the derivative is lipschitz we obtain a big o bound as a consequence of this relationship we obtain subsequence convergence to a kkt point even though gss algorithms lack explicit gradient information numerical results indicate that the bound provides a reasonable estimate of stationarity

accelerated random search for constrained global optimization assisted by radial basis function surrogates abstract the accelerated random search ars algorithm appel et al 2004 is a stochastic algorithm for the global optimization of black box functions subject only to bound constraints ars iteratively samples uniformly within a box centered at the current best point if the sample point is worse than the current best point then the size of the box is reduced thereby making it more likely to generate an improving solution if the current best point is not yet a local minimum otherwise if the sample point is an improvement over the current best point then the size of the box is reset to the initial value so that the box covers the entire search space ars has been shown theoretically and numerically to converge to the global minimum faster than the pure random search prs algorithm on bound constrained problems this article develops the constrained ars cars algorithm which is an extension of ars for optimization problems with black box inequality constraints under some mild conditions on the constrained optimization problem we prove the convergence to the global minimum in a probabilistic sense of a class of stochastic search algorithms called scattered uniform random search scat urs algorithms which includes cars as a special case to improve performance of cars on computationally expensive simulation based problems we incorporate radial basis function rbf surrogate models to approximate the objective and constrained functions and refer to the resulting algorithm as cars rbf numerical experiments show that cars consistently and substantially outperforms both the pure random search prs and accelerated particle swarm optimization apso algorithms yang 2010 on 31 test problems with dimensions ranging from 2 to 20 moreover cars rbf is an improvement over cars outperforms a sequential penalty derivative algorithm and competes with constrlmsrbf regis 2011 on the same test problems finally sensitivity analysis of cars rbf to the type of rbf model used shows that the cubic rbf model generally yields the best results on the test problems among six types of rbf models considered including the thin plate spline gaussian and multiquadric models

a naive multi scale search algorithm for global optimization problems this paper proposes a multi scale search algorithm for solving global optimization problems given a finite number of function evaluations we refer to this algorithm as the naive multi scale search optimization nmso nmso looks for the optimal solution by optimistically partitioning the search space over multiple scales in a hierarchical fashion based on a weak assumption about the function smoothness we present a theoretical analysis on its finite time and asymptotic convergence an empirical assessment of the algorithm has been conducted on the noiseless black box optimization benchmarking bbob testbed and compared with the state of the art optimistic as well as stochastic algorithms moreover the efficacy of nmso has been validated on the black box optimization competition within the gecco 15 conference where it has secured the third place out of twenty eight participating algorithms overall nmso is suitable for problems with limited function evaluations low dimensionality search space and objective functions that are separable or multi modal otherwise it is comparable with the top performing algorithms

a pattern search filter method for nonlinear programming without derivatives this paper formulates and analyzes a pattern search method for general constrained optimization based on filter methods for step acceptance roughly a filter method accepts a step that improves either the objective function value or the value of some function that measures the constraint violation the new algorithm does not compute or approximate any derivatives penalty constants or lagrange multipliers a key feature of the new algorithm is that it preserves the division into search and local poll steps which allows the explicit use of inexpensive surrogates or random search heuristics in the search step it is shown here that the algorithm identifies limit points at which optimality conditions depend on local smoothness of the functions and to a greater extent on the choice of a certain set of directions stronger optimality conditions are guaranteed for smoother functions and in the constrained case for a fortunate choice of the directions on which the algorithm depends these directional conditions generalize those given previously for linear constraints but they do not require a feasible starting point in the absence of general constraints the proposed algorithm and its convergence analysis generalize previous work on unconstrained bound constrained and linearly constrained generalized pattern search the algorithm is illustrated on some test examples and on an industrial wing planform engineering design application

an algorithm model for mixed variable programming in this paper we consider a particular class of nonlinear optimization problems involving both continuous and discrete variables the distinguishing feature of this class of nonlinear mixed variable optimization problems is that the structure and the number of variables of the problem depend on the values of some discrete variables in particular we define a general algorithm model for the solution of this class of problems that draws inspiration from the approach recently proposed by audet and dennis siam j optim 11 2001 pp 573 594 and is based on the strategy of combining in a suitable way a local search with respect to the continuous variables and a local search with respect to the discrete variables we prove global convergence of the algorithm model without specifying the local continuous search but only identifying some reasonable requirements moreover we define a particular derivative free algorithm for solving mixed variable programming problems where the continuous variables are linearly constrained and derivative information is not available finally we report numerical results obtained by the proposed algorithm in solving a real optimal design problem these results show the effectiveness of the approach

progressively censored reliability sampling plans for the weibull distribution this article presents progressively censored variable sampling plans for the weibull distribution approximate maximum likelihood estimators are developed for estimating the parameters of interest in the construction of sampling plans asymptotic distribution theory is used to determine the sample size and the acceptance constant sampling plans are tabulated for selected progressive censoring patterns and specifications for demonstration and comparison a monte carlo experiment conducted to investigate the accuracy of the asymptotic normal approximation has shown that the procedure is sufficiently accurate for practical purposes an example based on data reported by montanari and cacciari from progressively censored aging tests on xlpe insulated cable models is given for illustration

objective derivative free methods for constrained optimization we propose feasible descent methods for constrained minimization that do not make explicit use of the derivative of the objective function the methods iteratively sample the objective function value along a finite set of feasible search arcs and decrease the sampling stepsize if an improved objective function value is not sampled the search arcs are obtained by projecting search direction rays onto the feasible set and the search directions are chosen such that a subset approximately generates the cone of first order feasible variations at the current iterate we show that these methods have desirable convergence properties under certain regularity assumptions on the constraints in the case of linear constraints the projections are redundant and the regularity assumptions hold automatically numerical experience with the methods in the linearly constrained case is reported

generalized pattern searches with derivative information a common question asked by users of direct search algorithms is how to use derivative information at iterates where it is available this paper addresses that question with respect to generalized pattern search gps methods for unconstrained and linearly constrained optimization specifically this paper concentrates on the gps pollstep polling is done to certify the need to refine the current mesh and it requires o n function evaluations in the worst case we show that the use of derivative information significantly reduces the maximum number of function evaluations necessary for pollsteps even to a worst case of a single function evaluation with certain algorithmic choices given here furthermore we show that rather rough approximations to the gradient are sufficient to reduce the pollstep to a single function evaluation we prove that using these less expensive pollsteps does not weaken the known convergence properties of the method all of which depend only on the pollstep

generalized pattern search methods for a class of nonsmooth optimization problems with structure we propose a generalized pattern search gps method to solve a class of nonsmooth minimization problems where the set of nondifferentiability is included in the union of known hyperplanes and therefore is highly structured both unconstrained and linearly constrained problems are considered at each iteration the set of poll directions is enforced to conform to the geometry of both the nondifferentiability set and the boundary of the feasible region near the current iterate this is the key issue to guarantee the convergence of certain subsequences of iterates to points which satisfy first order optimality conditions numerical experiments on some classical problems validate the method

inexact restoration method for nonlinear optimization without derivatives a derivative free optimization method is proposed for solving a general nonlinear programming problem it is assumed that the derivatives of the objective function and the constraints are not available the new method is based on the inexact restoration scheme where each iteration is decomposed in two phases in the first one the violation of the feasibility is reduced in the second one the objective function is minimized onto a linearization of the nonlinear constraints at both phases polynomial interpolation models are used in order to approximate the objective function and the constraints at the first phase a derivative free solver for box constrained optimization can be used for the second phase we propose a new method ad hoc based on trust region strategy that uses the projection of the simplex gradient on the tangent space under suitable assumptions the algorithm is well defined and convergence results are proved a numerical implementation is described and numerical experiments are presented to validate the theoretical results

global and asymptotic convergence rate estimates for a class of projected gradient processes projected gradient processes of the goldstein levitin polyak type are considered for constrained minimization problems min omega f with omega a convex set in a hilbert space x and f x to mathbb r 1 a differentiable functional global and local convergence theorems are established for a large class of these processes including those generated with implicit step length rules proposed by bertsekas and goldstein in this analysis traditional uniform strong positivity conditions on the hessian nabla 2 f are replaced by weaker pseudoconvexity conditions and growth conditions on f when f has a unique minimizes in omega convergence rates are shown to depend on how rapidly the function gamma sigma inf r f x f xi mid x in omega x xi geqq sigma grows with increasing sigma 0 if gamma sigma geqq b sigma nu for some b 0 the processes f n in question converge to inf f like o n nu nu 2 linear

optimization by direct search new perspectives on some classical and modern methods direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis nonetheless users remained loyal to these methods most of which were easy to program some of which were reliable in the past fifteen years these methods have seen a revival due in part to the appearance of mathematical analysis as well as to interest in parallel and distributed com puting this review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results the underlying principles allow general ization to handle bound constraints and linear constraints we also discuss extensions to problems with nonlinear constraints

a comparison of three methods for selecting values of input variables in the analysis of output from a computer code two types of sampling plans are examined as alternatives to simple random sampling in monte carlo studies these plans are shown to be improvements over simple random sampling with respect to variance for a class of estimators which includes the sample mean and the empirical distribution function

spherical mesh adaptive direct search for separating quasi uncorrelated sources by range based independent component analysis it is seemingly paradoxical to the classical definition of the independent component analysis ica that in reality the true sources are often not strictly uncorrelated with this in mind this letter concerns a framework to extract quasi uncorrelated sources with finite supports by optimizing a range based contrast function under unit norm constraints to handle the inherent scaling indeterminacy of ica but without orthogonality constraints albeit the appealing contrast properties of the range based function e g the absence of mixing local optima the function is not differentiable everywhere unfortunately there is a dearth of literature on derivative free optimizers that effectively handle such a nonsmooth yet promising contrast function this is the compelling reason for the design of a nonsmooth optimization algorithm on a manifold of matrices having unit norm columns with the following objectives to ascertain convergence to a clarke stationary point of the contrast function and adhere to the necessary unit norm constraints more naturally the proposed nonsmooth optimization algorithm crucially relies on the design and analysis of an extension of the mesh adaptive direct search mads method to handle locally lipschitz objective functions defined on the sphere the applicability of the algorithm in the ica domain is demonstrated with simulations involving natural face aerial and texture images

sequential penalty derivative free methods for nonlinear constrained optimization we consider the problem of minimizing a continuously differentiable function of several variables subject to smooth nonlinear constraints we assume that the first order derivatives of the objective function and of the constraints can be neither calculated nor explicitly approximated hence every minimization procedure must use only a suitable sampling of the problem functions these problems arise in many industrial and scientific applications and this motivates the increasing interest in studying derivative free methods for their solution the aim of the paper is to extend to a derivative free context a sequential penalty approach for nonlinear programming this approach consists in solving the original problem by a sequence of approximate minimizations of a merit function where penalization of constraint violation is progressively increased in particular under some standard assumptions we introduce a general theoretical result regarding the connections between the sampling technique and the updating of the penalization which are able to guarantee convergence to stationary points of the constrained problem on the basis of the general theoretical result we propose a new method and prove its convergence to stationary points of the constrained problem the computational behavior of the method has been evaluated both on a set of test problems and on a real application the obtained results and the comparison with other well known derivative free software show the viability of the proposed sequential penalty approach

inexact restoration method for derivative free optimization with smooth constraints a new method is introduced for solving constrained optimization problems in which the derivatives of the constraints are available but the derivatives of the objective function are not the method is based on the inexact restoration framework by means of which each iteration is divided in two phases in the first phase one considers only the constraints in order to improve feasibility in the second phase one minimizes a suitable objective function subject to a linear approximation of the constraints the second phase must be solved using derivative free methods an algorithm introduced recently by kolda lewis and torczon for linearly constrained derivative free optimization is employed for this purpose under usual assumptions convergence to stationary points is proved a computer implementation is described and numerical experiments are presented

a nonderivative version of the gradient sampling algorithm for nonsmooth nonconvex optimization we give a nonderivative version of the gradient sampling algorithm of burke lewis and overton for minimizing a locally lipschitz function f on mathbb r n that is continuously differentiable on an open dense subset instead of gradients of f we use estimates of gradients of the steklov averages of f obtained by convolution with mollifiers which require f values only we show that the nonderivative version retains the convergence properties of the gradient sampling algorithm in particular with probability 1 it either drives the f values to infty or each of its cluster points is clarke stationary for f

a linesearch based derivative free approach for nonsmooth constrained optimization in this paper we propose new linesearch based methods for nonsmooth constrained optimization problems when first order information on the problem functions is not available in the first part we describe a general framework for bound constrained problems and analyze its convergence toward stationary points using the clarke jahn directional derivative in the second part we consider inequality constrained optimization problems where both objective function and constraints can possibly be nonsmooth in this case we first split the constraints into two subsets difficult general nonlinear constraints and simple bound constraints on the variables then we use an exact penalty function to tackle the difficult constraints and we prove that the original problem can be reformulated as the bound constrained minimization of the proposed exact penalty function finally we use the framework developed for the bound constrained case to solve the penalized problem moreover we prove that every accumulation poin

finding optimal algorithmic parameters using derivative free optimization the objectives of this paper are twofold we devise a general framework for identifying locally optimal algorithmic parameters algorithmic parameters are treated as decision variables in a problem for which no derivative knowledge or existence is assumed a derivative free method for optimization seeks to minimize some measure of performance of the algorithm being fine tuned this measure is treated as a black box and may be chosen by the user examples are given in the text the second objective is to illustrate this framework by specializing it to the identification of locally optimal trust region parameters in unconstrained optimization the derivative free method chosen to guide the process is the mesh adaptive direct search a generalization of pattern search methods we illustrate the flexibility of the latter and in particular make provision for surrogate objectives locally optimal parameters with respect to overall computational time on a set of test problems are identified each function call may take several hours and may not always return a predictable result a tailored surrogate function is used to guide the search towards a local solution the parameters thus identified differ from traditionally used values and allow one to solve a problem that remained otherwise unsolved in a reasonable time using traditional values

on flexible progressive censoring in life testing experiment under the setup of progressive type ii censoring we consider a new flexible scheme in which the experimenter removes r 1 r 1 r 1 units from the experiment if the first failure occurs before some predefined time t 1 and removes r 1 units when the first failure occurs after time t 1 if the second failure occurs before the time t 2 r 2 surviving items are removed at random otherwise r 2 surviving units are removed r 2 r 2 and so on finally after the mth failure all remaining items are removed under this setup of the experiment we study the distributions of failure times which are entitled as flexible progressive type ii censored fpc order statistics the simulation algorithm for generating fpc samples and an illustrative example for a special case of exponential distribution are also presented finally a monte carlo study is also conducted for the expected termination time under fcp

spinor focus wave modes new solutions of the homogeneous spinor wave equation are obtained they are similar to the focus wave mode solutions of maxwell s equations leading to a gaussian pulse energy a weighted superposition of these modes may supply finite energy pulses the particular case of bessel weight functions is discussed

revisiting asynchronous parallel pattern search for nonlinear optimization we present a new asynchronous parallel pattern search apps method which is different from that developed previously by hough kolda and torczon apps efficiently uses parallel and distributed computing platforms to solve science and engineering design optimization problems where derivatives are unavailable and cannot be approximated the original apps was designed to be fault tolerant as well as asynchronous and was based on a peer to peer design each process was in charge of a single fixed search direction our new version is based instead on a manager worker paradigm though less fault tolerant the resulting algorithm is more flexible in its use of distributed computing resources we further describe how to incorporate a zero order sufficient decrease condition and handle bound constraints convergence theory for all situations unconstrained and bound constrained as well as simple and sufficient decrease is developed we close with a discussion of how the new apps will better facilitate the future incorporation of linear and nonlinear constraints

analysis of direct searches for discontinuous functions it is known that the clarke generalized directional derivative is nonnegative along the limit directions generated by directional direct search methods at a limit point of certain subsequences of unsuccessful iterates if the function being minimized is lipschitz continuous near the limit point in this paper we generalize this result for discontinuous functions using rockafellar generalized directional derivatives upper subderivatives we show that rockafellar derivatives are also nonnegative along the limit directions of those subsequences of unsuccessful iterates when the function values converge to the function value at the limit point this result is obtained assuming that the function is directionally lipschitz with respect to the limit direction it is also possible under appropriate conditions to establish more insightful results by showing that the sequence of points generated by these methods eventually approaches the limit point along the locally best branch or step function when the number of steps is equal to two the results of this paper are presented for constrained optimization and illustrated numerically

generating set search methods for piecewise smooth problems we consider a direct search approach for solving nonsmooth minimization problems where the objective function is locally lipschitz continuous and piecewise continuously differentiable on a finite family of polyhedra a generating set search method is proposed which is named structured because the structure of the set of nondifferentiability near the current iterate is exploited to define the search directions at each iteration some numerical results are presented to validate the approach

implementing generating set search methods for linearly constrained minimization we discuss an implementation of a derivative free generating set search method for linearly constrained minimization with no assumption of nondegeneracy placed on the constraints the convergence guarantees for generating set search methods require that the set of search directions possesses certain geometrical properties that allow it to approximate the feasible region near the current iterate in the hard case the calculation of the search directions corresponds to finding the extreme rays of a cone with a degenerate vertex at the origin a difficult problem we discuss here how state of the art computational geometry methods make it tractable to solve this problem in connection with generating set search we also discuss a number of other practical issues of implementation such as the careful treatment of equality constraints and the desirability of augmenting the set of search directions beyond the theoretically minimal set we illustrate the behavior of the implementation on several problems from the cuter test suite we have found it to be successful on problems with several hundred variables and linear constraints

generalized pattern search methods for linearly equality constrained optimization problems torczon and several persons have designed pattern search methods for unconstrained optimization problem bound constrained and linearly constrained optimization problems in this paper we shall consider linear equality constrained problems as unconstrained ones using generalized pattern search algorithm except for a feasible starting point

focused waves and the scalar wave equation new solutions of the free space wave equation are studied and particular forms of these new solutions lead naturally to the localized wave solutions that have recently been reported in the literature localized waves exhibit a high degree of spatial and temporal localization choosing the characteristic variables z ct z ct the green s function is constructed for the scalar wave equation with this form for the green s function it is shown how a relatively simple line source of infinite extent aligned with the propagation axis gives rise to a propagated field which exhibits a degree of transverse focusing

active set identification for linearly constrained minimization without explicit derivatives we consider active set identification for linearly constrained optimization problems in the absence of explicit information about the derivative of the objective function we begin by presenting some general results on active set identification that are not tied to any particular algorithm these general results are sufficiently strong that given a sequence of iterates converging to a karush kuhn tucker point it is possible to identify binding constraints for which there are nonzero multipliers we then focus on generating set search methods a class of derivative free direct search methods we discuss why these general results which are posed in terms of the direction of steepest descent apply to generating set search even though these methods do not have explicit recourse to derivatives nevertheless there is a clearly identifiable subsequence of iterations at which we can reliably estimate the set of constraints that are binding at a solution we discuss how active set estimation can be used to accelerate generating set search methods and illustrate the appreciable improvement that can result using several examples from the cuter test suite we also introduce two algorithmic refinements for generating set search methods the first expands the subsequence of iterations at which we can make inferences about stationarity the second is a more flexible step acceptance criterion

analysis of generalized pattern searches this paper contains a new convergence analysis for the lewis and torczon generalized pattern search gps class of methods for unconstrained and linearly constrained optimization this analysis is motivated by a desire to understand the successful behavior of the algorithm under hypotheses that are satisfied by many practical problems specifically even if the objective function is discontinuous or extended valued the methods find a limit point with some minimizing properties simple examples show that the strength of the optimality conditions at a limit point depends not only on the algorithm but also on the directions it uses and on the smoothness of the objective at the limit point in question the contribution of this paper is to provide a simple convergence analysis that supplies detail about the relation of optimality conditions to objective smoothness properties and to the defining directions for the algorithm and it gives previous results as corollaries

pattern search algorithms for mixed variable programming many engineering optimization problems involve a special kind of discrete variable that can be represented by a number but this representation has no significance such variables arise when a decision involves some situation like a choice from an unordered list of categories this has two implications the standard approach of solving problems with continuous relaxations of discrete variables is not available and the notion of local optimality must be defined through a user specified set of neighboring points we present a class of direct search algorithms to provide limit points that satisfy some appropriate necessary conditions for local optimality for such problems we give a more expensive version of the algorithm that guarantees additional necessary optimality conditions a small example illustrates the differences between the two versions a real thermal insulation system design problem illustrates the efficacy of the user controls for this class of algorithms

a mesh adaptive direct search algorithm for multiobjective optimization abstract this work studies multiobjective optimization mop of nonsmooth functions subject to general constraints we first present definitions and optimality conditions as well as some single objective formulations of mop parameterized with respect to some reference point in the space of objective functions next we propose a new algorithm called m ulti m ads multiobjective mesh adaptive direct search for mop m ulti m ads generates an approximation of the pareto front by solving a series of single objective formulations of mop generated using the nbi natural boundary intersection framework these single objective problems are solved using the m ads mesh adaptive direct search algorithm for constrained nonsmooth optimization the pareto front approximation is shown to satisfy some first order necessary optimality conditions based on the clarke calculus m ulti m ads is then tested on problems from the literature with different pareto front landscapes and on a styrene production process simulation problem from chemical engineering

globally convergent evolution strategies in this paper we show how to modify a large class of evolution strategies es s for unconstrained optimization to rigorously achieve a form of global convergence meaning convergence to stationary points independently of the starting point the type of es under consideration recombines the parent points by means of a weighted sum around which the offspring points are computed by random generation one relevant instance of such an es is covariance matrix adaptation es cma es the modifications consist essentially of the reduction of the size of the steps whenever a sufficient decrease condition on the function values is not verified when such a condition is satisfied the step size can be reset to the step size maintained by the es s themselves as long as this latter one is sufficiently large we suggest a number of ways of imposing sufficient decrease for which global convergence holds under reasonable assumptions in particular density of certain limit directions in the unit sphere given a limited budget of function evaluations our numerical experiments have shown that the modified cma es is capable of further progress in function values moreover we have observed that such an improvement in efficiency comes without weakening significantly the performance of the underlying method in the presence of several local minimizers

parameter estimation of coupled water and energy balance models based on stationary constraints of surface states 1 we use a conditional averaging approach to estimate the parameters of a land surface water and energy balance model and then use the estimated parameters to partition net radiation into latent sensible and ground heat fluxes and precipitation into evapotranspiration and drainage plus runoff through conditional averaging of the modeled fluxes with respect to soil moisture and temperature we write an objective function that approximates the temperature and moisture dependent errors of the modeled fluxes in terms of atmospheric forcing e g precipitation and radiation surface states moisture s and temperature ts and model parameters the novelty of the approach is that the error term is estimated without comparison to measured fluxes instead it is inferred from the deviation of the conditionally averaged tendency terms expectation and expectation from zero since each of these terms equals zero in stationary systems but diverges from zero in the presence of misspecified parameters minimization of the approximated error yields parameters for model applications this strategy was previously studied for simple water balance models using soil moisture conditional averaging here we extend the idea to include energy balance fluxes and surface temperature conditioning the method is tested at two ameriflux sites vaira ranch california and kendall grassland arizona the estimated fluxes using only observed forcing and state variables are in reasonable agreement with field measurements because this method is based on conditional averages it can be applied to situations with subsampled or missing data that is continuous integration in time is not required

on the identification of active constraints nondegeneracy conditions that guarantee that the optimal active constraints are identified in a finite number of iterations are studied results of this type have only been established for a few algorithms and then under restrictive hypothesis the main result is a characterization of those algorithms that identify the optimal constraints in a finite number of iterations this result is obtained with a nondegeneracy assumption which is equivalent in the standard nonlinear programming problem to the assumption that there is a set of strictly complementary lagrange multipliers as an important consequence of the authors results the way that this characterization applies to gradient projection and sequential quadratic programming algorithms is shown

a derivative free algorithm for linearly constrained finite minimax problems in this paper we propose a new derivative free algorithm for linearly constrained finite minimax problems due to the nonsmoothness of this class of problems standard derivative free algorithms can locate only points which satisfy weak necessary optimality conditions in this work we define a new derivative free algorithm which is globally convergent toward standard stationary points of the finite minimax problem to this end we convert the original problem into a smooth one by using a smoothing technique based on the exponential penalty function of kort and bertsekas this technique depends on a smoothing parameter which controls the approximation to the finite minimax problem the proposed method is based on a sampling of the smooth function along a suitable search direction and on a particular updating rule for the smoothing parameter that depends on the sampling stepsize numerical results on a set of standard minimax test problems are reported

regularized virtual fields method for mechanical properties identification of composite materials abstract this paper presents a new material parameters identification method based on the virtual fields method vfm where a regularization term coming from a micromechanical model has been added in this regularized virtual fields method rvfm the system of equations associated to the vfm is solved within a constrained optimization framework the method was applied to virtual scanning electron microscope in situ tests aiming at determining the local properties of uni directional composites loaded in the transverse direction and for which the strain fields were obtained through digital image correlation dic the method was tested for composites with various volume fractions and properties contrasts and under different boundary conditions the potentials and limitations of the algorithm in the presence of noisy images were investigated and compared to that of the vfm and the other finite element based methods the corresponding results indicate that the proposed rvfm is more efficient than the other identification techniques in terms of both accuracy and computational time an optimum size of region of interest was also determined by taking into account dic magnification requirements and based on the accuracy of the identified parameters this kind of study can be very useful for determining a priori the level of resolution in the imaging system required for a favorable accuracy of the identified parameters

a globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds we give a pattern search method for nonlinearly constrained optimization that is an adaption of a bound constrained augmented lagrangian method first proposed by conn gould and toint siam j numer anal 28 1991 pp 545 572 in the pattern search adaptation we solve the bound constrained subproblem approximately using a pattern search method the stopping criterion proposed by conn gould and toint for the solution of the subproblem requires explicit knowledge of derivatives such information is presumed absent in pattern search methods however we show how we can replace this with a stopping criterion based on the pattern size in a way that preserves the convergence properties of the original algorithm in this way we proceed by successive inexact bound constrained minimization without knowing exactly how inexact the minimization is as far as we know this is the first provably convergent direct search method for general nonlinear programming

finding the observed information matrix when using the em algorithm a procedure is derived for extracting the observed information matrix when the em algorithm is used to find maximum likelihood estimates in incomplete data problems the technique requires computation of a complete data gradient vector or second derivative matrix but not those associated with the incomplete data likelihood in addition a method useful in speeding up the convergence of the em algorithm is developed two examples are presented

on the evanescent fields and the causality of the focus wave modes the diverging and converging field components of the source free focus wave modes are studied within the framework of both the whittaker and weyl plane wave expansions it is shown that in the weyl picture the evanescent fields associated with the diverging and converging components of the focus wave mode solution cancel each other identically the source free focus wave modes are hence composed solely of backward and forward propagating components of the whittaker type it will also be shown that no evanescent fields are associated with the causal excitation of an aperture by an initial focus wave mode field the diverging field however is composed solely of causal components that propagate away from the aperture with a specific choice of parameters the field generated by the aperture is a very good approximation to the source free solution under the same conditions the original focus wave mode solution is composed predominantly of causal forward propagating fields

pattern search methods for user provided points application to molecular geometry problems this paper deals with the application of pattern search methods to the numerical solution of a class of molecular geometry problems with important applications in molecular physics and chemistry the goal is to find a configuration of a cluster or a molecule with minimum total energy the minimization problems in this class of molecular geometry problems have no constraints and the objective function is smooth the difficulties arise from the existence of several local minima and especially from the expensive function evaluation total energy and the possible nonavailability of first order derivatives we introduce a pattern search approach that attempts to exploit the physical nature of the problem by using energy lowering geometrical transformations and to take advantage of parallelism without the use of derivatives numerical results for a particular instance of this new class of pattern search methods are presented showing the promise of our approach the new pattern search methods can be used in any other context where there is a user provided scheme to generate points leading to a potential objective function decrease