stationary fuzzy fokker planck learning for derivative free optimization stationary fuzzy fokker planck learning sffpl is a recently introduced computational method that applies fuzzy modeling to solve optimization problems this study develops a concept of applying sffpl based computations for nonlinear constrained optimization we consider the development of sffpl based optimization algorithms which do not require derivatives of the objective function and of the constraints the sequential penalty approach was used to handle the inequality constraints it was proved under some standard assumptions that the carefully designed sffpl based algorithms converge asymptotically to the stationary points the convergence proofs follow a simple mathematical approach and invoke mean value theorem the algorithms were evaluated on the test problems with the number of variables up to 50 the performance comparison of the proposed algorithms with some of the standard optimization algorithms further justifies our approach the sffpl based optimization approach due to its novelty could possibly be extended to several research directions

on the continual rubik s cube abstract let f and g be two continuous functions on the unit sphere s n 1 in r n n 3 and let their restrictions to any one dimensional great circle e coincide after some rotation e of this circle f e g e we prove that in this case f g or f g for all s n 1 this answers the question posed by richard gardner and vladimir golubyatnikov

using sampling and simplex derivatives in pattern search methods in this paper we introduce ways of making a pattern search more efficient by reusing previous evaluations of the objective function based on the computation of simplex derivatives e g simplex gradients at each iteration one can attempt to compute an accurate simplex gradient by identifying a sampling set of previously evaluated points with good geometrical properties this can be done using only past successful iterates or by considering all past function evaluations the simplex gradient can then be used to reorder the evaluations of the objective function associated with the directions used in the poll step or to update the mesh size parameter according to a sufficient decrease criterion neither of which requires new function evaluations we present these procedures in detail and apply them to a set of problems from the cuter collection numerical results show that these procedures can enhance significantly the practical performance of pattern search methods

convergence of the restricted nelder mead algorithm in two dimensions the nelder mead algorithm a longstanding direct search method for unconstrained optimization published in 1965 is designed to minimize a scalar valued function f of n real variables using only function values without any derivative information each nelder mead iteration is asso ciated with a nondegenerate simplex defined by n 1 vertices and their function values a typical iteration produces a new simplex by replacing the worst vertex by a new point despite the method s widespread use theoretical results have been limited for strictly convex objective functions of one variable with bounded level sets the algorithm always converges to the minimizer for such func tions of two variables the diameter of the simplex converges to zero but examples constructed by mckinnon show that the algorithm may converge to a nonminimizing point this paper considers the restricted nelder mead algorithm a variant that does not allow expansion steps in two dimen sions we show that for any nondegenerate starting simplex and any twice continuously differentiable function with positive definite hessian and bounded level sets the algorithm always converges to the minimizer the proof is based on treating the method as a discrete dynamical system and relies on several techniques that are nonstandard in convergence proofs for unconstrained optimization

kriging sparse polynomial dimensional decomposition surrogate model with adaptive refinement uncertainty quantification and global sensitivity analysis problems are made more difficult in the case of applications which involve expensive computer simulations this because a limited amount of simulations is available to build a sufficiently accurate metamodel of the quantities of interest in this work a numerical technique for the construction of a low cost and accurate metamodel is proposed having in mind applications with expensive computer codes two main points are intro duced firstly a technique which couples universal kriging with sparse polynomial dimensional decomposition pdd to build a metamodel with improved accuracy the polynomials selected by the adaptive pdd representation are used as a sparse basis to build an universal kriging surrogate model the second is a strategy derived from anisotropic mesh adaptation to adaptively add a fixed number of new training points to an existing design of experiments the convergence of the proposed algorithm is analyzed and assessed on different test functions with an increasing size of the input space finally the algorithm is used to propagate uncertainties in two high dimensional real problems related to the atmospheric reentry

a trust region algorithm for heterogeneous multiobjective optimization this paper presents a trust region method for multiobjective heterogeneous optimization problems one of the objective functions is an expensive black box function given for example by a time co

multi objective retrospective optimization using stochastic zigzag search abstract we propose a new retrospective optimization ro method for multi objective simulation optimization moso problems ro algorithms generate a sequence of sample path sp problems and solve these sp problems iteratively using a nonlinear optimizer in this study a stochastic zigzag search algorithm is chosen in the ro framework to solve sp problems the key idea of zigzag search is searching around the pareto front by applying an efficient local search procedure using the gradients of the objective functions many continuous moso problems have smooth objective functions and their non dominated objective function values form a smooth surface in the image space this fact motivates developing the zigzag search method embedded in ro for such relatively well posed moso problems a numerical implementation of this method multi objective retrospective optimization using zigzag search morozs is presented particularly for continuous bi objective simulation optimization boso problems with well connected pareto optimal solutions morozs is designed for boso problems in which a simulation oracle returns both objective function values and gradients due to the local nature of zigzag search morozs can only guarantee the asymptotic convergence to local pareto optimality the efficiency of morozs is studied using three boso problems with noisy objective functions and is compared to that of genetic algorithms based nsga ii and a recently developed method mo compass

feasible side global convergence in experimental optimization we propose a canonical form of the experimental optimization problem and review the state of the art methods to solve it as guarantees of global convergence to an optimal point via only feasible iterates are absent in these methods we present a set of sufficient conditions to enforce this behavior

class dependent discretization for inductive learning from continuous and mixed mode data inductive learning systems can be effectively used to acquire classification knowledge from examples many existing symbolic learning algorithms can be applied in domains with continuous attributes when integrated with a discretization algorithm to transform the continuous attributes into ordered discrete ones in this paper a new information theoretic discretization method optimized for supervised learning is proposed and described this approach seeks to maximize the mutual dependence as measured by the interdependence redundancy between the discrete intervals and the class labels and can automatically determine the most preferred number of intervals for an inductive learning application the method has been tested in a number of inductive learning examples to show that the class dependent discretizer can significantly improve the classification performance of many existing learning algorithms in domains containing numeric attributes

mesh adaptive direct search algorithms for constrained optimization this paper addresses the problem of minimization of a nonsmooth function under general nonsmooth constraints when no derivatives of the objective or constraint functions are available we introduce the mesh adaptive direct search mads class of algorithms which extends the generalized pattern search gps class by allowing local exploration called polling in an asymptotically dense set of directions in the space of optimization variables this means that under certain hypotheses including a weak constraint qualification due to rockafellar mads can treat constraints by the extreme barrier approach of setting the objective to infinity for infeasible points and treating the problem as unconstrained the main gps convergence result is to identify limit points hat x where the clarke generalized derivatives are nonnegative in a finite set of directions called refining directions although in the unconstrained case nonnegative combinations of these directions span the whole space the fact that there can only be finitely many gps refining directions limits rigorous justification of the barrier approach to finitely many linear constraints for gps the main result of this paper is that the general mads framework is flexible enough to allow the generation of an asymptotically dense set of refining directions along which the clarke derivatives are nonnegative we propose an instance of mads for which the refining directions are dense in the hypertangent cone at hat x with probability 1 whenever the iterates associated with the refining directions converge to a single hat x the instance of mads is compared to versions of gps on some test problems we also illustrate the limitation of our results with examples

orthomads a deterministic mads instance with orthogonal directions the purpose of this paper is to introduce a new way of choosing directions for the mesh adaptive direct search mads class of algorithms the advantages of this new orthomads instantiation of mads are that the polling directions are chosen deterministically ensuring that the results of a given run are repeatable and that they are orthogonal to each other which yields convex cones of missed directions at each iteration that are minimal in a reasonable measure convergence results for orthomads follow directly from those already published for mads and they hold deterministically rather than with probability one as is the case for ltmads the first mads instance the initial numerical results are quite good for both smooth and nonsmooth and constrained and unconstrained problems considered here

reducing the number of function evaluations in mesh adaptive direct search algorithms the mesh adaptive direct search mads class of algorithms is designed for nonsmooth optimization where the objective function and constraints are typically computed by launching a time consuming computer simulation each iteration of a mads algorithm attempts to improve the current best known solution by launching the simulation at a finite number of trial points common implementations of mads generate 2n trial points at each iteration where n is the number of variables in the optimization problem the objective of the present work is to dynamically reduce that number we present an algorithmic framework that reduces the number of simulations to exactly n 1 without impacting the theoretical guarantees from the convergence analysis numerical experiments are conducted for several different contexts the results suggest that these strategies allow the new algorithms to reach a better solution with fewer function evaluations

superlinear convergence and implicit filtering in this paper we show how the implicit filtering algorithm can be coupled with the bfgs quasi newton update to obtain a superlinearly convergent iteration if the noise in the objective function decays sufficiently rapidly as the optimal point is approached in this way we give insight into the observations of good performance in practice of quasi newton methods when they are coupled with implicit filtering we also report on numerical experiments that show how an implementation of implicit filtering that exploits these new results can improve the performance of the algorithm

pattern search algorithms for bound constrained minimization we present a convergence theory for pattern search methods for solving bound constrained nonlinear programs the analysis relies on the abstract structure of pattern search methods and an understanding of how the pattern interacts with the bound constraints this analysis makes it possible to develop pattern search methods for bound constrained problems while only slightly restricting the flexibility present in pattern search methods for unconstrained problems we prove global convergence despite the fact that pattern search methods do not have explicit information concerning the gradient and its projection onto the feasible region and consequently are unable to enforce explicitly a notion of sufficient feasible decrease

modeling and optimization of an ammonia reactor using a penalty like method abstract ammonia is one of the most important chemicals produced in the world due to its broad applicability modeling and simulation of process of ammonia has received considerable attention among the industrial processes ammonia is produced from the reaction of hydrogen and nitrogen at high temperature and high pressure along with a catalyst its production depends on temperature of feed gas at the top of the reactor the partial pressures of the reactants and the reactor length the optimal design problem requires obtaining the optimal reactor length with maximum economic return subject to a number of equality constraints involving solution of coupled nonlinear differential equations a more efficient algorithm for the solution of optimization constrained differential equations is proposed by coupling numerical techniques barrier function with a direct search method with runge kutta method the global convergence of the proposed algorithm is established under some usual conditions numerical results showed that the resulting algorithm based only on function values is highly competitive with other global optimization methods

controller design via nonsmooth multidirectional search we propose an algorithm which combines multidirectional search mds with nonsmooth optimization techniques to solve difficult problems in automatic control applications include static and fixed order output feedback controller design simultaneous stabilization h 2 h infty synthesis and much else we show how to combine direct search techniques with nonsmooth descent steps in order to obtain convergence certificates in the presence of nonsmoothness our technique is efficient when small and medium size controllers for plants with large state dimension are sought our numerical testing includes several benchmark examples for instance our algorithm needs 0 41 s to compute a static output feedback stabilizing controller for the boeing 767 flutter benchmark problem e e j davison ifac technical committee reports pergamon press oxford 1990 a system with 55 states the first static controller without performance specifications for this system was obtained in j burke a lewis and m overton siam j optim 15 2003 pp 751 779

a merit function approach for direct search in this paper it is proposed to equip direct search methods with a general procedure to minimize an objective function possibly nonsmooth without using derivatives and subject to constraints on the variables one aims at considering constraints most likely nonlinear or nonsmooth for which the derivatives of the corresponding functions are also unavailable the novelty of this contribution relies mostly on how relaxable constraints are handled such constraints which can be relaxed during the course of the optimization are taken care of by a merit function and if necessary by a restoration procedure constraints that are unrelaxable when present are treated by an extreme barrier approach one is able to show that the resulting merit function direct search algorithm exhibits global convergence properties for first order stationary constraints as in the progressive barrier method c audet and j e dennis jr siam j optim 20 2009 pp 445 472 we provide a mechanism to indicate the transf

optimality condition and complexity analysis for linearly constrained optimization without differentiability on the boundary in this paper we consider the minimization of a continuous function that is potentially not differentiable or not twice differentiable on the boundary of the feasible region by exploiting an interior point technique we present first and second order optimality conditions for this problem that reduces to classical ones when the derivative on the boundary is available for this type of problems existing necessary conditions often rely on the notion of subdifferential or become non trivially weaker than the kkt condition in the twice differentiable counterpart problems in contrast this paper presents a new set of first and second order necessary conditions that are derived without the use of subdifferential and reduce to exactly the kkt condition when twice differentiability holds as a result these conditions are stronger than some existing ones considered for the discussed minimization problem when only non negativity constraints are present to solve for these optimality conditions in the special but important case of linearly constrained problems we present two novel interior point trust region algorithms and show that their worst case computational efficiency in achieving the potentially stronger optimality conditions match the best known complexity bounds since this work considers a more general problem than those in the literature our results also indicate that best known existing complexity bounds are actually held for a wider class of nonlinear programming problems this new development is significant since optimality conditions play a fundamental role in computational optimization and more and more nonlinear and nonconvex problems need to be solved in practice

new results on a continuously differentiable exact penalty function the main motivation of this paper is to weaken the conditions that imply the correspondence between the solution of a constrained problem and the unconstrained minimization of a continuously differentiable function in particular a new continuously differentiable exact penalty function is proposed for the solution of nonlinear programming problems under mild assumptions a complete equivalence can be established between the solution of the original constrained problem and the unconstrained minimization of this penalty function on a perturbation of the feasible set this new penalty function and its exactness properties allow us to define globally and superlinearly convergent algorithms to solve nonlinear programming problems as an example a newton type algorithm is described which converges locally in one iteration in case of quadratic programming problems

modeling an augmented lagrangian for blackbox constrained optimization constrained blackbox optimization is a difficult problem with most approaches coming from the mathematical programming literature the statistical literature is sparse especially in addressing problems with nontrivial constraints this situation is unfortunate because statistical methods have many attractive properties global scope handling noisy objectives sensitivity analysis and so forth to narrow that gap we propose a combination of response surface modeling expected improvement and the augmented lagrangian numerical optimization framework this hybrid approach allows the statistical model to think globally and the augmented lagrangian to act locally we focus on problems where the constraints are the primary bottleneck requiring expensive simulation to evaluate and substantial modeling effort to map out in that context our hybridization presents a simple yet effective solution that allows existing objective oriented statistical approaches like those based on gaussian process surrogates

on the global convergence of derivative free methods for unconstrained optimization in this paper starting from the study of the common elements that some globally convergent direct search methods share a general convergence theory is established for unconstrained minimization methods employing only function values the introduced convergence conditions are useful for developing and analyzing new derivative free algorithms with guaranteed global convergence as examples we describe three new algorithms which combine pattern and line search approaches

parallel space decomposition of the mesh adaptive direct search algorithm this paper describes a parallel space decomposition psd technique for the mesh adaptive direct search mads algorithm mads extends a generalized pattern search for constrained nonsmooth optimization problems the objective of the present work is to obtain good solutions to larger problems than the ones typically solved by mads the new method psd mads is an asynchronous parallel algorithm in which the processes solve problems over subsets of variables the convergence analysis based on the clarke calculus is essentially the same as for the mads algorithm a practical implementation is described and some numerical results on problems with up to 500 variables illustrate the advantages and limitations of psd mads

on the convergence of grid based methods for unconstrained optimization the convergence of direct search methods for unconstrained minimization is examined in the case where the underlying method can be interpreted as a grid or pattern search over successively refined meshes an important aspect of the main convergence result is that translation rotation scaling and shearing of the successive grids are allowed

a method for stochastic constrained optimization using derivative free surrogate pattern search and collocation recent advances in coupling novel optimization methods to large scale computing problems have opened the door to tackling a diverse set of physically realistic engineering design problems a large computational overhead is associated with computing the cost function for most practical problems involving complex physical phenomena such problems are also plagued with uncertainties in a diverse set of parameters we present a novel stochastic derivative free optimization approach for tackling such problems our method extends the previously developed surrogate management framework smf to allow for uncertainties in both simulation parameters and design variables the stochastic collocation scheme is employed for stochastic variables whereas kriging based surrogate functions are employed for the cost function this approach is tested on four numerical optimization problems and is shown to have significant improvement in efficiency over traditional monte carlo schemes problems with multiple probabilistic constraints are also discussed

a derivative free algorithm for inequality constrained nonlinear programming via smoothing of an ell_ infty penalty function in this paper we consider inequality constrained nonlinear optimization problems where the first order derivatives of the objective function and the constraints cannot be used our starting point is the possibility to transform the original constrained problem into an unconstrained or linearly constrained minimization of a nonsmooth exact penalty function this approach shows two main difficulties the first one is the nonsmoothness of this class of exact penalty functions which may cause derivative free codes to converge to nonstationary points of the problem the second one is the fact that the equivalence between stationary points of the constrained problem and those of the exact penalty function can only be stated when the penalty parameter is smaller than a threshold value which is not known a priori in this paper we propose a derivative free algorithm which overcomes the preceding difficulties and produces a sequence of points that admits a subsequence converging to a karush kuhn tucker point of the constrained problem in particular the proposed algorithm is based on a smoothing of the nondifferentiable exact penalty function and includes an updating rule which after at most a finite number of updates is able to determine a right value for the penalty parameter furthermore we present the results obtained on a real world problem concerning the estimation of parameters in an insulin glucose model of the human body

unconstrained derivative free optimization by successive approximation we present an algorithmic framework for unconstrained derivative free optimization based on dividing the search space in regions partitions every partition is assigned a representative point the representative points form a grid a piecewise constant approximation to the function subject to optimization is constructed using a partitioning and its corresponding grid the convergence of the framework to a stationary point of a continuously differentiable function is guaranteed under mild assumptions the proposed framework is appropriate for upgrading heuristics that lack mathematical analysis into algorithms that guarantee convergence to a local minimizer a convergent variant of the nelder mead algorithm that conforms to the given framework is constructed the algorithm is compared to two previously published convergent variants of the nm algorithm the comparison is conducted on the more garbow hillstrom set of test problems and on four variably dimensional functions with dimension up to 100 the results of the comparison show that the proposed algorithm outperforms both previously published algorithms

fuzzy equalization in the construction of fuzzy sets in this note we introduce and study a concept of fuzzy equalization fuzzy equalization concerns a process of building information granules that are semantically and experimentally meaningful the experimental relevance of a given information granule fuzzy set is directly linked with an encapsulation of a certain experimental evidence conveyed by the respective probability density function of available data we establish a detailed equalization algorithm developed for triangular fuzzy sets the study elaborates on the role of the fuzzy equalization in system design

pattern search ranking and selection algorithms for mixed variable simulation based optimization the class of generalized pattern search gps algorithms for mixed variable optimization is extended to problems with stochastic objective functions because random noise in the objective function makes it more difficult to compare trial points and ascertain which points are truly better than others replications are needed to generate sufficient statistical power to draw conclusions rather than comparing pairs of points the approach taken here augments pattern search with a ranking and selection r s procedure which allows for comparing many function values simultaneously asymptotic convergence for the algorithm is established numerical issues are discussed and performance of the algorithm is studied on a set of test problems

accelerated random search for constrained global optimization assisted by radial basis function surrogates abstract the accelerated random search ars algorithm appel et al 2004 is a stochastic algorithm for the global optimization of black box functions subject only to bound constraints ars iteratively samples uniformly within a box centered at the current best point if the sample point is worse than the current best point then the size of the box is reduced thereby making it more likely to generate an improving solution if the current best point is not yet a local minimum otherwise if the sample point is an improvement over the current best point then the size of the box is reset to the initial value so that the box covers the entire search space ars has been shown theoretically and numerically to converge to the global minimum faster than the pure random search prs algorithm on bound constrained problems this article develops the constrained ars cars algorithm which is an extension of ars for optimization problems with black box inequality constraints under some mild conditions on the constrained optimization problem we prove the convergence to the global minimum in a probabilistic sense of a class of stochastic search algorithms called scattered uniform random search scat urs algorithms which includes cars as a special case to improve performance of cars on computationally expensive simulation based problems we incorporate radial basis function rbf surrogate models to approximate the objective and constrained functions and refer to the resulting algorithm as cars rbf numerical experiments show that cars consistently and substantially outperforms both the pure random search prs and accelerated particle swarm optimization apso algorithms yang 2010 on 31 test problems with dimensions ranging from 2 to 20 moreover cars rbf is an improvement over cars outperforms a sequential penalty derivative algorithm and competes with constrlmsrbf regis 2011 on the same test problems finally sensitivity analysis of cars rbf to the type of rbf model used shows that the cubic rbf model generally yields the best results on the test problems among six types of rbf models considered including the thin plate spline gaussian and multiquadric models

finding local optima of high dimensional functions using direct search methods this paper focuses on a subclass of box constrained non linear optimization problems we are particularly concerned with settings where gradient information is unreliable or too costly to calculate and the function evaluations themselves are very costly this encourages the use of derivative free optimization methods and especially a subclass of these referred to as direct search methods the thrust of our investigation is twofold first we implement and evaluate a number of traditional direct search methods according to the premise that they should be suitable as local optimizers when used in a metaheuristic framework second we introduce a new direct search method based on scatter search designed to remedy the lack of a good derivative free method for solving problems of high dimensions our new direct search method has convergence properties comparable to those of existing methods in addition to being able to solve larger problems more effectively

a pattern search filter method for nonlinear programming without derivatives this paper formulates and analyzes a pattern search method for general constrained optimization based on filter methods for step acceptance roughly a filter method accepts a step that improves either the objective function value or the value of some function that measures the constraint violation the new algorithm does not compute or approximate any derivatives penalty constants or lagrange multipliers a key feature of the new algorithm is that it preserves the division into search and local poll steps which allows the explicit use of inexpensive surrogates or random search heuristics in the search step it is shown here that the algorithm identifies limit points at which optimality conditions depend on local smoothness of the functions and to a greater extent on the choice of a certain set of directions stronger optimality conditions are guaranteed for smoother functions and in the constrained case for a fortunate choice of the directions on which the algorithm depends these directional conditions generalize those given previously for linear constraints but they do not require a feasible starting point in the absence of general constraints the proposed algorithm and its convergence analysis generalize previous work on unconstrained bound constrained and linearly constrained generalized pattern search the algorithm is illustrated on some test examples and on an industrial wing planform engineering design application

a naive multi scale search algorithm for global optimization problems this paper proposes a multi scale search algorithm for solving global optimization problems given a finite number of function evaluations we refer to this algorithm as the naive multi scale search optimization nmso nmso looks for the optimal solution by optimistically partitioning the search space over multiple scales in a hierarchical fashion based on a weak assumption about the function smoothness we present a theoretical analysis on its finite time and asymptotic convergence an empirical assessment of the algorithm has been conducted on the noiseless black box optimization benchmarking bbob testbed and compared with the state of the art optimistic as well as stochastic algorithms moreover the efficacy of nmso has been validated on the black box optimization competition within the gecco 15 conference where it has secured the third place out of twenty eight participating algorithms overall nmso is suitable for problems with limited function evaluations low dimensionality search space and objective functions that are separable or multi modal otherwise it is comparable with the top performing algorithms

a progressive barrier for derivative free nonlinear programming we propose a new constraint handling approach for general constraints that is applicable to a widely used class of constrained derivative free optimization methods as in many methods that allow infeasible iterates constraint violations are aggregated into a single constraint violation function as in filter methods a threshold or barrier is imposed on the constraint violation function and any trial point whose constraint violation function value exceeds this threshold is discarded from consideration in the new algorithm unlike the filter method the amount of constraint violation subject to the barrier is progressively decreased adaptively as the iteration evolves we test this progressive barrier pb approach versus the extreme barrier eb with the generalized pattern search gps and the lower triangular mesh adaptive direct search ltmads methods for nonlinear derivative free optimization tests are also conducted using the gps filter which uses a version of the fletcher leyffer filter approach we know that gps cannot be shown to yield kkt points with this strategy or the filter but we use the clarke nonsmooth calculus to prove clarke stationarity of the sequences of feasible and infeasible trial points for ltmads pb numerical experiments are conducted on three academic test problems with up to 50 variables and on a chemical engineering problem the new ltmads pb method generally outperforms our ltmads eb in the case where no feasible initial points are known and it does as well when feasible points are known which leads us to recommend ltmads pb thus the ltmads pb is a useful practical extension of our earlier ltmads eb algorithm particularly in the common case for real problems where no feasible point is known the same conclusions hold for gps pb versus gps eb

automatically determine the membership function based on the maximum entropy principle abstract fuzzy set theory has been successfully applied to many image processing and pattern recognition tasks in order to use the fuzzy logic approach membership functions for fuzzy sets must be determined first because many properties of images such as brightness of graylevels are strongly context dependent it is not easy to determine the membership function correctly in this paper the fuzzy set brightness of graylevels of an image is used as an example to illustrate how the membership function can be determined automatically we start with the concept of fuzzy event and use the maximum entropy principle as the criterion to find a membership function which will best represent the membership of brightness for each graylevel in an image that is the membership function is determined by finding a membership function such that the corresponding fuzzy event has maximum entropy the membership of brightness can be represented by an s function whose shape can be determined by the parameters a b and c the problem becomes to find a best parameter combination aopt bopt copt which is a combinatorial optimization problem the simulated annealing algorithm is used to solve this problem in this paper we have done the experiments on several images the results have shown that the proposed method can automatically and effectively find the brightness membership function for images the robustness of the proposed algorithm is also proved by the experiments though we used image processing as the application domain of the proposed approach the basic idea can be extended to other applications easily

objective derivative free methods for constrained optimization we propose feasible descent methods for constrained minimization that do not make explicit use of the derivative of the objective function the methods iteratively sample the objective function value along a finite set of feasible search arcs and decrease the sampling stepsize if an improved objective function value is not sampled the search arcs are obtained by projecting search direction rays onto the feasible set and the search directions are chosen such that a subset approximately generates the cone of first order feasible variations at the current iterate we show that these methods have desirable convergence properties under certain regularity assumptions on the constraints in the case of linear constraints the projections are redundant and the regularity assumptions hold automatically numerical experience with the methods in the linearly constrained case is reported

generalized pattern searches with derivative information a common question asked by users of direct search algorithms is how to use derivative information at iterates where it is available this paper addresses that question with respect to generalized pattern search gps methods for unconstrained and linearly constrained optimization specifically this paper concentrates on the gps pollstep polling is done to certify the need to refine the current mesh and it requires o n function evaluations in the worst case we show that the use of derivative information significantly reduces the maximum number of function evaluations necessary for pollsteps even to a worst case of a single function evaluation with certain algorithmic choices given here furthermore we show that rather rough approximations to the gradient are sufficient to reduce the pollstep to a single function evaluation we prove that using these less expensive pollsteps does not weaken the known convergence properties of the method all of which depend only on the pollstep

global convergence of radial basis function trust region derivative free algorithms we analyze globally convergent derivative free trust region algorithms relying on radial basis function interpolation models our results extend the recent work of conn scheinberg and vicente siam j optim 20 2009 pp 387 415 to fully linear models that have a nonlinear term we characterize the types of radial basis functions that fit in our analysis and thus show global convergence to first order critical points for the orbit algorithm of wild regis and shoemaker siam j sci comput 30 2008 pp 3197 3219 using orbit we present numerical results for different types of radial basis functions on a series of test problems we also demonstrate the use of orbit in finding local minima on a computationally expensive environmental engineering problem

convergence of mesh adaptive direct search to second order stationary points a previous analysis of second order behavior of generalized pattern search algorithms for unconstrained and linearly constrained minimization is extended to the more general class of mesh adaptive direct search mads algorithms for general constrained optimization because of the ability of mads to generate an asymptotically dense set of search directions we are able to establish reasonable conditions under which a subsequence of mads iterates converges to a limit point satisfying second order necessary or sufficient optimality conditions for general set constrained optimization problems

optimization by direct search new perspectives on some classical and modern methods direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis nonetheless users remained loyal to these methods most of which were easy to program some of which were reliable in the past fifteen years these methods have seen a revival due in part to the appearance of mathematical analysis as well as to interest in parallel and distributed com puting this review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results the underlying principles allow general ization to handle bound constraints and linear constraints we also discuss extensions to problems with nonlinear constraints

a comparison of three methods for selecting values of input variables in the analysis of output from a computer code two types of sampling plans are examined as alternatives to simple random sampling in monte carlo studies these plans are shown to be improvements over simple random sampling with respect to variance for a class of estimators which includes the sample mean and the empirical distribution function

spherical mesh adaptive direct search for separating quasi uncorrelated sources by range based independent component analysis it is seemingly paradoxical to the classical definition of the independent component analysis ica that in reality the true sources are often not strictly uncorrelated with this in mind this letter concerns a framework to extract quasi uncorrelated sources with finite supports by optimizing a range based contrast function under unit norm constraints to handle the inherent scaling indeterminacy of ica but without orthogonality constraints albeit the appealing contrast properties of the range based function e g the absence of mixing local optima the function is not differentiable everywhere unfortunately there is a dearth of literature on derivative free optimizers that effectively handle such a nonsmooth yet promising contrast function this is the compelling reason for the design of a nonsmooth optimization algorithm on a manifold of matrices having unit norm columns with the following objectives to ascertain convergence to a clarke stationary point of the contrast function and adhere to the necessary unit norm constraints more naturally the proposed nonsmooth optimization algorithm crucially relies on the design and analysis of an extension of the mesh adaptive direct search mads method to handle locally lipschitz objective functions defined on the sphere the applicability of the algorithm in the ica domain is demonstrated with simulations involving natural face aerial and texture images

sequential penalty derivative free methods for nonlinear constrained optimization we consider the problem of minimizing a continuously differentiable function of several variables subject to smooth nonlinear constraints we assume that the first order derivatives of the objective function and of the constraints can be neither calculated nor explicitly approximated hence every minimization procedure must use only a suitable sampling of the problem functions these problems arise in many industrial and scientific applications and this motivates the increasing interest in studying derivative free methods for their solution the aim of the paper is to extend to a derivative free context a sequential penalty approach for nonlinear programming this approach consists in solving the original problem by a sequence of approximate minimizations of a merit function where penalization of constraint violation is progressively increased in particular under some standard assumptions we introduce a general theoretical result regarding the connections between the sampling technique and the updating of the penalization which are able to guarantee convergence to stationary points of the constrained problem on the basis of the general theoretical result we propose a new method and prove its convergence to stationary points of the constrained problem the computational behavior of the method has been evaluated both on a set of test problems and on a real application the obtained results and the comparison with other well known derivative free software show the viability of the proposed sequential penalty approach

inexact restoration method for derivative free optimization with smooth constraints a new method is introduced for solving constrained optimization problems in which the derivatives of the constraints are available but the derivatives of the objective function are not the method is based on the inexact restoration framework by means of which each iteration is divided in two phases in the first phase one considers only the constraints in order to improve feasibility in the second phase one minimizes a suitable objective function subject to a linear approximation of the constraints the second phase must be solved using derivative free methods an algorithm introduced recently by kolda lewis and torczon for linearly constrained derivative free optimization is employed for this purpose under usual assumptions convergence to stationary points is proved a computer implementation is described and numerical experiments are presented

a nonderivative version of the gradient sampling algorithm for nonsmooth nonconvex optimization we give a nonderivative version of the gradient sampling algorithm of burke lewis and overton for minimizing a locally lipschitz function f on mathbb r n that is continuously differentiable on an open dense subset instead of gradients of f we use estimates of gradients of the steklov averages of f obtained by convolution with mollifiers which require f values only we show that the nonderivative version retains the convergence properties of the gradient sampling algorithm in particular with probability 1 it either drives the f values to infty or each of its cluster points is clarke stationary for f

a linesearch based derivative free approach for nonsmooth constrained optimization in this paper we propose new linesearch based methods for nonsmooth constrained optimization problems when first order information on the problem functions is not available in the first part we describe a general framework for bound constrained problems and analyze its convergence toward stationary points using the clarke jahn directional derivative in the second part we consider inequality constrained optimization problems where both objective function and constraints can possibly be nonsmooth in this case we first split the constraints into two subsets difficult general nonlinear constraints and simple bound constraints on the variables then we use an exact penalty function to tackle the difficult constraints and we prove that the original problem can be reformulated as the bound constrained minimization of the proposed exact penalty function finally we use the framework developed for the bound constrained case to solve the penalized problem moreover we prove that every accumulation poin

finding optimal algorithmic parameters using derivative free optimization the objectives of this paper are twofold we devise a general framework for identifying locally optimal algorithmic parameters algorithmic parameters are treated as decision variables in a problem for which no derivative knowledge or existence is assumed a derivative free method for optimization seeks to minimize some measure of performance of the algorithm being fine tuned this measure is treated as a black box and may be chosen by the user examples are given in the text the second objective is to illustrate this framework by specializing it to the identification of locally optimal trust region parameters in unconstrained optimization the derivative free method chosen to guide the process is the mesh adaptive direct search a generalization of pattern search methods we illustrate the flexibility of the latter and in particular make provision for surrogate objectives locally optimal parameters with respect to overall computational time on a set of test problems are identified each function call may take several hours and may not always return a predictable result a tailored surrogate function is used to guide the search towards a local solution the parameters thus identified differ from traditionally used values and allow one to solve a problem that remained otherwise unsolved in a reasonable time using traditional values

analysis of direct searches for discontinuous functions it is known that the clarke generalized directional derivative is nonnegative along the limit directions generated by directional direct search methods at a limit point of certain subsequences of unsuccessful iterates if the function being minimized is lipschitz continuous near the limit point in this paper we generalize this result for discontinuous functions using rockafellar generalized directional derivatives upper subderivatives we show that rockafellar derivatives are also nonnegative along the limit directions of those subsequences of unsuccessful iterates when the function values converge to the function value at the limit point this result is obtained assuming that the function is directionally lipschitz with respect to the limit direction it is also possible under appropriate conditions to establish more insightful results by showing that the sequence of points generated by these methods eventually approaches the limit point along the locally best branch or step function when the number of steps is equal to two the results of this paper are presented for constrained optimization and illustrated numerically

unfair coins and necessity measures towards a possibilistic interpretation of histograms this short note defines a bijective mapping which turns a probability measure into a possibility measure this transformation has intuitive grounds especially the converse mapping corresponds to the usual assignment of identical probability values to elementary events of a given set when no information is available

efficient fuzzy partition of pattern space for classification problems abstract this paper proposes an efficient fuzzy partition method of a pattern space for classification problems the proposed method is based on the sequential subdivision of fuzzy subspaces and the generated fuzzy subspaces have different sizes in the proposed method first an n dimensional pattern space is divided into 2 n fuzzy subspaces with the same size next one of the fuzzy subspaces is selected and subdivided into 2 n fuzzy subspaces this procedure is iterated until a stopping condition is satisfied some criteria for selecting a fuzzy subspace to be subdivided are proposed and compared with each other by computer simulations the proposed method is also compared with other fuzzy classification methods

induction of fuzzy decision trees abstract most decision tree induction methods used for extracting knowledge in classification problems do not deal with cognitive uncertainties such as vagueness and ambiguity associated with human thinking and perception in this paper cognitive uncertainties involved in classification problems are explicitly represented measured and incorporated into the knowledge induction process a fuzzy decision tree induction method which is based on the reduction of classification ambiguity with fuzzy evidence is developed fuzzy decision trees represent classification knowledge more naturally to the way of human thinking and are more robust in tolerating imprecise conflict and missing information

direct multisearch for multiobjective optimization in practical applications of optimization it is common to have several conflicting objective functions to optimize frequently these functions are subject to noise or can be of black box type preventing the use of derivative based techniques we propose a novel multiobjective derivative free methodology calling it direct multisearch dms which does not aggregate any of the objective functions our framework is inspired by the search poll paradigm of direct search methods of directional type and uses the concept of pareto dominance to maintain a list of nondominated points from which the new iterates or poll centers are chosen the aim of our method is to generate as many points in the pareto front as possible from the polling procedure itself while keeping the whole framework general enough to accommodate other disseminating strategies in particular when using the here also optional search step dms generalizes to multiobjective optimization moo all direct search methods of directional type

analysis of generalized pattern searches this paper contains a new convergence analysis for the lewis and torczon generalized pattern search gps class of methods for unconstrained and linearly constrained optimization this analysis is motivated by a desire to understand the successful behavior of the algorithm under hypotheses that are satisfied by many practical problems specifically even if the objective function is discontinuous or extended valued the methods find a limit point with some minimizing properties simple examples show that the strength of the optimality conditions at a limit point depends not only on the algorithm but also on the directions it uses and on the smoothness of the objective at the limit point in question the contribution of this paper is to provide a simple convergence analysis that supplies detail about the relation of optimality conditions to objective smoothness properties and to the defining directions for the algorithm and it gives previous results as corollaries

global convergence of radial basis function trust region algorithms for derivative free optimization we analyze globally convergent derivative free trust region algorithms relying on radial basis function interpolation models our results extend the recent work of conn scheinberg and vicente siam j optim 20 2009 pp 387 415 to fully linear models that have a nonlinear term we characterize the types of radial basis functions that fit in our analysis and thus show global convergence to first order critical points for the orbit algorithm of wild regis and shoemaker siam j sci comput 30 2008 pp 3197 3219 using orbit we present numerical results for different types of radial basis functions on a series of test problems we also demonstrate the use of orbit in finding local minima on a computationally expensive environmental engineering problem involving remediation of contaminated groundwater

a mesh adaptive direct search algorithm for multiobjective optimization abstract this work studies multiobjective optimization mop of nonsmooth functions subject to general constraints we first present definitions and optimality conditions as well as some single objective formulations of mop parameterized with respect to some reference point in the space of objective functions next we propose a new algorithm called m ulti m ads multiobjective mesh adaptive direct search for mop m ulti m ads generates an approximation of the pareto front by solving a series of single objective formulations of mop generated using the nbi natural boundary intersection framework these single objective problems are solved using the m ads mesh adaptive direct search algorithm for constrained nonsmooth optimization the pareto front approximation is shown to satisfy some first order necessary optimality conditions based on the clarke calculus m ulti m ads is then tested on problems from the literature with different pareto front landscapes and on a styrene production process simulation problem from chemical engineering

globally convergent evolution strategies in this paper we show how to modify a large class of evolution strategies es s for unconstrained optimization to rigorously achieve a form of global convergence meaning convergence to stationary points independently of the starting point the type of es under consideration recombines the parent points by means of a weighted sum around which the offspring points are computed by random generation one relevant instance of such an es is covariance matrix adaptation es cma es the modifications consist essentially of the reduction of the size of the steps whenever a sufficient decrease condition on the function values is not verified when such a condition is satisfied the step size can be reset to the step size maintained by the es s themselves as long as this latter one is sufficiently large we suggest a number of ways of imposing sufficient decrease for which global convergence holds under reasonable assumptions in particular density of certain limit directions in the unit sphere given a limited budget of function evaluations our numerical experiments have shown that the modified cma es is capable of further progress in function values moreover we have observed that such an improvement in efficiency comes without weakening significantly the performance of the underlying method in the presence of several local minimizers

on polytopes with congruent projections or sections abstract let 2 k d 1 and let p and q be two convex polytopes in e d assume that their projections p h q h onto every k dimensional subspace h are congruent in this paper we show that p and q or p and q are translates of each other we also prove an analogous result for sections by showing that p q or p q provided the polytopes contain the origin in their interior and their sections p h q h by every k dimensional subspace h are congruent

fuzzy partitioning of continuous attributes through discretization methods to construct fuzzy decision tree classifiers abstract generating membership functions mfs from data is one of the fundamental challenges associated with the applications of fuzzy set theory this paper proposes a new two step algorithm which uses discretization methods for initial partitioning to generate mfs from data in the first step discretization algorithm divides domain of attributes to several partitions and then in the second step an mf is defined on each partition four different methods are proposed to define mfs in the second step the first method is based on partition width the second is based on standard deviation of examples the third is based on coverage rate of neighbor partitions npcr and the last one is based on coverage rate of partition pcr coverage rate of partition and coverage rate of neighbor partition are two new introduced parameters which can be used for mf generation in addition this paper proposes a new mf generation algorithm called fuzzy entropy based fuzzy partitioning febfp which is a specific version of the proposed two step algorithm with some modifications febfp uses fuzzy entropy of partition to generate mfs and involves the parameters of mfs in the process of mf generation to combine two steps of the algorithm non parametric statistical tests are used to compare fuzzy decision trees fdts induced using the mfs generated by the proposed methods employing different discretization algorithms as well as four mf generation methods experimental results show that eight methods outperform the others in terms of both accuracy and number of nodes among them trapezoidal mfs that are defined by pcr on partitions generated by zeta discretization algorithm outperform the others when the accuracy and complexity of fdt have the same degree of importance moreover the results show that the pcr and npcr mf definition methods perform better than the other ones

inferring decision trees using the minimum description length principle abstract we explore the use of rissanen s minimum description length principle for the construction of decision trees empirical results comparing this approach to other methods are given

an extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons in a recently published paper in jmlr dem sar 2006 recommends a set of non parametric statistical tests and procedures which can be safely used for comparing the performance of classifiers over multiple data sets after studying the paper we realize that the paper correctly introduces the basic procedures and some of the most advanced ones when comparing a control method however it does not deal with some advanced topics in depth regarding these topics we focus on more powerful proposals of statistical procedures for comparing n n classifiers moreover we illustrate an easy way of obtaining adjusted and comparable p values in multiple comparison procedures

a derivative free algorithm for linearly constrained finite minimax problems in this paper we propose a new derivative free algorithm for linearly constrained finite minimax problems due to the nonsmoothness of this class of problems standard derivative free algorithms can locate only points which satisfy weak necessary optimality conditions in this work we define a new derivative free algorithm which is globally convergent toward standard stationary points of the finite minimax problem to this end we convert the original problem into a smooth one by using a smoothing technique based on the exponential penalty function of kort and bertsekas this technique depends on a smoothing parameter which controls the approximation to the finite minimax problem the proposed method is based on a sampling of the smooth function along a suitable search direction and on a particular updating rule for the smoothing parameter that depends on the sampling stepsize numerical results on a set of standard minimax test problems are reported

regularized virtual fields method for mechanical properties identification of composite materials abstract this paper presents a new material parameters identification method based on the virtual fields method vfm where a regularization term coming from a micromechanical model has been added in this regularized virtual fields method rvfm the system of equations associated to the vfm is solved within a constrained optimization framework the method was applied to virtual scanning electron microscope in situ tests aiming at determining the local properties of uni directional composites loaded in the transverse direction and for which the strain fields were obtained through digital image correlation dic the method was tested for composites with various volume fractions and properties contrasts and under different boundary conditions the potentials and limitations of the algorithm in the presence of noisy images were investigated and compared to that of the vfm and the other finite element based methods the corresponding results indicate that the proposed rvfm is more efficient than the other identification techniques in terms of both accuracy and computational time an optimum size of region of interest was also determined by taking into account dic magnification requirements and based on the accuracy of the identified parameters this kind of study can be very useful for determining a priori the level of resolution in the imaging system required for a favorable accuracy of the identified parameters

an adaptive strategy on the error of the objective functions for uncertainty based derivative free optimization in this work a strategy is developed to deal with the error affecting the objective functions in uncertainty based optimization we refer to the problems where the objective functions are the statistics of a quantity of interest computed by an uncertainty quantification technique that propagates some uncertainties of the input variables through the system under consideration in real problems the statistics are computed by a numerical method and therefore they are affected by a certain level of error depending on the chosen accuracy the errors on the objective function can be interpreted with the abstraction of a bounding box around the nominal estimation in the objective functions space in addition in some cases the uncertainty quantification methods providing the objective functions also supply the possibility of adaptive refinement to reduce the error bounding box the novel method relies on the exchange of information between the outer loop based on the optimization algorithm and the inner uncertainty quantification loop in particular in the inner uncertainty quantification loop a control is performed to decide whether a refinement of the bounding box for the current design is appropriate or not in single objective problems the current bounding box is compared to the current optimal design in multi objective problems the decision is based on the comparison of the error bounding box of the current design and the current pareto front with this strategy fewer computations are made for clearly dominated solutions and an accurate estimate of the objective function is provided for the interesting non dominated solutions the results presented in this work prove that the proposed method improves the efficiency of the global loop while preserving the accuracy of the final pareto front the proposed method improves the efficiency of the uncertainty based optimization by reducing the number of objective evaluations the method relies on the presence of a derivative free optimization method and an adaptive uncertainty quantification method the proposed method is conceived for multi objective problems but it applies to single objective optimization as well results prove that the proposed method saves computations on the dominated solutions and predicts with accuracy the pareto front

a globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds we give a pattern search method for nonlinearly constrained optimization that is an adaption of a bound constrained augmented lagrangian method first proposed by conn gould and toint siam j numer anal 28 1991 pp 545 572 in the pattern search adaptation we solve the bound constrained subproblem approximately using a pattern search method the stopping criterion proposed by conn gould and toint for the solution of the subproblem requires explicit knowledge of derivatives such information is presumed absent in pattern search methods however we show how we can replace this with a stopping criterion based on the pattern size in a way that preserves the convergence properties of the original algorithm in this way we proceed by successive inexact bound constrained minimization without knowing exactly how inexact the minimization is as far as we know this is the first provably convergent direct search method for general nonlinear programming