problem method classification in optimization and control abstract there are many classifications in optimization and control continuous discrete integer problems convex or concave minimization steepest descent branch and bound path following space filling random search methods smooth or non smooth problems etc the aim of this short note is not to propose yet another classification but to clarify existing ambiguities and imprecisions in problem formulations and to emphasize certain fundamental distinctions in solution methods currently used

on the worst case complexity of potential reduction algorithms for linear programming there are several classes of interior point algorithms that solve linear programming problems in o nl iterations among them several potential reduction algorithms combine both theoretical o nl iterations and practical efficiency as they allow the flexibility of line searches in the potential function and thus can lead to practical implementations it is a significant open question whether interior point algorithms can lead to better complexity bounds in the present paper we give some negative answers to this question for the class of potential reduction algorithms we show that even if we allow line searches in the potential function and even for problems that have network structure the bound o nl is tight for several potential reduction algorithms i e there is a class of examples with network structure in which the algorithms need at least nl iterations to find an optimal solution

an approximate approach of global optimization for polynomial programming problems abstract many methods for solving polynomial programming problems can only find locally optimal solutions this paper proposes a method for finding the approximately globally optimal solutions of polynomial programs representing a bounded continuous variable x i as the addition of a discrete variable d i and a small variable i a polynomial term x i x i can be expanded as the sum of d i x j d j i and i j a procedure is then developed to fully linearize d i x j and d j e i and to approximately linearize i j with an error below a pre specified tolerance this linearization procedure can also be extended to higher order polynomial programs several polynomial programming examples in the literature are tested to demonstrate that the proposed method can systematically solve these examples to find the global optimum within a pre specified error

global optimization and simulated annealing in this paper we are concerned with global optimization which can be defined as the problem of finding points on a bounded subset of n in which some real valued functionf assumes its optimal maximal or minimal value we present a stochastic approach which is based on the simulated annealing algorithm the approach closely follows the formulation of the simulated annealing algorithm as originally given for discrete optimization problems the mathematical formulation is extended to continuous optimization problems and we prove asymptotic convergence to the set of global optima furthermore we discuss an implementation of the algorithm and compare its performance with other well known algorithms the performance evaluation is carried out for a standard set of test functions from the literature

hesitant adaptive search the distribution of the number of iterations to convergence hesitant adaptive search is a stochastic optimisation procedure which accommodates hesitation or pausing at objective function values it lies between pure adaptive search which strictly improves at each iteration and simulated annealing with constant temperature which allows backtracking or the acceptance of worse function values in this paper we build on an earlier work and make two contributions first we link hesitant adaptive search to standard counting process theory and second we use this to derive the exact distribution of the number of iterations of hesitant adaptive search to termination

set partition by globally optimized cluster seed points abstract a new method is proposed for solving set partition clustering problems the suggested approach is based on the application of global optimization methodology for selecting best cluster seed points the applicability of the method is illustrated by numerical examples

hit and run algorithms for the identification of nonredundant linear inequalities two probabilistic hit and run algorithms are presented to detect nonredundant constraints in a full dimensional system of linear inequalities the algorithms proceed by generating a random sequence of interior points whose limiting distribution is uniform and by searching for a nonredundant constraint in the direction of a random vector from each point in the sequence in the hypersphere directions algorithm the direction vector is drawn from a uniform distribution on a hypersphere in the computationally superior coordinate directions algorithm a search is carried out along one of the coordinate vectors the algorithms are terminated through the use of a bayesian stopping rule computational experience with the algorithms and the stopping rule will be reported

robust pole placement design a new technique is presented for the robust design of linear feedback controllers such that the system closed loop poles stay inside a prescribed region in the complex plane despite bounded plant parameter variations the method is based on the definition of a robust measure for any given controller and on the search by an iterative method for the controller which maximizes that measure

pure adaptive search in global optimization pure adaptive seach iteratively constructs a sequence of interior points uniformly distributed within the corresponding sequence of nested improving regions of the feasible space that is at any iteration the next point in the sequence is uniformly distributed over the region of feasible space containing all points that are strictly superior in value to the previous points in the sequence the complexity of this algorithm is measured by the expected number of iterations required to achieve a given accuracy of solution we show that for global mathematical programs satisfying the lipschitz condition its complexity increases at mostlinearly in the dimension of the problem

towards ideal multistart a stochastic approach for locating the minima of a continuous function inside a bounded domain a stochastic global optimization method based on multistart is presented in this the local search is conditionally applied with a probability that takes in account the topology of the objective function at the detail offered by the current status of exploration as a result the number of unnecessary local searches is drastically limited yielding an efficient method results of its application on a set of common test functions are reported along with a performance comparison against other established methods of similar nature

pure adaptive search for finite global optimization pure adaptive search is a stochastic algorithm which has been analyzed for continuous global optimization when a uniform distribution is used in pas it has been shown to have complexity which is linear in dimension we define strong and weak variations of pas in the setting of finite global optimization and prove analogous results in particular for then dimensional lattice 1 k n the expected number of iterations to find the global optimum is linear inn many discrete combinatorial optimization problems although having intractably large domains have quite small ranges the strong version of pas for all problems and the weak version of pas for a limited class of problems has complexity the order of the size of the range

a metropolis algorithm combined with hooke jeeves local search method applied to global optimization abstract a hybridization of a recently introduced metropolis algorithm named the particle collision algorithm pca and the hooke jeeves local search method is applied to a testbed of global optimization functions and to real world chemical equilibrium nonlinear systems the results obtained by this method called hjpca are compared against those achieved by two state of the art global optimization methods c grasp and global hjpca performs better than both algorithms thus demonstrating its potential for other applications

grover s quantum algorithm applied to global optimization grover s quantum computational search procedure can provide the basis for implementing adaptive global optimization algorithms a brief overview of the procedure is given and a framework called grover adaptive search is set up a method of durr and hoyer and one introduced by the authors fit into this framework and are compared

training ν support vector classifiers theory and algorithms the support vector machine svm for classification proposed by scholkopf smola williamson and bartlett 2000 has the advantage of using a parameter on controlling the number of support vectors in this article we investigate the relation between svm and c svm in detail we show that in general they are two different problems with the same optimal solution set hence we may expect that many numerical aspects of solving them are similar however compared to regular c svm the formulation of svm is more complicated so up to now there have been no effective methods for solving large scale svm we propose a decomposition method for svm that is competitive with existing methods for c svm we also discuss the behavior of svm by some numerical experiments

a direct stochastic algorithm for global search this paper presents a new algorithm called probabilistic global search lausanne pgsl pgsl is founded on the assumption that optimal solutions can be identified through focusing search around sets of good solutions tests on benchmark problems having multi parameter non linear objective functions revealed that pgsl performs better than genetic algorithms and advanced algorithms for simulated annealing in 19 out of 23 cases studied furthermore as problem sizes increase pgsl performs increasingly better than these other approaches empirical evidence of the convergence of pgsl is provided through its application to lennard jones cluster optimisation problem finally pgsl has already proved to be valuable for engineering tasks in areas of design diagnosis and control

recent developments and trends in global optimization many optimization problems in engineering and science require solutions that are globally optimal these optimization problems are characterized by the nonconvexity of the feasible domain or the objective function and may involve continuous and or discrete variables in this paper we highlight some recent results and discuss current research trends on deterministic and stochastic global optimization and global continuous approaches to discrete optimization

nested partitions method for global optimization we propose a new randomized method for solving global optimization problems this method the nested partitions np method systematically partitions the feasible region and concentrates the search in regions that are the most promising the most promising region is selected in each iteration based on information obtained from random sampling of the entire feasible region and local search the method hence combines global and local search we first develop the method for discrete problems and then show that the method can be extended to continuous global optimization the method is shown to converge with probability one to a global optimum in finite time in addition we provide bounds on the expected number of iterations required for convergence and we suggest two stopping criteria numerical examples are also presented to demonstrate the effectiveness of the method

a deterministic algorithm for global optimization we present an algorithm for finding the global maximum of a multimodal multivariate function for which derivatives are available the algorithm assumes a bound on the second derivatives of the function and uses this to construct an upper envelope successive function evaluations lower this envelope until the value of the global maximum is known to the required degree of accuracy the algorithm has been implemented in ratfor and execution times for standard test functions are presented at the end of the paper

convergence analysis and improvements of quantum behaved particle swarm optimization motivated by concepts in quantum mechanics and particle swarm optimization pso quantum behaved particle swarm optimization qpso was proposed as a variant of pso with better global search ability although it has been shown to perform well in finding optimal solutions for many optimization problems there has so far been little theoretical analysis on its convergence and performance this paper presents a convergence analysis and performance evaluation of the qpso algorithm and it also proposes two variants of the qpso algorithm first we investigate in detail the convergence of the qpso algorithm on a probabilistic metric space and prove that the qpso algorithm is a form of contraction mapping and can converge to the global optimum this is the first time that the theory of probabilistic metric spaces has been employed to analyze a stochastic optimization algorithm we provided a new definition for the convergence rate of a stochastic algorithm as well as definitions for three types of convergence according to the correlations between the convergence rate and the objective function values with these definitions the effectiveness of the qpso is evaluated by computing and analyzing the time complexity and the convergence rate of the algorithm then the qpso with random mean best position qpso rm and the qpso with ranking operator qpso ro are proposed as two improvements of the qpso algorithm finally some empirical studies on popular benchmark functions are performed in order to make a full performance evaluation and comparison between qpso qpso rm qpso ro and other variants of pso

a simulated annealing driven multi start algorithm for bound constrained global optimization a derivative free simulated annealing driven multi start algorithm for continuous global optimization is presented we first propose a trial point generation scheme in continuous simulated annealing which eliminates the need for the gradient based trial point generation we then suitably embed the multi start procedure within the simulated annealing algorithm we modify the derivative free pattern search method and use it as the local search in the multi start procedure we study the convergence properties of the algorithm and test its performance on a set of 50 problems numerical results are presented which show the robustness of the algorithm numerical comparisons with a gradient based simulated annealing algorithm and three population based global optimization algorithms show that the new algorithm could offer a reasonable alternative to many currently available global optimization algorithms specially for problems requiring direct search type algorithm

sparseness of support vector machines support vector machines svms construct decision functions that are linear combinations of kernel evaluations on the training set the samples with non vanishing coefficients are called support vectors in this work we establish lower asymptotical bounds on the number of support vectors on our way we prove several results which are of great importance for the understanding of svms in particular we describe to which limit svm decision functions tend discuss the corresponding notion of convergence and provide some results on the stability of svms using subdifferential calculus in the associated reproducing kernel hilbert space

on the optimal parameter choice for spl nu support vector machines we determine the asymptotically optimal choice of the parameter spl nu for classifiers of spl nu support vector machine spl nu svm type which has been introduced by scholkopf et al 2000 it turns out that spl nu should be a close upper estimate of twice the optimal bayes risk provided that the classifier uses a so called universal kernel such as the gaussian rbf kernel moreover several experiments show that this result can be used to implement some modified cross validation procedures which improve standard cross validation for spl nu svms

a statistical estimate of the structure of multi extremal problems the solution of ak extremal problem is defined as the set of pairs xi i i 1 k where xt isith local minimum and i is the volume of the set of attraction of this minimum a bayesian estimate ofk and 1 k is constructed

minimum energy point charge configurations on a circular disk stochastic global optimization methods are used together with problem specific heuristics to find configurations of n equal point charges on a circular disk such that the potential energy is minimized previous such optimal or conjecturally optimal configurations have been published only for and n 29 30 50 earlier results are examined and the best currently known configurations most of them new for are presented

a standard form variant and safeguarded linesearch for the modified karmarkar algorithm in his original analysis of the projective algorithm for linear programming karmarkar proposed a modified method which improved the worst case arithmetic complexity of the original algorithm by a factor of sqrt n karmarkar s analysis of the sqrt n improvement is based on a primal dual formulation and requires that small steps be taken on each iteration however in practice the original algorithm can be easily applied to standard form problems and is considerably improved by performing a linesearch on each iteration generally leading to much larger steps we show here that by incorporating a simple safeguard a linesearch may be performed in the modified algorithm while retaining the sqrt n complexity improvement over the original algorithm we then show that the modified algorithm with safeguarded linesearch can be applied directly to a standard form linear program with unknown optimal objective value the resulting algorithm enjoys a sqrt n complexity advantage over standard form variants of the original algorithm

applying electromagnetism like mechanism for feature selection in recent years methods of feature selection have been increasingly emphasized as venues for reducing cost and shortening the length of time required for computation in data mining this study utilizes electromagnetism like mechanism as a wrapper approach to feature selection birbil and fang proposed em in 2003 em uses the attraction repulsion mechanism of the electromagnetism theory to ascertain the optimal solution although em has been applied to the topic of optimization in continuous space and a small number of studies on discrete problems it has not been applied to the subject of feature selection in this study em combined with 1 nearest neighbor 1nn was applied for feature selection and classification this study utilized the total force exerted on a particle and evaluated this force to determine which features are to be selected the most crucial features were selected according to the proposed method based on the minimum miss classification rate which was attained through 1nn an unknown datum was classified by 1nn based on the chosen reduced model to estimate the effectiveness of the proposed method a numerical experiment was conducted using several data sets with diverse sizes features separability and classes experimental results indicated that the proposed method outperformed other well known algorithms in not only balanced classification accuracy but also efficiency of feature selection lastly this study used an actual case concerning gestational diabetes mellitus to demonstrate the workability of the proposed method

convergence guarantees for generalized adaptive stochastic search methods for continuous global optimization this paper presents some simple technical conditions that guarantee the convergence of a general class of adaptive stochastic global optimization algorithms by imposing some conditions on the probability distributions that generate the iterates these stochastic algorithms can be shown to converge to the global optimum in a probabilistic sense these results also apply to global optimization algorithms that combine local and global stochastic search strategies and also those algorithms that combine deterministic and stochastic search strategies this makes the results applicable to a wide range of global optimization algorithms that are useful in practice moreover this paper provides convergence conditions involving the conditional densities of the random vector iterates that are easy to verify in practice it also provides some convergence conditions in the special case when the iterates are generated by elliptical distributions such as the multivariate normal and cauchy distributions these results are then used to prove the convergence of some practical stochastic global optimization algorithms including an evolutionary programming algorithm in addition this paper introduces the notion of a stochastic algorithm being probabilistically dense in the domain of the function and shows that under simple assumptions this is equivalent to seeing any point in the domain with probability 1 this in turn is equivalent to almost sure convergence to the global minimum finally some simple results on convergence rates are also proved

stochastic global optimization methods part 1 clustering methods in this stochastic approach to global optimization clustering techniques are applied to identify local minima of a real valued objective function that are potentially global three different methods of this type are described their accuracy and efficiency are analyzed in detail

conjugate relation between loss functions and uncertainty sets in classification problems there are two main approaches to binary classification problems the loss function approach and the uncertainty set approach the loss function approach is widely used in real world data analysis statistical decision theory has been used to elucidate its properties such as statistical consistency conditional probabilities can also be estimated by using the minimum solution of the loss function in the uncertainty set approach an uncertainty set is defined for each binary label from training samples the best separating hyperplane between the two uncertainty sets is used as the decision function although the uncertainty set approach provides an intuitive understanding of learning algorithms its statistical properties have not been sufficiently studied in this paper we show that the uncertainty set is deeply connected with the convex conjugate of a loss function on the basis of the conjugate relation we propose a way of revising the uncertainty set approach so that it will have good statistical properties such as statistical consistency we also introduce statistical models corresponding to uncertainty sets in order to estimate conditional probabilities finally we present numerical experiments verifying that the learning with revised uncertainty sets improves the prediction accuracy

an optimal control approach to nonlinear mixed integer programming problems abstract motivated by the recent developments of the control parametrization enhancing technique cpet a novel method for solving a general class of nonlinear mixed integer programming problems is introduced in this paper by imposing appropriate dynamics as well as a set of statistical variance type of functional constraints a problem with mixed integer decision variables is first transformed into a discrete valued optimal control problem and then transformed by applying cpet into a standard optimization problem involving only continuous values

a short cut potential reduction algorithm for linear programming as most interior point algorithms iterate they repeatedly perform costly matrix operations such as projections on the entire constraint matrix for large scale linear programming problems such operations consume the great majority of the computation time required however for problems where the number of variables far exceeds the number of constraints operations over the entire constraint matrix are unnecessary we will examine and extend decomposition techniques which greatly reduce the amount of work required by such interior point methods as the dual affine scaling and the dual potential reduction algorithms in an effort to judge the practical viability of the decompositioning we compare the performance of the dual potential reduction algorithm with and without decompositioning over a set of randomly generated transportation problems accompanying a theoretical justification of these techniques we focus on the implementation details and computational results of one such technique

the application of a unified bayesian stopping criterion in competing parallel algorithms for global optimization abstract the unconstrained global programming problem is addressed using a multistart multialgorithminfrastructure in which different algorithms compete in parallel for a contribution towards a single global stopping criterion denoted the unified bayesian global stopping criterion the use of different algorithms is motivated by the observation that no single global optimization algorithm consistently outperforms all other algorithms when applied to large sets of problems from different classes the bayesian stopping criterion is based on the single assumption that the probability of each algorithm converging to the global optimum is at least as large as the probability of convergence to any other local minimum this assumption is often valid in the case of practical problems of physical origin e g determining physical configurations corresponding to minimum potential energy results for parallel clusters of up to 128 machines are presented

stochastic global optimization methods part 11 multi level methods in part ii of our paper two stochastic methods for global optimization are described that with probability 1 find all relevant local minima of the objective function with the smallest possible number of local searches the computational performance of these methods is examined both analytically and empirically

hesitant adaptive search for global optimisation pure adaptive search is a stochastic algorithm which has been analysed in distinct ways for finite and continuous global optimisation in this paper motivated by the behaviour of practical algorithms such as simulated annealing we extend these ideas we present a unified theory which yields both the finite and continuous results for pure adaptive search at the same time we allow our extended algorithm to hesitate before improvement continues results are obtained for the expected number of iterations to convergence for such an algorithm 1998 the mathematical programming society inc published by elsevier science b v

new support vector algorithms we propose a new class of support vector algorithms for regression and classification in these algorithms a parameter lets one effectively control the number of support vectors while this can be useful in its own right the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm the accuracy parameter epsilon in the regression case and the regularization constant c in the classification case we describe the algorithms give some theoretical results concerning the meaning and the choice of and report experimental results

an algorithm for linear programming which requires o m n n 2 m n 1 5 n l arithmetic operations we present an algorithm for linear programming which requires o m n n 2 m n 1 5 n l arithmetic operations wherem is the number of constraints andn is the number of variables each operation is performed to a precision of o l bits l is bounded by the number of bits in the input the worst case running time of the algorithm is better than that of karmarkar s algorithm by a factor of sqrt m n

the nonlinear geometry of linear programming ii legendre transform coordinates and central trajectories karmarkar s projective scaling algorithm for solving linear programming problems associates to each objective function a vector field defined in the interior of the polytope of feasible solutions of the problem this paper studies the set of trajectories obtained by integrating this vector field called p trajectories as well as a related set of trajectories called a trajectories the a trajectories arise from another linear programming algorithm the affine scaling algorithm the affine and projective scaling vector fields are each defined for linear programs of a special form called standard form and canonical form respectively these trajectories are studied using a nonlinear change of variables called legendre transform coordinates which is a projection of the gradient of a logarithmic barrier function the legendre transform coordinate mapping is given by rational functions and its inverse mapping is algebraic it depends only on the constraints of the linear program and is a one to one mapping for canonical form linear programs when the polytope of feasible solutions is bounded there is a unique point mapping to zero called the center the a trajectories of standard form linear programs are linearized by the legendre transform coordinate mapping when the polytope of feasible solutions is bounded they are the complete set of geodesics of a riemannian geometry isometric to euclidean geometry each a trajectory is part of a real algebraic curve each p trajectory for a canonical form linear program lies in a plane in legendre transform coordinates the p trajectory through 0 in legendre transform coordinates called the central p trajectory is part of a straight line and is contained in the a trajectory through 0 called the central a trajectory each p trajectory is part of a real algebraic curve the central a trajectory is the locus of centers of a family of linear programs obtained by adding an extra equality constraint of the form c x u it is also the set of minima of a parametrized family of logarithmic barrier functions power series expansions are derived for the central a trajectory which is also the central p trajectory these power series have a simple recursive form and are useful in developing higher order analogues of karmarkar s algorithm a trajectories are defined for a general linear program using this definition it is shown that the limit point x 0 of a central a trajectory on the boundary of the feasible solution polytope p is the center of the unique face of p containing x 0 in its relative interior received by the editors october 8 1986 and in revised form june 9 1987 and march 25 1988 1980 mathematics subject classification 1985 revision primary 90c05 secondary 52a40 34a34 the first author was partially supported by onr contract n00014 87 k0214 1989 american mathematical society 0002 9947 89 1 00 25 per page

a stochastic method for global optimization a stochastic method for global optimization is described and evaluated the method involves a combination of sampling clustering and local search and terminates with a range of confidence intervals on the value of the global optimum computational results on standard test functions are included as well

some old traditions in mathematics and in mathematical education abstract continuing the discussion by azbelev about the harmful influence of some classical traditions on the study of differential equations 1 this note presents further instances of old traditions and of their impact on the development of certain areas of mathematics and its applications

consistency of support vector machines and other regularized kernel classifiers it is shown that various classifiers that are based on minimization of a regularized risk are universally consistent i e they can asymptotically learn in every classification task the role of the loss functions used in these algorithms is considered in detail as an application of our general framework several types of support vector machines svms as well as regularization networks are treated our methods combine techniques from stochastics approximation theory and functional analysis

application of the multilevel single linkage method to one dimensional electromagnetic inverse scattering problem an inverse scattering method for the reconstruction of the permittivity and conductivity profiles of a multilayered medium and for that of the impedance profile of a nonuniform transmission line is proposed the inversion is based on the global minimization of an objective function by the multilevel single linkage method the objective function is defined as the mean square error between the measured data and the data obtained from the solution of the forward problem an exact formulation for the gradient of the objective function in closed form is derived the necessary condition for the unique solution of the inverse problem of a nonuniform transmission line is discussed reconstruction examples using both experimental and noisy synthetic data are presented

pure adaptive search in monte carlo optimization pure adaptive search constructs a sequence of points uniformly distributed within a corresponding sequence of nested regions of the feasible space at any stage the next point in the sequence is chosen uniformly distributed over the region of feasible space containing all points that are equal or superior in value to the previous points in the sequence we show that for convex programs the number of iterations required to achieve a given accuracy of solution increases at most linearly in the dimension of the problem this compares to exponential growth in iterations required for pure random search

on the complexity of a class of projective interior point methods we present a generic projective interior point algorithm for linear programming which includes modified versions of karmarkar s original algorithm and most other projective algorithms that have been proposed as special cases we show that this class of algorithms has a worst case iteration bound of o nl and is closely related to the class of potential reduction interior point methods

integral global optimization method for differential games with application to pursuit evasion games abstract a variation free method for global solution of differential games is proposed the method is based on the integral global optimization algorithm and it does not employ variation based techniques nor the notion of convexity the existence of a saddle point is not assumed a priori instead the method delivers the global minimax and maximin solutions for both players a necessary and sufficient global minimax condition is presented an application is made to the solution of nonlinear pursuit evasion games the ideas are illustrated by examples

global optimization of statistical functions with simulated annealing abstract many statistical methods rely on numerical optimization to estimate a model s parameters unfortunately conventional algorithms sometimes fail even when they do converge there is no assurance that they have found the global rather than a local optimum we test a new optimization algorithm simulated annealing on four econometric problems and compare it to three common conventional algorithms not only can simulated annealing find the global optimum it is also less likely to fail on difficult functions because it is a very robust algorithm the promise of simulated annealing is demonstrated on the four econometric problems

on the mixed integer signomial programming problems this paper proposes an approximate method to solve the mixed integer signomial programming problem for which the objective function and the constraints may contain product terms with exponents and decision variables which could be continuous or integral a linear programming relaxation is derived for the problem based on piecewise linearization techniques which first convert a signomial term into the sum of absolute terms these absolute terms are then linearized by linearization strategies in addition a novel approach is included for solving integer and undefined problems in the logarithmic piecewise technique which leads to more usefulness of the proposed method the proposed method could reach a solution as close as possible to the global optimum

an efficient algorithm for large scale global optimization of continuous functions a fast descent algorithm resorting to a stretching function technique and built on one hybrid method grsa which combines simulated annealing sa algorithm and gradient based methods for large scale global optimizations is proposed unlike the previously proposed method in which the original objective functions remain unchanged during the whole course of optimization the new method firstly constructs an auxiliary function on one local minimizer obtained by gradient based methods and then sa is executed on this constructed auxiliary function instead of on the original objective function in order that we can improve the jumping ability of sa algorithm to escape from the currently discovered local minimum to a better one from which the gradient based methods restart a new local search the above procedure is repeated until a global minimum is detected in addition corresponding to the adopted stretching technique a new next trial point generating scheme is designed it is verified by simulation especially on large scale problems that the convergence speed is greatly accelerated which is its main difference from many other reported methods that mostly cope with functions with less than 50 variables and does not apply to large scale optimization problems furthermore the new algorithm functions as a global optimization procedure with a high success probability and high solution precision

efficient monte carlo procedures for generating points uniformly distributed over bounded regions we consider the monte carlo problem of generating points uniformly distributed within an arbitrary bounded measurable region the class of markovian methods considered generate points asymptotically uniformly distributed within the region computational experience suggests the methods are potentially superior to conventional rejection techniques for large dimensional regions

learning multicriteria classification models from examples decision rules in continuous space abstract the classification problem statement of multicriteria decision analysis is to model the classification of the alternatives actions according to the decision maker s preferences these models are based on outranking relations utility functions or linear discriminant functions model parameters can be given explicitly or learnt from a preclassified set of alternatives actions in this paper we propose a novel approach the continuous decision cd method to learn parameters of a discriminant function and we also introduce its extension the continuous decision tree cdt method which describes the classification more accurately the proposed methods are results of integration of machine learning methods in decision analysis from a machine learning point of view the cdt method can be considered as an extension of the c4 5 decision tree building algorithm that handles only numeric criteria but applies more complex tests in the inner nodes of the tree for the sake of easier interpretation the decision trees are transformed to rules

a polynomial method of approximate centers for linear programming we present a path following algorithm for the linear programming problem with a surprisingly simple and elegant proof of its polynomial behaviour this is done both for the problem in standard form and for its dual problem we also discuss some implementation strategies

convergence qualification of adaptive partition algorithms in global optimization following the presentation of a general partition algorithm scheme for seeking the globally best solution in multiextremal optimization problems necessary and sufficient convergence conditions are formulated in terms of respectively implied or postulated properties of the partition operator the convergence results obtained are pertinent to a number of deterministic algorithms in global optimization permitting their diverse modifications and generalizations

using modifications to grover s search algorithm for quantum global optimization we study the problem of finding a global optimal solution to discrete optimization problems using a heuristic based on quantum computing methods knowledge of quantum computing ideas is not necessary to read this paper we focus on a successful quantum computing method introduced by baritompa bulger and wood that we refer to as the bbw algorithm and develop two modifications first we modify the bbw algorithm to achieve a dramatic speedup that lets us extend the known bbw static schedule from 33 to 43 points thereby increasing its applicability we further modify it by converting it from a static method to a dynamic one experimental results show the value of this latter modification

improved global convergence probability using multiple independent optimizations for some problems global optimization algorithms may have a significant probability of not converging to the global optimum or require an extremely large number of function evaluations to reach it for such problems the probability of finding the global optimum may be improved by performing multiple independent short searches rather than using the entire available budget of function evaluations on a single long search the main difficulty in adopting such a strategy is to decide how many searches to carry out for a given function evaluation budget the basic premise of this paper is that different searches may have substantially different outcomes but they all start with rapid initial improvement of the objective function followed by much slower progress later on furthermore we assume that the number of function evaluations to the end of the initial stage of rapid progress does not change drastically from one search to another for a given problem and algorithmic setting therefore we propose that the number of function evaluations required for this rapid progress stage be estimated with one or two runs and then the same number of function evaluations be allocated to all subsequent searches we show that these assumptions work well for the particle swarm optimization algorithm applied to a set of difficult analytical test problems with known global solutions for these problems we show that the proposed strategy can substantially improve the probability of obtaining the global optimum for a given budget of function evaluations we also test a bayesian criterion for estimating the probability of having reached the global optimum at the end of the series of searches and find that it can provide a conservative estimate for most problems finally we demonstrate the approach on a particularly challenging engineering design problem constructed so as to have at least 32 widely separated local optima copyright 2006 john wiley sons ltd

comparative analysis of affine scaling algorithms based on simplifying assumptions we analyze several affine potential reduction algorithms for linear programming based on simplifying assumptions we show that under a strong probabilistic assumption regarding the distribution of the data in an iteration the decrease in the primal potential function will be omega rho sqrt log n with high probability compared to the guaranteed 1 2n is a parameter in the potential function andn is the number of variables under the same assumption we further show that the objective reduction rate of dikin s affine scaling algorithm is 1 1 sqrt log n with high probability compared to no guaranteed convergence rate

the globally convexized filled functions for global optimization a new filled function the globally convexized filled function is proposed the value of this function tends to positive infinity as the norm of the variable vector tends to infinity and it has no stationary point in the region in which the objective function value is greater than or equal to the least value found so far but it does have a minimizer not only along the radius direction from the minimizer with the least objective function value found so far like filled function found earlier by ge in the region in which the objective function value is less than the least value found so far but with a prefixed minimizer how to adjust parameters in the filled function is considered such a filled function can be appropriately combined with stochastic stopping rules proposed by boender rinnooy kan and timmer the computational results show that this algorithm is quite efficient and reliable

numerical methods in control we study classical control problems like pole assignment stabilization linear quadratic control and h control from a numerical analysis point of view we present several examples that show the difficulties with classical approaches and suggest reformulations of the problems in a more general framework we also discuss some new algorithmic approaches

newtonian clustering an approach based on molecular dynamics and global optimization given a data set a dynamical procedure is applied to the data points in order to shrink and separate possibly overlapping clusters namely newton s equations of motion are employed to concentrate the data points around their cluster centers using an attractive potential constructed specially for this purpose during this process important information is gathered concerning the spread of each cluster in succession this information is used to create an objective function that maps each cluster to a local maximum global optimization is then used to retrieve the positions of the maxima that correspond to the locations of the cluster centers further refinement is achieved by applying the em algorithm to a gaussian mixture model whose construction and initialization is based on the acquired information to assess the effectiveness of our method we have conducted experiments on a plethora of benchmark data sets in addition we have compared its performance against four clustering techniques that are well established in the literature

robust pole assignment abstract this paper presents new theorems on the theory of interval matrix inequalities and the theory of polynomials with interval roots and applies them to the problem of robust pole placement we formulate optimization problems and derive convergent iterative algorithms which allow the designer to find controllers that place closed loop poles within desired intervals for plants with unknown but bounded parameter uncertainties the algorithms are computationally reasonable and provide a useful addition to currently existing control cad tools

an o n 3 l primal dual potential reduction algorithm for solving convex quadratic programs in this paper we combine partial updating and an adaptation of anstreicher s safeguarded linesearch of the primal dual potential function with kojima mizuno and yoshise s potential reduction algorithm for the linear complementarity problem to obtain an o n3l algorithm for convex quadratic programming our modified algorithm is a long step method that requires at most o sqrt n l steps

adaptive search in quasi monte carlo optimization motivated by a linear time complexity result for an adaptive monte carlo algorithm we propose and analyze an adaptive deterministic algorithm we restrict a grid search to nested subregions that promise to provide improvement of the current solution and we obtain an exponential rate of convergence in function evaluations for proving the main result we restrict ourselves to functions on hypercubes in a final section we outline how to extend the method to the general case and give some numerical examples

regularization path for nu support vector classification the v support vector classification v svc proposed by scholkopf has the advantage of using a regularization parameter v for controlling the number of support vectors and margin errors however compared to c svc its formulation is more complicated and to date there are no effective methods for computing its regularization path in this paper we propose a new regularization path algorithm which is designed on the basis of a modified formulation of v svc and traces the solution path with respect to the parameter v through theoretical analysis and confirmatory experiments we show that our algorithm can avoid the infeasible updating path under several assumptions i e assumptions 1 and 2 and fit the entire solution path in a finite number of steps when the regularization path of v svc is available a novel approach proposed by yang and ong can be applied to obtain the global optimal solution of common validation functions for v svc and the computation for the whole process is minimal numerical experiments show that it is more efficient than various kinds of grid search methods for selecting the optimal regularization parameter v

random linkage a family of acceptance rejection algorithms for global optimisation this paper introduces a new infinite class of stochastic algorithms for the optimization of multimodal functions defined over a compact set the main idea is that of generalizing some well known methods inspired by clustering techniques which make use of local search routines started at selected points in a random sample it is shown that there exist interesting classes of global optimization algorithms all possessing the following theoretical properties convergence to the global optimum with probability 1 detection of the global optimum in a finite number of iterations w p 1 probability of starting a local search decreasing to 0 total number of local searches performed even if the algorithm is run forever finite w p 1

on accelerated random search a new variant of pure random search prs for function optimization is introduced the basic finite descent accelerated random search ars algorithm is simple the search is confined to shrinking neighborhoods of a previous record generating value with the search neighborhood reinitialized to the entire space when a new record is found local maxima are avoided by including an automatic restart feature which reinitializes the search neighborhood after some number of shrink steps have been performed one goal of this article is to provide rigorous mathematical comparisons of ars to prs it is shown that the sequence produced by the ars process converges with probability one to the maximum of a continuous objective function faster than that of the prs process by adjustably large multiples of the time step theorem regarding an infinite descent no automatic restart version of ars it is shown that if the objective function satisfies a local nonflatness condition then the right tails of the distributions of inter record times are exponentially smaller than those of prs theorem performance comparisons between ars prs and three quasi newton type optimization routines are reported in attempting to find extrema of i each of a small collection of standard test functions of two variables and ii d dimensional polynomials with random roots also reported is a three way performance comparison between ars prs and a simulated annealing algorithm in attempting to solve traveling salesman problems

a potential reduction algorithm with user specified phase i phase ii balance for solving a linear program from an infeasible warm start this paper develops a potential reduction algorithm for solving a linear programming problem directly from a warm start initial point that is neither feasible nor optimal the algorithm is an interior point variety that seeks to reduce a single potential function which simultaneously coerces feasibility improvement phase i and objective value improvement phase ii the key feature of the algorithm is the ability to specify beforehand the desired balance between infeasibility and nonoptimality in the following sense given a prespecified balancing parameter beta 0 the algorithm maintains the following phase i phase ii beta balancing constraint throughout left c t x z right beta xi t x where c t x is the objective function z is the unknown optimal objective value of the linear program and xi t x measures the infeasibility of the current iterate x this balancing constraint can be used to either emphasize rapid attainment of feasibility set beta large

technical note the asymptotic extreme value distribution of the sample minimum of a concave function under linear constraints we show that the minimum value of a sample of feasible points uniformly distributed over a linear constraint set is for concave functions asymptotically weibull distributed with shape parameter equal to the dimension of the feasible region

minimization by random search techniques we give two general convergence proofs for random search algorithms we review the literature and show how our results extend those available for specific variants of the conceptual algorithm studied here we then exploit the convergence results to examine convergence rates and to actually design implementable methods finally we report on some computational experience

generating random vectors uniformly distributed inside and on the surface of different regions abstract an acceptance rejection algorithm for generating random vectors uniformly distributed over inside or on the surface of a complex region inserted in a minimal multidimensional rectangle is considered for regions having simple forms simplex hypersphere hyperellipsoid several algorithms are presented as well

polynomial time algorithms for linear programming based only on primal scaling and projected gradients of a potential function this paper presents extensions and further analytical properties of algorithms for linear programming based only on primal scaling and projected gradients of a potential function the paper contains extensions and analysis of two polynomial time algorithms for linear programming we first present an extension of gonzaga s o nl iteration algorithm that computes dual variables and does not assume a known optimal objective function value this algorithm uses only affine scaling and is based on computing the projected gradient of the potential function q ln x t s sum limits j 1 n ln x j wherex is the vector of primal variables ands is the vector of dual slack variables and q n sqrt n the algorithm takes either a primal step or recomputes dual variables at each iteration we next present an alternate form of ye s o sqrt n l iteration algorithm that is an extension of the first algorithm of the paper but uses the potential function q ln x t s sum limits j 1 n ln x j sum limits j 1 n ln s j where q n sqrt n we use this alternate form of ye s algorithm to show that ye s algorithm is optimal with respect to the choice of the parameterq in the following sense suppose thatq n n t wheret 0 then the algorithm will solve the linear program in o n r l iterations wherer max t 1 t thus the value oft that minimizes the complexity bound ist 1 2 yielding ye s o sqrt n l iteration bound

adaptive lqg control of input output systems a cost biased approach in this paper we consider linear systems in input output form and introduce a new adaptive linear quadratic gaussian lqg control scheme which is shown to be self optimizing the identification algorithm incorporates a cost biasing term which favors the parameters with smaller lqg optimal cost and a second term that aims at moderating the time variability of the estimate the corresponding closed loop scheme is proven to be stable and to achieve an asymptotic lqg cost equal to the one obtained under complete knowledge of the true system self optimization the results of this paper extend in a nontrivial way previous results established along the cost biased approach in other settings

direction choice for accelerated convergence in hit and run sampling hit and run algorithms are monte carlo procedures for generating points that are asymptotically distributed according to general absolutely continuous target distributions g over open bounded regions s applications include nonredundant constraint identification global optimization and monte carlo integration these algorithms are reversible random walks that commonly incorporate uniformly distributed step directions we investigate nonuniform direction choice and show that under regularity conditions on the region s and target distribution g there exists a unique direction choice distribution characterized by necessary and sufficient conditions depending on s and g which optimizes the doob bound on rate of convergence we include computational results demonstrating greatly accelerated convergence for this optimizing direction choice as well as for more easily implemented adaptive heuristic rules

mean shift mode seeking and clustering mean shift a simple interactive procedure that shifts each data point to the average of data points in its neighborhood is generalized and analyzed in the paper this generalization makes some k means like clustering algorithms its special cases it is shown that mean shift is a mode seeking process on the surface constructed with a shadow kernal for gaussian kernels mean shift is a gradient mapping convergence is studied for mean shift iterations cluster analysis if treated as a deterministic problem of finding a fixed point of mean shift that characterizes the data applications in clustering and hough transform are demonstrated mean shift is also considered as an evolutionary strategy that performs multistart global optimization

estimation of the minimum of a function using order statistics abstract suppose a is a bounded set in d and f a real function defined on a suppose m min f x x a exists using a random sample x 1 x 2 x n from a uniform distribution over a we construct a confidence interval for m using asymptotic theory our results contain some statistical results valid in general extreme value theory estimation of the main parameter of extreme value distributions

polynomial affine algorithms for linear programming the method of steepest descent with scaling affine scaling applied to the potential functionq logc x i 1n logxi solves the linear programming problem in polynomial time forq n ifq n then the algorithm terminates in no more than o n2l iterations if q n sqrt n withq o n then it takes no more than o nl iterations a modified algorithm using rank 1 updates for matrix inversions achieves respectively o n4l and o n3 5l arithmetic computions

on the influence of the kernel on the consistency of support vector machines in this article we study the generalization abilities of several classifiers of support vector machine svm type using a certain class of kernels that we call universal it is shown that the soft margin algorithms with universal kernels are consistent for a large class of classification problems including some kind of noisy tasks provided that the regularization parameter is chosen well in particular we derive a simple sufficient condition for this parameter in the case of gaussian rbf kernels on the one hand our considerations are based on an investigation of an approximation property the so called universality of the used kernels that ensures that all continuous functions can be approximated by certain kernel expressions this approximation property also gives a new insight into the role of kernels in these and other algorithms on the other hand the results are achieved by a precise study of the underlying optimization problems of the classifiers furthermore we show consistency for the maximal margin classifier as well as for the soft margin svm s in the presence of large margins in this case it turns out that also constant regularization parameters ensure consistency for the soft margin svm s finally we prove that even for simple noise free classification problems svm s with polynomial kernels can behave arbitrarily badly