# Application of the Parabola Method in Nonconvex Optimization
## 4 Numerical Comparison of Zero-Order Methods: Multivariate Optimization Problems  
We tested and compared the following zero-order methods: the Hooke and Jeeves method combined with the ATSA, the Rosenbrock method with a discrete step in the search procedure, the Rosenbrock method combined with the ATSA, and the coordinate descent method combined with the ATSA. In the current section, we give a brief description of these methods. Many of them are described in monographs and review articles on optimization methods [10; 11; 15; 16; 17; 18; 19]. Hooke and Jeeves methodThe pattern search method of Hooke and Jeeves consists of a sequence of exploratory moves about a base point that, if successful, are followed by pattern moves. The exploratory moves acquire information about the function \(f(x)\) in the neighborhood of the current base point \(b^{k}=(x_{1}^{k},\ldots,x_{n}^{k})\). Each variable \(x_{i}^{k}\), in turn, is given an increment \(\varepsilon_{i}\) (first, in the positive direction and then, if necessary, in the negative direction), and a check is made of the new function value. If any move results in a reduced function value, the new value of that variable will be retained. After all the variables have been considered, a new base point \(b^{k+1}\) will be reached. If \(b^{k+1}=b^{k}\), no function reduction has been achieved. The step length \(\varepsilon_{i}\) is reduced, and the procedure is repeated. If \(b^{k+1}\
eq b^{k}\), a pattern move from \(b^{k}\) is made. A pattern move attempts to speed up the search by using the information already acquired about \(f(x)\) to identify the best search direction. A move is made from \(b^{k+1}\) in the direction \(d=b^{k+1}-b^{k}\), since a move in this direction leads to a decrease in the function value. In this step, we use the ATSA to solve a univariate optimization problem \(f(\lambda)=\min_{\lambda}f(b^{k+1}+\lambda d)\) and obtain a new point \(p^{k}=b^{k+1}+\lambda^{*}d\). The search continues with a new sequence of exploratory moves about \(p_{k}\). If the lowest function value obtained is less than \(f(b^{k})\), then a new base point \(b^{k+2}\) has been reached. In this case, a second pattern move is made. If not, the pattern move from \(b^{k+1}\) is abandoned, and we continue with a new sequence of exploratory moves about \(b^{k+1}\). Rosenbrock methodThe main idea of the method is to iteratively find the descending direction of a function along linearly independent and orthogonal directions. A successful step in the current direction leads to an increase in this step on the following iteration by means of a stretch coefficient \(\rho>0\); otherwise, the coefficient \(0<\rho<1\) is used to decrease the step. The search within the current direction system is implemented until all possibilities of function reduction are exhausted. If there are no successful directions, a new set of linearly independent and orthogonal directions is constructed by means of rotating the previous ones in an appropriate manner. To obtain a new direction system, the Gram-Schmidt procedure is used. Iterative coordinate descent methodThis method is a simplified version of the Hooke and Jeeves method. It uses only the analogue of exploratory moves with an appropriate step size obtained from the line search procedure along the current basis vector. Combined versionsWe designed three modifications of the described methods to estimate the potential for finding the global optimum. The iterative coordinate descent method, the Hooke and Jeeves method, and the Rosenbrock method are combined with the ATSA and presented in this section. The following test problems were used to perform numerical experiments. 1. _Branin function_ \[f(x)=\left(x_{2}-\frac{5.1x_{1}^{2}}{4\pi^{2}}+\frac{5x_{1}}{\pi}-6\right)^{2}+10 \bigg{(}1-\frac{1}{8\pi}\bigg{)}\cos x_{1}+10,\] \[X=\Big{\{}x\in R^{2}:\;-5\leq x_{1}\leq 10,\;0\leq x_{2}\leq 15,\Big{\}}.\] Global optimum: \(x^{\star}=(3.141593,2.275)\), \(x^{\star\star}=(-3.141593,12.275)\), \(f^{\star}=0.397667\). 2.