Repository: OpenSourceEconomics/estimagic

Issue:
Add selected nlopt optimizers
### Current situation

Nlopt is not supported

### Desired Situation

Add local derivative free and derivative based optimizers from nlopt. 



Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics) Report
> Merging [#212](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics) (ee920be) into [master](https://codecov.io/gh/OpenSourceEconomics/estimagic/commit/6ff45a388b6731cc9d78dcf509243eeccb917495?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics) (6ff45a3) will **decrease** coverage by `2.62%`.
> The diff coverage is `36.66%`.

[![Impacted file tree graph](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/graphs/tree.svg?width=650&height=150&src=pr&token=INbcX5rQK2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics)](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics)

```diff
@@            Coverage Diff             @@
##           master     #212      +/-   ##
==========================================
- Coverage   83.54%   80.91%   -2.63%     
==========================================
  Files          58       59       +1     
  Lines        3992     4050      +58     
==========================================
- Hits         3335     3277      -58     
- Misses        657      773     +116     
```


| [Impacted Files](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics) | Coverage Δ | |
|---|---|---|
| [estimagic/optimization/nlopt\_optimizers.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL29wdGltaXphdGlvbi9ubG9wdF9vcHRpbWl6ZXJzLnB5) | `28.00% <28.00%> (ø)` | |
| [estimagic/optimization/\_\_init\_\_.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL29wdGltaXphdGlvbi9fX2luaXRfXy5weQ==) | `86.36% <75.00%> (-2.53%)` | :arrow_down: |
| [estimagic/config.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL2NvbmZpZy5weQ==) | `85.71% <80.00%> (-0.65%)` | :arrow_down: |
| [estimagic/visualization/estimation\_table.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL3Zpc3VhbGl6YXRpb24vZXN0aW1hdGlvbl90YWJsZS5weQ==) | `60.76% <100.00%> (-0.16%)` | :arrow_down: |
| [estimagic/optimization/scipy\_optimizers.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL29wdGltaXphdGlvbi9zY2lweV9vcHRpbWl6ZXJzLnB5) | `37.70% <0.00%> (-61.48%)` | :arrow_down: |
| [...imagic/optimization/internal\_criterion\_template.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL29wdGltaXphdGlvbi9pbnRlcm5hbF9jcml0ZXJpb25fdGVtcGxhdGUucHk=) | `72.33% <0.00%> (-0.98%)` | :arrow_down: |
| [estimagic/optimization/optimize.py](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics#diff-ZXN0aW1hZ2ljL29wdGltaXphdGlvbi9vcHRpbWl6ZS5weQ==) | `78.21% <0.00%> (-0.56%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics). Last update [6ff45a3...ee920be](https://codecov.io/gh/OpenSourceEconomics/estimagic/pull/212?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenSourceEconomics).


Patch:
diff --git a/estimagic/optimization/__init__.py b/estimagic/optimization/__init__.py
index 543eb5a..71541b2 100644
--- a/estimagic/optimization/__init__.py
+++ b/estimagic/optimization/__init__.py
@@ -44,3 +44,5 @@ if IS_DFOLS_INSTALLED:
 
 if IS_CYIPOPT_INSTALLED:
     AVAILABLE_ALGORITHMS["ipopt"] = cyipopt_optimizers.ipopt
+
+GLOBAL_ALGORITHMS = ["nlopt_direct", "nlopt_esch", "nlopt_isres", "nlopt_crs2_lm"]
diff --git a/estimagic/optimization/algo_options.py b/estimagic/optimization/algo_options.py
index 38d8857..6ff1ddc 100644
--- a/estimagic/optimization/algo_options.py
+++ b/estimagic/optimization/algo_options.py
@@ -100,6 +100,17 @@ STOPPING_MAX_CRITERION_EVALUATIONS = 1_000_000
 
 """
 
+
+STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL = 5_000
+"""int:
+    If the maximum number of function evaluation is reached, the optimization stops
+    but we do not count this as successful convergence. The function evaluations used
+    to evaluate a numerical gradient do not count for this. Set to a lower number than
+    STOPPING_MAX_CRITERION_EVALUATIONS for global optimizers.
+
+"""
+
+
 STOPPING_MAX_ITERATIONS = 1_000_000
 """int:
     If the maximum number of iterations is reached, the
diff --git a/estimagic/optimization/nlopt_optimizers.py b/estimagic/optimization/nlopt_optimizers.py
index 179776b..b6981c5 100644
--- a/estimagic/optimization/nlopt_optimizers.py
+++ b/estimagic/optimization/nlopt_optimizers.py
@@ -1,5 +1,8 @@
-import warnings
+"""Implement `nlopt` algorithms.
 
+The documentation is heavily based on (nlopt documentation)[nlopt.readthedocs.io].
+
+"""
 import numpy as np
 
 from estimagic.config import IS_NLOPT_INSTALLED
@@ -8,6 +11,10 @@ from estimagic.optimization.algo_options import CONVERGENCE_ABSOLUTE_PARAMS_TOLE
 from estimagic.optimization.algo_options import CONVERGENCE_RELATIVE_CRITERION_TOLERANCE
 from estimagic.optimization.algo_options import CONVERGENCE_RELATIVE_PARAMS_TOLERANCE
 from estimagic.optimization.algo_options import STOPPING_MAX_CRITERION_EVALUATIONS
+from estimagic.optimization.algo_options import (
+    STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL,
+)
+
 
 if IS_NLOPT_INSTALLED:
     import nlopt
@@ -37,19 +44,29 @@ def nlopt_bobyqa(
 
     The implementation is derived from the BOBYQA subroutine of M. J. D. Powell.
 
-    ADD REMAINING EXPLANATIONS FROM NLOPT DOCUMENTATION HERE.
+    The algorithm performs derivative free bound-constrained optimization using
+    an iteratively constructed quadratic approximation for the objective function.
+    Due to its use of quadratic appoximation, the algorithm may perform poorly
+    for objective functions that are not twice-differentiable.
+
+    For details see:
+    M. J. D. Powell, "The BOBYQA algorithm for bound constrained optimization
+    without derivatives," Department of Applied Mathematics and Theoretical
+    Physics, Cambridge England, technical report NA2009/06 (2009).
 
     ``nlopt_bobyqa`` supports the following ``algo_options``:
 
     - convergence.relative_params_tolerance (float):  Stop when the relative movement
       between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
     - convergence.relative_criterion_tolerance (float): Stop when the relative
       improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
     - stopping.max_criterion_evaluations (int): If the maximum number of function
       evaluation is reached, the optimization stops but we do not count this
       as convergence.
-    - stopping_max_iterations (int): If the maximum number of iterations is reached,
-      the optimization stops, but we do not count this as convergence.
 
     """
     out = _minimize_nlopt(
@@ -83,11 +100,6 @@ def nlopt_neldermead(
 ):
     """Minimize a scalar function using the Nelder-Mead simplex algorithm.
 
-    Do not call this function directly but pass its name "nlopt_bobyqa" to
-    estimagic's maximize or minimize function as `algorithm` argument. Specify
-    your desired arguments as a dictionary and pass them as `algo_options` to
-    minimize or maximize.
-
     The basic algorithm is described in:
     J. A. Nelder and R. Mead, "A simplex method for function minimization,"
     The Computer Journal 7, p. 308-313 (1965).
@@ -96,33 +108,24 @@ def nlopt_neldermead(
     that the nlopt version supports bounds. This is done by moving all new points that
     would lie outside the bounds exactly on the bounds.
 
-    Below, only details of the optional algorithm options are listed. For the mandatory
-    arguments see :ref:`internal_optimizer_interface`. For more background on those
-    options, see :ref:`naming_conventions`.
-
-    Args:
-        convergence_relative_params_tolerance (float): Stop when the relative movement
-            between parameter vectors is smaller than this.
-        convergence_relative_criterion_tolerance (float): Stop when the relative
-            improvement between two iterations is smaller than this.
-            In contrast to other algorithms the relative criterion tolerance is set
-            to zero by default because setting it to any non-zero value made the
-            algorithm stop too early even on the most simple test functions.
-        stopping_max_criterion_evaluations (int): If the maximum number of function
-            evaluation is reached, the optimization stops but we do not count this
-            as convergence.
-        stopping_max_iterations (int): If the maximum number of iterations is reached,
-            the optimization stops, but we do not count this as convergence.
+    ``nlopt_neldermead`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
 
     Returns:
         dict: See :ref:`internal_optimizer_output` for details.
 
     """
-    if np.isfinite(lower_bounds).any():
-        warnings.warn(
-            "nlopt_neldermead failed on simple benchmark functions if some but not all "
-            "bounds were finite. Add finite bounds for all parameters for more safety."
-        )
 
     out = _minimize_nlopt(
         criterion_and_derivative,
@@ -141,22 +144,958 @@ def nlopt_neldermead(
     return out
 
 
-def _minimize_nlopt(
+def nlopt_praxis(
+    criterion_and_derivative,
+    x,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Minimize a scalar function using principal-axis method.
+
+    This is a gradient-free local optimizer originally described in:
+    Richard Brent, Algorithms for Minimization without Derivatives
+    (Prentice-Hall, 1972). (Reprinted by Dover, 2002.). It assumes quadratic
+    form of the optimized function and repeatedly updates a set of conjugate
+    search directions.
+
+    The algorithm is not invariant to scaling of the objective function and may
+    fail under its certain rank-preserving transformations (e.g., will lead to
+    a non-quadratic shape of the objective function).
+
+    The algorithm is not determenistic and it is not possible to achieve
+    detereminancy via seed setting.
+
+    The algorithm failed on a simple benchmark function with finite parameter bounds.
+    Passing arguments `lower_bounds` and `upper_bounds` has been disabled for this
+    algorithm.
+
+    The difference between the nlopt implementation an the original implementation is
+    that the nlopt version supports bounds. This is done by returning infinity (Inf)
+    when the constraints are violated. The implementation of bound constraints
+    is achieved at the const of significantly reduced speed of convergence.
+    In case of bounded constraints, this method is dominated by `nlopt_bobyqa`
+    and `nlopt_cobyla`.
+
+    ``nlopt_praxis`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+    algo_info = DEFAULT_ALGO_INFO.copy()
+    algo_info.update({"needs_scaling": True})
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds=None,
+        upper_bounds=None,
+        algorithm=nlopt.LN_PRAXIS,
+        algorithm_name="nlopt_praxis",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+        algo_info=algo_info,
+    )
+
+    return out
+
+
+def nlopt_cobyla(
     criterion_and_derivative,
     x,
     lower_bounds,
     upper_bounds,
-    algorithm,
-    algorithm_name,
     *,
-    convergence_xtol_rel=None,
-    convergence_xtol_abs=None,
-    convergence_ftol_rel=None,
-    convergence_ftol_abs=None,
-    stopping_max_eval=None,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
 ):
-    """Run actual nlopt optimization argument, set relevant attributes."""
-    algo_info = DEFAULT_ALGO_INFO.copy()
+    """Minimize a scalar function using the cobyla method.
+
+    The alggorithm is derived from Powell's Constrained Optimization BY Linear
+    Approximations (COBYLA) algorithm. It is a derivative-free optimizer with
+    nonlinear inequality and equality constrains, described in:
+
+    M. J. D. Powell, "A direct search optimization method that models the
+    objective and constraint functions by linear interpolation," in Advances in
+    Optimization and Numerical Analysis, eds. S. Gomez and J.-P. Hennart (Kluwer
+    Academic: Dordrecht, 1994), p. 51-67
+
+    It constructs successive linear approximations of the objective function and
+    constraints via a simplex of n+1 points (in n dimensions), and optimizes these
+    approximations in a trust region at each step.
+
+    The the nlopt implementation differs from the original implementation in a
+    a few ways:
+    - Incorporates all of the NLopt termination criteria.
+    - Adds explicit support for bound constraints.
+    - Allows the algorithm to increase the trust-reion radius if the predicted
+    imptoovement was approximately right and the simplex is satisfactory.
+    - Pseudo-randomizes simplex steps in the algorithm, aimproving robustness by
+    avoiding accidentally taking steps that don't improve conditioning, preserving
+    the deterministic nature of the algorithm.
+    - Supports unequal initial-step sizes in the different parameters.
+
+
+    ``nlopt_cobyla`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LN_COBYLA,
+        algorithm_name="nlopt_cobyla",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_sbplx(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Minimize a scalar function using the "Subplex" algorithm.
+
+    The alggorithm is a reimplementation of  Tom Rowan's "Subplex" algorithm.
+    See: T. Rowan, "Functional Stability Analysis of Numerical Algorithms",
+    Ph.D. thesis, Department of Computer Sciences, University of Texas at
+    Austin, 1990.
+
+    Subplex is a variant of Nedler-Mead that uses Nedler-Mead on a sequence of
+    subspaces. It is climed to be more efficient and robust than the original
+    Nedler-Mead algorithm.
+
+    The difference between this re-implementation and the original algorithm
+    of Rowan, is that it explicitly supports bound constraints providing big
+    improvement in the case where the optimum lies against one of the constraints.
+
+    ``nlopt_sbplx`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LN_SBPLX,
+        algorithm_name="nlopt_sbplx",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_newuoa(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Minimize a scalar function using the NEWUOA algorithm.
+
+    The algorithm is derived from the NEWUOA subroutine of M.J.D Powell which
+    uses iteratively constructed quadratic approximation of the objctive fucntion
+    to perform derivative-free unconstrained optimization. Fore more details see:
+    M. J. D. Powell, "The NEWUOA software for unconstrained optimization without
+    derivatives," Proc. 40th Workshop on Large Scale Nonlinear Optimization
+    (Erice, Italy, 2004).
+
+    The algorithm in `nlopt` has been modified to support bound constraints. If all
+    of the bound constraints are infinite, this function calls the `nlopt.LN_NEWUOA`
+    optimizers for uncsonstrained optimization. Otherwise, the `nlopt.LN_NEWUOA_BOUND`
+    optimizer for constrained problems.
+
+    `NEWUOA` requires the dimension n of the parameter space to be `≥ 2`, i.e. the
+    implementation does not handle one-dimensional optimization problems.
+
+    ``nlopt_newuoa`` takes the following algo_options
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+    if np.any(np.isfinite(lower_bounds)) or np.any(np.isfinite(upper_bounds)):
+        algo = nlopt.LN_NEWUOA_BOUND
+    else:
+        algo = nlopt.LN_NEWUOA
+
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=algo,
+        algorithm_name="nlopt_newuoa",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_tnewton(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Minimize a scalar function using the "TNEWTON" algorithm.
+
+    The alggorithm is based on a Fortran implementation of a preconditioned
+    inexact truncated Newton algorithm written by Prof. Ladislav Luksan.
+
+    Truncated Newton methods are a set of algorithms designed to solve large scale
+    optimization problems. The algorithms use (inaccurate) approximations of the
+    solutions to Newton equations, using conjugate gradient methodds, to handle the
+    expensive calculations of derivatives during each iteration.
+
+    Detailed description of algorithms is given in: R. S. Dembo and T. Steihaug,
+    "Truncated Newton algorithms for large-scale optimization," Math. Programming
+    26, p. 190-212 (1983), http://doi.org/10.1007/BF02592055.
+
+    ``nlopt_tnewton`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LD_TNEWTON,
+        algorithm_name="nlopt_tnewton",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_lbfgs(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Minimize a scalar function using the "LBFGS" algorithm.
+
+    The alggorithm is based on a Fortran implementation of low storage BFGS algorithm
+    written by Prof. Ladislav Luksan.
+
+    LFBGS is an approximation of the original Broyden–Fletcher–Goldfarb–Shanno algorithm
+    based on limited use of memory. Memory efficiency is obtained by preserving a limi-
+    ted number (<10) of past updates of candidate points and gradient values and using
+    them to approximate the hessian matrix.
+
+    Detailed description of algorithms is given in:
+    J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comput.
+    35, 773-782 (1980).
+    D. C. Liu and J. Nocedal, "On the limited memory BFGS method for large scale
+    optimization," ''Math. Programming' 45, p. 503-528 (1989).
+
+    ``nlopt_lbfgs`` takes the following ``algo_options``
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+
+    Returns:
+        dict: See :ref:`internal_optimizer_output` for details.
+
+    """
+
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LD_TNEWTON,
+        algorithm_name="nlopt_tnewton",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_ccsaq(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+
+    """Minimize a scalar function using CCSAQ algorithm.
+
+    CCSAQ uses the quadratic variant of the conservative convex separable approximation.
+    The algorithm performs gradient based local optimization with equality (but not
+    inequality) constraints. At each candidate point x, a quadratic approximation
+    to the criterion faunction is computed using the value of gradient at point x. A
+    penalty term is incorporated to render optimizaion convex and conservative. The
+    algorithm is "globally convergent" in the sense that it is guaranteed to con-
+    verge to a local optimum from any feasible starting point.
+
+    The implementation is based on CCSA algorithm described in:
+    Krister Svanberg, "A class of globally convergent optimization methods based
+    on conservative convex separable approximations," SIAM J. Optim. 12 (2), p.
+    555-573 (2002)
+
+
+
+    ``nlopt_ccsaq`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    """
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LD_CCSAQ,
+        algorithm_name="nlopt_ccsaq",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_mma(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+
+    """Minimize a scalar function using the method of moving asymptotes (MMA).
+
+    The implementation is based on an algorithm described in:
+    Krister Svanberg, "A class of globally convergent optimization methods based
+    on conservative convex separable approximations," SIAM J. Optim. 12 (2), p.
+    555-573 (2002)
+
+    The algorithm performs gradient based local optimization with equality (but
+    not inequality) constraints. At each candidate point x, an approximation to the
+    criterion faunction is computed using the value of gradient at point x. A quadratic
+    penalty term is incorporated to render optimizaion convex and conservative. The
+    algorithm is "globally convergent" in the sense that it is guaranteed to con-
+    verge to a local optimum from any feasible starting point.
+
+
+
+    ``nlopt_mma`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    """
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LD_MMA,
+        algorithm_name="nlopt_mma",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_var(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+    rank_1_update=True,
+):
+
+    """Minimize a scalar function limited memory switching variable-metric method.
+
+    The algorithm relies on saving only limited number M of past updates of the
+    gradient to approximate the inverse hessian. The large is M, the more memory is
+    consumed
+
+    Detailed explanation of the algorithm, including its two variations of  rank-2 and
+    rank-1 methods can be found in the following paper:
+    J. Vlcek and L. Luksan, "Shifted limited-memory variable metric methods for
+    large-scale unconstrained minimization," J. Computational Appl. Math. 186,
+    p. 365-390 (2006).
+
+    ``nlopt_svmm`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+    - rank_1_update (bool): Whether I rank-1 or rank-2 update is used.
+
+    """
+    if rank_1_update:
+        algo = nlopt.LD_VAR1
+    else:
+        algo = nlopt.LD_VAR2
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=algo,
+        algorithm_name="nlopt_var",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+
+    return out
+
+
+def nlopt_slsqp(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS,
+):
+    """Optimize a scalar function based on SLSQP method.
+
+    SLSQP solves gradient based nonlinearly constrained optimization problems.
+    The algorithm treats the optimization problem as a sequence of constrained
+    least-squares problems.
+
+    The implementation is based on the procedure described in:
+    Dieter Kraft, "A software package for sequential quadratic programming",
+    Technical Report DFVLR-FB 88-28, Institut für Dynamik der Flugsysteme,
+    Oberpfaffenhofen, July 1988.
+    Dieter Kraft, "Algorithm 733: TOMP–Fortran modules for optimal control
+    calculations," ACM Transactions on Mathematical Software, vol. 20, no. 3,
+    pp. 262-281 (1994).
+
+    ``nlopt_slsqp`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    """
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.LD_SLSQP,
+        algorithm_name="nlopt_slsqp",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+    return out
+
+
+def nlopt_direct(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL,
+    locally_biased=False,
+    random_search=False,
+    unscaled_bounds=False,
+):
+    """Optimize a scalar function based on DIRECT method.
+
+    DIRECT is the DIviding RECTangles algorithm for global optimization, described in:
+    D. R. Jones, C. D. Perttunen, and B. E. Stuckmann, "Lipschitzian optimization
+    without the lipschitz constant," J. Optimization Theory and Applications, vol.
+    79, p. 157 (1993).
+
+    Variations of the algorithm include locally biased routines (distinguished by _L
+    suffix) that prove to be more efficients for functions that have few local minima.
+    See the following for the DIRECT_L variant:
+
+    J. M. Gablonsky and C. T. Kelley, "A locally-biased form of the DIRECT algorithm,"
+    J. Global Optimization, vol. 21 (1), p. 27-37 (2001).
+
+    Locally biased algorithms can be implmented both with deterministic and random
+    (distinguished by _RAND suffix) search algorithm.
+
+    Finally, both original and locally biased variants can be implemented with and
+    without the rescaling of the bound constraints.
+
+    Boolean arguments `locally_biased`, 'random_search', and 'unscaled_bouds' can be
+    set to `True` or `False` to determine which method is run. The comprehensive list
+    of available methods are:
+    - DIRECT
+    - DIRECT_L
+    - DIRECT_L_NOSCAL
+    - DIRECT_L_RAND
+    - DIRECT_L_RAND_NOSCAL
+    - DIRECT_RAND
+
+    ``nlopt_direct`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+    - locally_biased (bool): Whether the "L" version of the algorithm is selected.
+    - random_search (bool): Whether the randomized version of the algorithm is selected.
+    - unscaled_bounds (bool): Whether the "NOSCAL" version of the algorithm is selected.
+
+    """
+    if not locally_biased and not random_search and not unscaled_bounds:
+        algo = nlopt.GN_DIRECT
+    elif locally_biased and not random_search and not unscaled_bounds:
+        algo = nlopt.GN_DIRECT_L
+    elif locally_biased and not random_search and unscaled_bounds:
+        algo = nlopt.GN_DIRECT_L_NOSCAL
+    elif locally_biased and random_search and not unscaled_bounds:
+        algo = nlopt.GN_DIRECT_L_RAND
+    elif locally_biased and random_search and unscaled_bounds:
+        algo = nlopt.GN_DIRECT_L_RAND_NOSCAL
+    elif not locally_biased and not random_search and unscaled_bounds:
+        algo = nlopt.GN_DIRECT_NOSCAL
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=algo,
+        algorithm_name="nlopt_direct",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+    return out
+
+
+def nlopt_esch(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL,
+):
+    """Optimize a scalar function using the ESCH algorithm.
+
+    ESCH is an evolutionary algorithm that supports bound constraints only. Specifi
+    cally, it does not support nonlinear constraints.
+
+    More information on this method can be found in:
+    C. H. da Silva Santos, M. S. Goncalves, and H. E. Hernandez-Figueroa, "Designing
+    Novel Photonic Devices by Bio-Inspired Computing," IEEE Photonics Technology
+    Letters 22 (15), pp. 1177–1179 (2010).
+    C. H. da Silva Santos, "Parallel and Bio-Inspired Computing Applied to Analyze
+    Microwave and Photonic Metamaterial Strucutures," Ph.D. thesis, University of
+    Campinas, (2010).
+    H.-G. Beyer and H.-P. Schwefel, "Evolution Strategies: A Comprehensive Introduction,
+    "Journal Natural Computing, 1 (1), pp. 3–52 (2002).
+    Ingo Rechenberg, "Evolutionsstrategie – Optimierung technischer Systeme nach
+    Prinzipien der biologischen Evolution," Ph.D. thesis (1971), Reprinted by
+    Fromman-Holzboog (1973).
+
+    ``nlopt_esch`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    """
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.GN_ESCH,
+        algorithm_name="nlopt_esch",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+    return out
+
+
+def nlopt_isres(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL,
+):
+    """Optimize a scalar function using the ISRES algorithm.
+
+    ISRES is an implementation of "Improved Stochastic Evolution Strategy" written
+    for solving optimization problems with non-linear constraints. The algorithm
+    is supposed to be a global method, in that it has heuristics to avoid local
+    minima. However, no convergence proof is available.
+
+    The original method and a refined version can be found, respecively, in:
+    Thomas Philip Runarsson and Xin Yao, "Search biases in constrained evolutionary
+    optimization," IEEE Trans. on Systems, Man, and Cybernetics Part C: Applications
+    and Reviews, vol. 35 (no. 2), pp. 233-243 (2005).
+    Thomas P. Runarsson and Xin Yao, "Stochastic ranking for constrained evolutionary
+    optimization," IEEE Trans. Evolutionary Computation, vol. 4 (no. 3), pp. 284-294
+    (2000).
+
+
+    ``nlopt_isres`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+
+    """
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.GN_ISRES,
+        algorithm_name="nlopt_isres",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+    )
+    return out
+
+
+def nlopt_crs2_lm(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    *,
+    convergence_relative_params_tolerance=CONVERGENCE_RELATIVE_PARAMS_TOLERANCE,
+    convergence_absolute_params_tolerance=CONVERGENCE_ABSOLUTE_PARAMS_TOLERANCE,
+    convergence_relative_criterion_tolerance=CONVERGENCE_RELATIVE_CRITERION_TOLERANCE,
+    convergence_absolute_criterion_tolerance=CONVERGENCE_ABSOLUTE_CRITERION_TOLERANCE,
+    stopping_max_criterion_evaluations=STOPPING_MAX_CRITERION_EVALUATIONS_GLOBAL,
+    genetic_popsize=None,
+):
+    """Optimize a scalar function using the CRS2_LM algorithm.
+
+    This implementation of controlled random search method with local mutation is
+    based on:
+    P. Kaelo and M. M. Ali, "Some variants of the controlled random search algorithm
+    for global optimization," J. Optim. Theory Appl. 130 (2), 253-264 (2006).
+
+    The original CRS method is described in:
+    W. L. Price, "A controlled random search procedure for global optimization,"
+    in Towards Global Optimization 2, p. 71-84 edited by L. C. W. Dixon and G. P.
+    Szego (North-Holland Press, Amsterdam, 1978).
+    W. L. Price, "Global optimization by controlled random search," J. Optim. Theory
+    Appl. 40 (3), p. 333-348 (1983).
+
+    CRS class of algorithms starts with random population of points and evolves the
+    points "randomly". The size of the initial population can be set via the param-
+    meter genetic_popsize. If the user doesn't specify a value, it is
+    set to the nlopt default of 10*(n+1).
+
+    ``nlopt_isres`` supports the following ``algo_options``:
+
+    - convergence.relative_params_tolerance (float):  Stop when the relative movement
+      between parameter vectors is smaller than this.
+    - convergence.absolute_params_tolerance (float): Stop when the absolute movement
+      between parameter vectors is smaller than this.
+    - convergence.relative_criterion_tolerance (float): Stop when the relative
+      improvement between two iterations is smaller than this.
+    - convergence.absolute_criterion_tolerance (float): Stop when the change of the
+      criterion function between two iterations is smaller than this.
+    - stopping.max_criterion_evaluations (int): If the maximum number of function
+      evaluation is reached, the optimization stops but we do not count this
+      as convergence.
+    - genetic_popsize(int): The size of the population of the starting
+      points.
+
+    """
+    if not genetic_popsize:
+        genetic_popsize = 10 * (len(x) + 1)
+    out = _minimize_nlopt(
+        criterion_and_derivative,
+        x,
+        lower_bounds,
+        upper_bounds,
+        algorithm=nlopt.GN_CRS2_LM,
+        algorithm_name="nlopt_crs2_lm",
+        convergence_xtol_rel=convergence_relative_params_tolerance,
+        convergence_xtol_abs=convergence_absolute_params_tolerance,
+        convergence_ftol_rel=convergence_relative_criterion_tolerance,
+        convergence_ftol_abs=convergence_absolute_criterion_tolerance,
+        stopping_max_eval=stopping_max_criterion_evaluations,
+        population_size=genetic_popsize,
+    )
+    return out
+
+
+def _minimize_nlopt(
+    criterion_and_derivative,
+    x,
+    lower_bounds,
+    upper_bounds,
+    algorithm,
+    algorithm_name,
+    *,
+    convergence_xtol_rel=None,
+    convergence_xtol_abs=None,
+    convergence_ftol_rel=None,
+    convergence_ftol_abs=None,
+    stopping_max_eval=None,
+    population_size=None,
+    algo_info=None,
+):
+    """Run actual nlopt optimization argument, set relevant attributes."""
+    if algo_info is None:
+        algo_info = DEFAULT_ALGO_INFO.copy()
+    else:
+        algo_info = algo_info.copy()
     algo_info["name"] = algorithm_name
 
     def func(x, grad):
@@ -190,6 +1129,8 @@ def _minimize_nlopt(
         opt.set_upper_bounds(upper_bounds)
     if stopping_max_eval is not None:
         opt.set_maxeval(stopping_max_eval)
+    if population_size is not None:
+        opt.set_population(population_size)
     opt.set_min_objective(func)
     solution_x = opt.optimize(x)
     return _process_nlopt_results(opt, solution_x)
diff --git a/tox.ini b/tox.ini
index d9fdc43..cc7371b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -13,6 +13,9 @@ conda_channels =
     opensourceeconomics
     conda-forge
     defaults
+deps =
+    Py-BOBYQA
+    DFO-LS
 conda_deps =
     bokeh >= 1.3
     click
@@ -32,6 +35,7 @@ conda_deps =
     seaborn
     dill
     cyipopt
+    nlopt
 commands = pytest {posargs}
 
 


Test patch:
diff --git a/estimagic/tests/optimization/test_many_algorithms.py b/estimagic/tests/optimization/test_many_algorithms.py
index c1a3e78..7421659 100644
--- a/estimagic/tests/optimization/test_many_algorithms.py
+++ b/estimagic/tests/optimization/test_many_algorithms.py
@@ -13,17 +13,28 @@ from numpy.testing import assert_array_almost_equal as aaae
 
 from estimagic.examples.criterion_functions import sos_dict_criterion
 from estimagic.optimization import AVAILABLE_ALGORITHMS
+from estimagic.optimization import GLOBAL_ALGORITHMS
 from estimagic.optimization.optimize import minimize
 
 
+LOCAL_ALGORITHMS = {
+    key: value
+    for key, value in AVAILABLE_ALGORITHMS.items()
+    if key not in GLOBAL_ALGORITHMS
+}
+
+GLOBAL_ALGORITHMS_AVAILABLE = [
+    name for name in AVAILABLE_ALGORITHMS if name in GLOBAL_ALGORITHMS
+]
+
 BOUNDED_ALGORITHMS = []
-for name, func in AVAILABLE_ALGORITHMS.items():
+for name, func in LOCAL_ALGORITHMS.items():
     arguments = list(inspect.signature(func).parameters)
     if "lower_bounds" in arguments and "upper_bounds" in arguments:
         BOUNDED_ALGORITHMS.append(name)
 
 
-@pytest.mark.parametrize("algorithm", AVAILABLE_ALGORITHMS)
+@pytest.mark.parametrize("algorithm", LOCAL_ALGORITHMS)
 def test_algorithm_on_sum_of_squares(algorithm):
     params = pd.DataFrame()
     params["value"] = [1, 2, 3]
@@ -51,3 +62,19 @@ def test_algorithm_on_sum_of_squares_with_binding_bounds(algorithm):
     )
 
     aaae(res["solution_params"]["value"].to_numpy(), np.array([1, 0, -1]), decimal=3)
+
+
+@pytest.mark.parametrize("algorithm", GLOBAL_ALGORITHMS_AVAILABLE)
+def test_global_algorithms_on_sum_of_squares(algorithm):
+    params = pd.DataFrame()
+    params["value"] = [2.5, 2, -1]
+    params["lower_bound"] = [0.5, -1, -1.5]
+    params["upper_bound"] = [3, 3, -0.5]
+    res = minimize(
+        criterion=sos_dict_criterion,
+        params=params,
+        algorithm=algorithm,
+    )
+    aaae(
+        res["solution_params"]["value"].to_numpy(), np.array([0.5, 0, -0.5]), decimal=1
+    )