Package: gradient-free-optimizers
Simple and reliable optimization with local, global, population-based and sequential techniques in numerical discrete search spaces.Master status:Code quality:Latest versions:IntroductionGradient-Free-Optimizers provides a collection of easy to use optimization techniques,
whose objective function only requires an arbitrary score that gets maximized.
This makes gradient-free methods capable of solving various optimization problems, including:Optimizing arbitrary mathematical functions.Fitting multiple gauss-distributions to data.Hyperparameter-optimization of machine-learning methods.Gradient-Free-Optimizers is the optimization backend ofHyperactive(in v3.0.0 and higher) but it can also be used by itself as a leaner and simpler optimization toolkit.Optimization algorithms•Installation•Examples•API reference•RoadmapMain featuresEasy to use:Simple API-designYou can optimize anything that can be defined in a python function. For example a simple parabola function:defobjective_function(para):score=para["x1"]*para["x1"]return-scoreDefine where to search via numpy ranges:search_space={"x":np.arange(0,5,0.1),}That`s all the information the algorithm needs to search for the maximum in the objective function:fromgradient_free_optimizersimportRandomSearchOptimizeropt=RandomSearchOptimizer(search_space)opt.search(objective_function,n_iter=100000)Receive prepared information about ongoing and finished optimization runsDuring the optimization you will receive ongoing information in a progress bar:current best scorethe position in the search space of the current best scorethe iteration when the current best score was foundother information about the progress native to tqdmHigh performance:Modern optimization techniquesGradient-Free-Optimizers provides not just meta-heuristic optimization methods but also sequential model based optimizers like bayesian optimization, which delivers good results for expensive objetive functions like deep-learning models.Lightweight backendEven for the very simple parabola function the optimization time is about 60% of the entire iteration time when optimizing with random search.  This shows, that (despite all its features) Gradient-Free-Optimizers has an efficient optimization backend without any unnecessary slowdown.Save time with memory dictionaryPer default Gradient-Free-Optimizers will look for the current position in a memory dictionary before evaluating the objective function.If the position is not in the dictionary the objective function will be evaluated and the position and score is saved in the dictionary.If a position is already saved in the dictionary Gradient-Free-Optimizers will just extract the score from it instead of evaluating the objective function. This avoids reevaluating computationally expensive objective functions (machine- or deep-learning) and therefore saves time.High reliability:Extensive testingGradient-Free-Optimizers is extensivly tested with more than 400 tests in 2500 lines of test code. This includes the testing of:Each optimization algorithmEach optimization parameterAll attributes that are part of the public apiPerformance test for each optimizerEach optimization algorithm must perform above a certain threshold to be included. Poorly performing algorithms are reworked or scraped.Optimization algorithms:Gradient-Free-Optimizers supports a variety of optimization algorithms, which can make choosing the right algorithm a tedious endeavor. The gifs in this section give a visual representation how the different optimization algorithms explore the search space and exploit the collected information about the search space for a convex and non-convex objective function. More detailed explanations of all optimization algorithms can be found in theofficial documentation.Local OptimizationHill ClimbingEvaluates the score of n neighbours in an epsilon environment and moves to the best one.Convex FunctionNon-convex FunctionStochastic Hill ClimbingAdds a probability to the hill climbing to move to a worse position in the search-space to escape local optima.Convex FunctionNon-convex FunctionRepulsing Hill ClimbingHill climbing algorithm with the addition of increasing epsilon by a factor if no better neighbour was found.Convex FunctionNon-convex FunctionSimulated AnnealingAdds a probability to the hill climbing to move to a worse position in the search-space to escape local optima with decreasing probability over time.Convex FunctionNon-convex FunctionDownhill Simplex OptimizationConstructs a simplex from multiple positions that moves through the search-space by reflecting, expanding, contracting or shrinking.Convex FunctionNon-convex FunctionGlobal OptimizationRandom SearchMoves to random positions in each iteration.Convex FunctionNon-convex FunctionGrid SearchGrid-search that moves through search-space diagonal (with step-size=1) starting from a corner.Convex FunctionNon-convex FunctionRandom Restart Hill ClimbingHill climbingm, that moves to a random position after n iterations.Convex FunctionNon-convex FunctionRandom AnnealingHill Climbing, that has large epsilon at the start of the search decreasing over time.Convex FunctionNon-convex FunctionPattern SearchCreates cross-shaped collection of positions that move through search-space by moving as a whole towards optima or shrinking the cross.Convex FunctionNon-convex FunctionPowell's MethodOptimizes each search-space dimension at a time with a hill-climbing algorithm.Convex FunctionNon-convex FunctionPopulation-Based OptimizationParallel TemperingPopulation of n simulated annealers, which occasionally swap transition probabilities.Convex FunctionNon-convex FunctionParticle Swarm OptimizationPopulation of n particles attracting each other and moving towards the best particle.Convex FunctionNon-convex FunctionSpiral OptimizationPopulation of n particles moving in a spiral pattern around the best position.Convex FunctionNon-convex FunctionEvolution StrategyPopulation of n hill climbers occasionally mixing positional information and removing worst positions from population.Convex FunctionNon-convex FunctionSequential Model-Based OptimizationBayesian OptimizationGaussian process fitting to explored positions and predicting promising new positions.Convex FunctionNon-convex FunctionLipschitz OptimizationCalculates an upper bound from the distances of the previously explored positions to find new promising positions.Convex FunctionNon-convex FunctionDIRECT algorithmSeparates search space into subspaces. It evaluates the center position of each subspace to decide which subspace to sepate further.Convex FunctionNon-convex FunctionTree of Parzen EstimatorsKernel density estimators fitting to good and bad explored positions and predicting promising new positions.Convex FunctionNon-convex FunctionForest OptimizerEnsemble of decision trees fitting to explored positions and predicting promising new positions.Convex FunctionNon-convex FunctionSideprojects and ToolsThe following packages are designed to support Gradient-Free-Optimizers and expand its use cases.PackageDescriptionSearch-Data-CollectorSimple tool to save search-data during or after the optimization run into csv-files.Search-Data-ExplorerVisualize search-data with plotly inside a streamlit dashboard.If you want news about Gradient-Free-Optimizers and related projects you can follow me ontwitter.InstallationThe most recent version of Gradient-Free-Optimizers is available on PyPi:pip install gradient-free-optimizersExamplesConvex functionimportnumpyasnpfromgradient_free_optimizersimportRandomSearchOptimizerdefparabola_function(para):loss=para["x"]*para["x"]return-losssearch_space={"x":np.arange(-10,10,0.1)}opt=RandomSearchOptimizer(search_space)opt.search(parabola_function,n_iter=100000)Non-convex functionimportnumpyasnpfromgradient_free_optimizersimportRandomSearchOptimizerdefackley_function(pos_new):x=pos_new["x1"]y=pos_new["x2"]a1=-20*np.exp(-0.2*np.sqrt(0.5*(x*x+y*y)))a2=-np.exp(0.5*(np.cos(2*np.pi*x)+np.cos(2*np.pi*y)))score=a1+a2+20return-scoresearch_space={"x1":np.arange(-100,101,0.1),"x2":np.arange(-100,101,0.1),}opt=RandomSearchOptimizer(search_space)opt.search(ackley_function,n_iter=30000)Machine learning exampleimportnumpyasnpfromsklearn.model_selectionimportcross_val_scorefromsklearn.ensembleimportGradientBoostingClassifierfromsklearn.datasetsimportload_winefromgradient_free_optimizersimportHillClimbingOptimizerdata=load_wine()X,y=data.data,data.targetdefmodel(para):gbc=GradientBoostingClassifier(n_estimators=para["n_estimators"],max_depth=para["max_depth"],min_samples_split=para["min_samples_split"],min_samples_leaf=para["min_samples_leaf"],)scores=cross_val_score(gbc,X,y,cv=3)returnscores.mean()search_space={"n_estimators":np.arange(20,120,1),"max_depth":np.arange(2,12,1),"min_samples_split":np.arange(2,12,1),"min_samples_leaf":np.arange(1,12,1),}opt=HillClimbingOptimizer(search_space)opt.search(model,n_iter=50)Roadmapv0.3.0:heavy_check_mark:add sampling parameter to Bayesian optimizeradd warnings parameter to Bayesian optimizerimprove access to parameters of optimizers within population-based-optimizers (e.g. annealing rate of simulated annealing population in parallel tempering)v0.4.0:heavy_check_mark:add early stopping parameterv0.5.0:heavy_check_mark:add grid-search to optimizersimpoved performance testing for optimizersv1.0.0:heavy_check_mark:Finalize API (1.0.0)add Downhill-simplex algorithm to optimizersadd Pattern search to optimizersadd Powell's method to optimizersadd parallel random annealing to optimizersadd ensemble-optimizer to optimizersv1.1.0:heavy_check_mark:add Spiral Optimizationadd Lipschitz Optimizerprint the random seed for reproducibilityv1.2.0:heavy_check_mark:add DIRECT algorithmautomatically add random initial positions if necessary (often requested)v1.3.0add support for constrained optimizationv1.4.0add API, testing and doc to (better) use GFO as backend-optimization packageadd Grid search paraneter that changes direction of searchadd Random search parameter that enables to avoid replacement of the samplingadd SMBO parameter that enables to avoid replacement of the samplingv1.5.0add Ant-colony optimizationv2.0.0add other acquisition functions to smbo (Probability of improvement, Entropy search, ...)...Gradient Free Optimizers <=> HyperactiveGradient-Free-Optimizers was created as the optimization backend of theHyperactive package. Therefore the algorithms are exactly the same in both packages and deliver the same results.
However you can still use Gradient-Free-Optimizers as a standalone package.
The separation of Gradient-Free-Optimizers from Hyperactive enables multiple advantages:Even easier to use than HyperactiveSeparate and more thorough testingOther developers can easily use GFOs as an optimizaton backend if desiredBetter isolation from the complex information flow in Hyperactive. GFOs only uses positions and scores in a N-dimensional search-space. It returns only the new position after each iteration.a smaller and cleaner code base, if you want to explore my implementation of these optimization techniques.While Gradient-Free-Optimizers is relatively simple, Hyperactive is a more complex project with additional features. The differences between Gradient-Free-Optimizers and Hyperactive are listed in the following table:Gradient-Free-OptimizersHyperactiveSearch space compositiononly numericalnumbers, strings and functionsParallel Computingnot supportedyes, via multiprocessing or joblibDistributed computingnot supportedyes, via data sharing at runtimeVisualizationnot supportedyes, via a streamlit-dashboardCitation@Misc{gfo2020,
  author =   {{Simon Blanke}},
  title =    {{Gradient-Free-Optimizers}: Simple and reliable optimization with local, global, population-based and sequential techniques in numerical search spaces.},
  howpublished = {\url{https://github.com/SimonBlanke}},
  year = {since 2020}
}LicenseGradient-Free-Optimizers is licensed under the following License: