Package: hyponic
HypONIC -- Hyperparameter Optimization with Nature Inspired ComputingHypONICis a hyperparameter optimization library that uses various nature inspired computing algorithms to optimize
the hyperparameters of machine learning models. The library provides a simple interface for sklearn models.Documentation:https://hyponic.readthedocs.io/en/latest/Library Structure|-- hyponic/
|   |-- hyponic.py
|   |-- space.py
|   |
|   |-- metrics/
|   |   |-- classification.py
|   |   |-- regression.py
|   |   |-- decorators.py
|   |
|   |-- optimizers/
|   |   |-- base_optimizer.py
|   |   |
|   |   |-- swarm_based/
|   |   |   |-- ABC.py
|   |   |   |-- ACO.py
|   |   |   |-- PSO.py
|   |   |
|   |   |-- physics_based/
|   |   |   |-- SA.py
|   |   |
|   |   |-- genetic_based/
|   |       |-- GA.py
|   |
|   |-- space/
|   |   |-- encoders.py (planned)
|   |   |-- mappers.py  (planned)
|   |
|   |-- utils/
|       |-- history.py
|       |-- problem_identifier.py
|-- examples/
    |-- datasets/InstallationInstall from PyPIpipinstallhyponicMinimal ExampleThis is a minimal example for tuning hyperparameters of theSVCfromsklearn.
The default optimization isInertia Weight Particle Swarm Optimization(IWPSO),
as it has high applicability and has shown fast convergence.# 1. Import of the model, dataset, metric and optimizer#    with the algorithm of choicefromsklearn.svmimportSVCfromsklearn.datasetsimportload_winefromhyponic.hyponicimportHypONIC#    ...in this case, log_loss and Inertia Weight PSOfromhyponic.metrics.classificationimportlog_lossfromhyponic.optimizers.swarm_based.PSOimportIWPSO# 2. Creating the model and loading the datasetmodel=SVC()X,y=load_wine(return_X_y=True)# 3. Defining our problemhyperparams={"C":(0.01,1),"kernel":["linear","poly","rbf","sigmoid"],"degree":[2,3,4,5],}# 4. Parameterizing the optimizeroptimizer_kwargs={"epoch":10,"population_size":10,}# 5. Passing all to the optimizerhyponic=HypONIC(model,X,y,log_loss,optimizer=IWPSO,**optimizer_kwargs)# 6. Solving for the given problemhyponic.optimize(hyperparams,verbose=True)print(hyponic.get_optimized_parameters())print(hyponic.get_optimized_metric())print(hyponic.get_optimized_model())Implementation detailsHypONIClibrary follows a common high-level approach by providing a unified interface for applying an optimization algorithm
of choice to the parameters of a machine learning model (based on theBaseEstimatorfrom thesklearn). Evaluation of the
optimization process is done by a metric of choice.MetricsHypONICprovides a list of classical evaluation metrics:Mean Squared ErrorMean Absolute ErrorRoot Mean Square ErrorHuber LossLog LossBinary Cross entropyPrecision ScoreAccuracy ScoreRecall ScoreF1 ScoreR2 Score...and many others.HypONIClibrary provides decorators for annotating custom metrics:@minimize_metric,@maximize_metric, and@add_metric_aliases.Use case for the first two decorators is to annotate the metric function with the optimization goal, i.e. let's say
we want to add a custom metric functioncustom_metricto the library, and we want to minimize it. To do so, we
annotate the function with the@minimize_metricdecorator:fromhyponic.hyponicimportHypONICfromhyponic.optimizers.swarm_based.PSOimportIWPSOfromhyponic.metrics.decoratorsimportminimize_metricmodel=...X,y=...,...optimizer_kwargs=...# -- HERE IS THE DECORATOR --@minimize_metricdefcustom_metric(y_true,y_pred):...hyponic=HypONIC(model,X,y,custom_metric,optimizer=IWPSO,**optimizer_kwargs)Under the hood, the decorator adds aminimizeattribute to the function, which is later used by the optimizer to
determine the optimization goal.Not only that, but the decorator also adds the function to the list of available metrics, so that it can be used
as a string literal in theHypONICconstructor, i.e.fromhyponic.hyponicimportHypONICfromhyponic.optimizers.swarm_based.PSOimportIWPSOfromhyponic.metrics.decoratorsimportminimize_metricmodel=...X,y=...,...optimizer_kwargs=...@minimize_metricdefcustom_metric(y_true,y_pred):...hyponic=HypONIC(model,X,y,"custom_metric",optimizer=IWPSO,**optimizer_kwargs)For the standard metrics, the decorators are already applied, so that they can be used as string literals in theHypONICconstructor without the need to import them from thehyponic.metricsmodule.By default, function aliases are generated from the function name, however, it is possible to add custom aliases
using the@add_metric_aliasesdecorator:fromhyponic.hyponicimportHypONICfromhyponic.optimizers.swarm_based.PSOimportIWPSOfromhyponic.metrics.decoratorsimportminimize_metric,add_metric_aliasesmodel=...X,y=...,...optimizer_kwargs=...hyperparams=...@minimize_metric@add_metric_aliases("custom_metric","short_alias")defcustom_metric(y_true,y_pred):...hyponic=HypONIC(model,X,y,"short_alias",optimizer=IWPSO,**optimizer_kwargs)hyponic.optimize(hyperparams,verbose=True)Output that this code produces will show that theshort_alias, used as a metric alias, resolves to thecustom_metric.Hyperparameter SpaceHypONIClibrary supports mixed space of hyperparameters, meaning that one optimization space can have both
dimensions of discrete and continuous parameters. The notation for each space is the following:hyperparams={"min_impurity_decrease":(0.0,0.9),# Continuous space -- (lower bound, upper bound)"min_samples_split":[2,3,4,5,6],# Discrete space   -- a list of values of any type"criterion":["absolute_error","squared_error"]# Discrete space   -- a list of values of any type}For the discrete space an automatic uniform mapping* to the continuous space is provided if needed. Discrete space of
non-numerical values is encoded first and then uniformly mapped to the continuous space.*is subject to changeFeaturesHypONICsupports different nature inspired computing algorithms.
Please note that the library is currently under development and may contain a significant number of bugs and errors.
All algorithms are subject to further improvement.The following algorithms are currently implemented or are planned to be implemented:Swarm-based:[x] Particle Swarm Optimization (PSO)[x] Inertia Weight PSO (IWPSO)[x] Ant Colony Optimization (ACO)[x] Artificial Bee Colony (ABC)[x] Grey Wolf Optimizer (GWO)[x] Cuckoo Search (CS)[ ] Firefly Algorithm (FA)Physics-based:[x] Simulated Annealing (SA)Genetic-based:[x] Genetic Algorithm (GA)The following features are currently implemented or are planned to be implemented:[x] Early stopping based on the number of epoch without improvement[x] History of optimization[x] Parallelization[x] Visualization of the history[ ] Visualization of the optimization process[ ] Customizable stopping criteria[ ] Partial fit[ ] Different population initializations[x] Identifying the problem type (regression, classification)[ ] Support for pytorch or keras models