Package: pymetagen-datalabupo
MetaGen: A framework for metaheuristic development and hyperparameter optimization in machine and deep learningMachine and deep learning have transformed the field of computing in recent decades, delivering impressive accuracy
results when processing large datasets. However, a common challenge arises when implementing these algorithms: setting
the hyperparameters. Many algorithms are sensitive to these adjustments, and the quality of the results heavily depends
on the choices made. Metaheuristics are a widely used strategy for finding optimal hyperparameter settings for a
specific algorithm.MetaGenis a comprehensive and user-friendly metaheuristic development framework that provides tools to define and
solve hyperparameter optimization in machine and deep learning.InstalationMetaGenonly requires python (>=3.10) as the library has been built completely on Python.For the installation, the easiest way is to use pip:pip install pymetagen-datalabupoAPI referenceThe official documentation is available in:https://pymetagen.readthedocs.io.DevelopmentNew contributors from all experience levels are welcomed. To contribute, you can open an issue or sending a pull
request.For testing the repository you just need to execute the following command:pytest testIn this document:MetaGenFeaturesMetaGenvs. other proposalsHow to implement a metaheuristic withMetaGenHow to perform a hyperparameter optimization withMetaGenResourcesMetaGenfeaturesMetaheuristic development framework.It provides convenient solution management for the developer and isolates them
from the intricacies of the search space.Deployment platform for metaheuristic execution.It offers an effortless problem definition process that doesn't require advanced
programming skills.Standard interface between metaheuristic developers and users.Both developers and users can "speak the same language" through theMetaGenpackage.Specialized tools for deep learning hyperparameter tuning.It provides optimization capabilities not just at the layer level but also at the architecture level.Dynamic deep learning architecture adjustment.Users are not restricted to optimizing a fixed architecture (e.g., a fixed number of layers); they can also adjust
specific hyperparameters per layer in a variable architecture.Python type hints annotations.Implemented using Python type hints annotations, it provides developers and users with an efficient environment for
coding, debugging, and maintenance.Full compatibility with Python machine and deep learning packages.Thanks to its standard and flexible interface, it is compatible with all Python packages such
asscikit-learn,keras, andtensorflow.RS, GA and CVOA metaheuristic implementations.It offers tow popular metaheuristic implementations, Random search and Genetic Algorithm optimization. Furthermore, it
includes the implementation of CVOA, a metaheuristic inspired by the SARS-CoV-2 propagation model, the virus
responsible for COVID-19. CVOA was initially used to optimize long-short-term memory (LSTM) networks for electricity
demand forecasting, but its positive results have inspired its use for other machine and deep learning models.MetaGen vs. other proposalsMetaGenkerastuneroptunachocolatehyperopt and hyperassherpabtbtalostest-tubeadvisorray-tuneA new framework for developing custom metaheuristics with a simplified coding processYesNoYesNoNoYesNoNoNoNoNoRepresentation of problems and solutions implementation, allowing metaheuristic developers to provide a standard and intuitive interfaceYesNoNoNoNoNoNoNoNoNoNoSpecific mechanisms for optimizing deep learning architectures and hyperparametersYesYesNoNoNoNoNoNoNoNoNoDynamic changes to a deep learning model’s architecture allow the number of layers and hyperparameters to adjust during metaheuristic executionsYesNoNoYesNoNoNoNoNoNoNoUser-friendly and accessible to the general public with limited programming skillsYesNoNoNoNoNoNoNoNoNoYesPython-type hints implementation to help third-party developers and ease debuggingYesNoNoNoNoNoNoNoNoNoNoMechanisms to improve metaheuristics execution across distributed systemsNoYesYesNoYesYesNoYesYesNoYesThe package includes visualization tools to analyze the performance of metaheuristicsNoYesYesNoNoNoNoNoYesYesNoHow to implement a metaheuristic withMetaGenThe random search algorithm generates a search space with a specific number of potential solutions, then alters these
potential solutions a specified number of times. The potential solution with the lowest fitness function value after
each iteration is considered the global solution of the random search.TheDomainandSolutionclasses are required, so they are
imported from themetagen.frameworkpackage.Finally, theCallableandListclasses are imported for typing management, and thedeepcopymethod from the
standard copy package is used to preserve a consistent copy of the global solution for each iterationfromcopyimportdeepcopyfromtypingimportCallable,Listfrommetagen.frameworkimportDomain,SolutionPrototypeTheRandomSearchmetaheuristic function takes as input a problem definition, which is composed of
aDomainobject and a
fitness function that takes aSolutionobject as
input and returns afloat.The method also has two arguments to control the metaheuristic, thesearch space size(default set to30) and thenumber of iterations(default set to20). Thesearch space sizecontrols the number of potential solutions
generated, and thenumber of iterationsdetermine the number of times the search space will be altered.The result of anRandomSearchrun will be
aSolutionobject that assigns the variables of
theDomainobject in a way
that optimizes the function.In order to encapsulate all these characteristics, aRandomSearchclass is defined.classRandomSearch:def__init__(self,domain:Domain,fitness:Callable[[Solution],float],search_space_size:int=30,iterations:int=20)->None:self.domain=domainself.fitness=fitnessself.search_space_size=search_space_sizeself.iterations=iterationsSearch space buildingThe first step of the method involves constructing the search space, which consists ofsearch_space_sizepotential
solutions orSolutionobjects.Each new potential solution is randomly generated using
theSolution'sinitmethod, which creates
aSolutionobject from
aDomain.The newly createdSolutionis then evaluated
using theevaluatemethod passing the fitness function, and all the potential solutions are stored in aListofSolution. A copy of
theSolutionwith the minimum function value is
also kept.defrun(self)->Solution:search_space:List[Solution]=list()for_inrange(0,search_space_size):initial_solution:Solution=Solution()initial_solution.evaluate(self.fitness)search_space.append(initial_solution)global_solution:Solution=deepcopy(min(search_space))Altering the search spaceThe final step involves modifying the potential solutions in the search space overiterationiterations.
EachSolutionis modified using themutatemethod, which modifies the variable values of
aSolutionwhile taking into account
theDomain.The method randomly selects a variable in
theSolutionto modify and changes its value
randomly while also respecting
theDomain. If the
modifiedSolutionis better than the current
globalSolution, the latter is updated with a
copy of the former.Finally, theglobal_solutionis returned.for_inrange(0,iterations):forpsinsearch_space:ps.mutate()ps.evaluate(self.fitness)ifps<solution:global_solution=deepcopy(ps)returnglobal_solutionSee alsoSolution APIDomain APIGoogle Colab Notebook ofImplementing Random Search metaheuristicHow to perform a hyperparameter optimization withMetaGenIn a typical machine learning regression problem. The goal is to find the hyperparameters that build
the best model for a training set. To do this, theDomainobject is constructed by defining a variable for each parameter to optimize.Defining the Domain.In this case, three variables are defined: a $REAL$ variable calledalpha, with values in the range of $[0.0001, 0.001]$, is defined using thedefine_realmethod, and an $INTEGER$ variable callediterations, with values in the range of $[5, 200]$, is defined using thedefine_integermethod. Additionally, a $CATEGORICAL$ variable is defined using thedefine_categoricalmethod, with the namelossand a list of unique values includingsquared_error,huber, andepsilon_insensitive.frommetagen.frameworkimportDomainregression_domain=Domain()regression_domain.define_real("alpha",0.0001,0.001)regression_domain.define_integer("iterations",5,200)regression_domain.define_categorical("loss",["squared_error","huber","epsilon_insensitive"])Implementing the fitness functionThe fitness function must then construct a regression model using the training dataset and the hyperparameters of the potential solution. In this case, the sklearn package is used for the machine learning operations.A synthetic training dataset with $1000$ instances and $4$ features is generated using themake_regressionmethod from thesklearn.datasetspackage, and it is loaded into two variables, the inputsXand the expected outputsy.fromsklearn.datasetsimportmake_regressionX,y=make_regression(n_samples=1000,n_features=4)The functionregression_fitnessis defined with aSolutionobject as an input parameter. The values ofloss,iterations, and the hyperparameteralphaare obtained through the bracketPythonoperator.A regression model using stochastic gradient descent is constructed using theSGDRegressorclass from thesklearn.linear_modelpackage and the obtained values. Cross-validation training is performed using thecross_val_scorefunction from thesklearn.model_selectionpackage by passing the configured model and the training dataset (Xandy). The cross-validation process is set to return the negative value of the mean absolute percentage error (mape), which is specified in the scoring argument.To find the solution with the least error (i.e., the smallestmape), the resultingmapevalue must be multiplied by $-1$.frommetagen.frameworkimportSolutionfromsklearn.linear_modelimportSGDRegressorfromsklearn.model_selectionimportcross_val_scoredefregression_fitness(solution:Solution):loss=solution["loss"]# In this case, we get the builtin by getting the value property.iterations=solution["iterations"]alpha=solution["alpha"]model=SGDRegressor(loss=loss,alpha=alpha,max_iter=iterations)mape=cross_val_score(model,X,y,scoring="neg_mean_absolute_percentage_error").mean()*-1returnmapeTo conclude, theregression_domainandregression_fitnesselements are passed to theRandomSearchmetaheuristic, obtaining a hyperparameter solution for this problem by calling therunmethod.regression_solution:Solution=RandomSearch(regression_domain,regression_fitness).run()Finally, theregression_solutionis printed.print(regression_solution)See alsoDomain APISolution APIGoogle Colab Notebook ofOptimizing the hyperparameters of a regression modelResourcesGoogle Colab Notebooks:Implementing Random Search metaheuristicOptimizing f(x) = x+5Optimizing f(x) = x2Optimizing the hyperparameters of a regression modelOptimizing a Deep Learning modelCVOA (CoronaVirus Optimization Algorithm) paper