Package: optunapi
API to distribute hyperparameters optimization through HTTP requestsOptunAPIis a simple API designed for Machine Learning applications that allows to distribute an automatic
hyperparameters optimization over different machines throughHTTP requests. Each set of hyperparameters
can be studied independently since the minima research does't require any gradients computation, but instead
is performed through aBayesian optimizationbased onOptuna. The machine running
Optuna manages centrally the optimization studies -- the so-called "Optuna-server" -- providing sets of
hyperparameters and assessing them by the scores evaluated and sent back by the single computing instance,
named "Trainer-client". The HTTP requests underlying such client-server system are powered byFastAPI.Key FeaturesOptunAPI inherits most of the modern functionalities of Optuna and FastAPI:Lightweight and versatileOptunAPI is entirely written in Python and has few dependencies.Easy to configureFor hyperparameters sampling, OptunAPI relies onconfiguration fileseasy to set up.Easy to integrateThe hyperparameters values can be easily recoverdecoding the HTTP response contentfrom the server.Easy parallelizationDifferent machines can run the hyperparameters study in parallel, centrally coordinated by the server.Efficient optimization algorithmsThe optimization task is headed by Optuna and its state-of-the-art algorithms.Quick visualization for study analysisTODO- OptunAPI provides a set of reports to monitor the status of the hyperparameters study.Key ComponentsTo understand how OptunAPI works, we need to spend a couple of words about its components:StudyandTrialobjects
from OptunaOptuna's Ask-and-Tell interfaceHTTP requests to map the hyperparameters spaceStudy and TrialAstudycorresponds to an optimization task, i.e., a set of trials. This object provides interfaces to run a newTrialand access trials' history. OptunAPI is designed so that, when the first machine ask for a hyperparameters set, it
starts a new study (create_study()) identified according to the HTTP request submitted. Any other machines referring
to the same optimization session don't initialize a new study, but recover the previous one (load_study()) contributing
to mapping the hyperparameters space.Atrialallows to prepare a particular set of hyperparameters and evaluate its capability of optimizing a objective
function, not necessarily available in an explicit form as in the case of very complex Machine Learning algorithms.
This object provides the following interfaces to get parameter suggestion:optuna.trial.Trial.suggest_categorical()for categorical parametersoptuna.trial.Trial.suggest_int()for integer parametersoptuna.trial.Trial.suggest_float()for floating point parametersWith optional arguments ofstepandlog, we can discretize or take the logarithm of integer and floating point parameters.
The following code block is taken from theOptuna tutorialand shows a standard use of these features:importoptunadefobjective(trial):# Categorical parameteroptimizer=trial.suggest_categorical('optimizer',['RMSprop','Adam'])# Integer parameternum_layers=trial.suggest_int('num_layers',1,3)# Integer parameter (log)num_channels=trial.suggest_int('num_channels',32,512,log=True)# Integer parameter (discretized)num_units=trial.suggest_int('num_units',10,100,step=5)# Floating point parameterdropout_rate=trial.suggest_float('dropout_rate',0.0,1.0)# Floating point parameter (log)learning_rate=trial.suggest_float('learning_rate',1e-5,1e-2,log=True)# Floating point parameter (discretized)drop_path_rate=trial.suggest_float('drop_path_rate',0.0,1.0,step=0.1)OptunAPI uses these methods internally and requires only aconfiguration filecorrectly filled to run the studies.Ask-and-Tell InterfaceThe Optuna'sAsk-and-Tellinterface provides a more flexible interface for hyperparameter optimization
based on the two following methods:optuna.study.Study.ask()creates a trial that can sample hyperparametersoptuna.study.Study.tell()finishes the trial by passingtrialand an objective valueOptunAPI uses these methods in two different moments. When a machine ask for a set of hyperparameters,
that set belongs to a trial resulting from anaskinstance. Then, once the objective function was
evaluated with that particular set of hyperparameters, the machine sends a new request encoding the
objective value allowing to close the corresponding trial with atellinstance.HTTP RequestsOptunAPI provides a simple Python module to run a server able to centrally manage the optimization studies:optuna/optuna/server.py. It is
equipped with a set ofpath operation functionsrelying on the FastAPI ecosystem:ping_serverthepathis/optunapi/pingtheoperationisGETthefunctionallows to verify if the server is runningread_hparamsthepathis/optuna/hparams/{model_name}(model_nameis apath parameter)theoperationisGETthefunctionallows to start (or load) an Optuna study and send sets of hyperparameterssend_scorethepathis/optuna/score/{model_name}?trail_id=TRIAL_ID&score=SCORE(withquery parameters)theoperationisGETthefunctionallows to finish the trial identified bytrial_idwith thescorevalueRequirementsPython 3.6+OptunAPI is based on two modern and highly performant frameworks:Optunafor the optimization parts.FastAPIfor the HTTP requests parts.InstallationOptunAPI is apublic repositoryon GitHub.$gitclonehttps://github.com/mbarbetti/optunapi.git---> 100%To run and use OptunAPI it's preferable to create a virtual environment with Python 3.6+ and install Optuna and FastAPI within it.$pipinstalloptunafastapi---> 100%Standing on the shoulder of FastAPI, OptunAPI needs an ASGI server to run the so-called Optuna-server,
such asUvicornorHypercorn.$pipinstalluvicorn[standard]---> 100%ExampleConfiguration fileThe high-level functions provided by Optunato suggest values for the hyperparametersare replaced with an appropriateconfiguration filein OptunAPI. Referring to the example reported in
theOptuna tutorial,
what follows is the corresponding YAML configuration file:# Categorical parameteroptimizer:name:optimizertype:categoricalchoices:-RMSprop-Adam# Integer parameternum_layers:name:num_layerstype:intlow:1high:3# Integer parameter (log)num_channels:name:num_channelstype:intlow:32high:52log:True# Integer parameter (discretized)num_units:name:num_unitstype:intlow:10high:100step:5# Floating point parameterdropout_rate:name:dropout_ratetype:floatlow:0.0high:1.0# Floating point parameter (log)learning_rate:name:learning_raletype:floatlow:1e-5high:1e-2log:True# Floating point parameter (discretized)drop_path_rate:name:drop_path_ratetype:floatlow:0.0high:1.0step:0.1Optuna-serverPrepared the configuration file for the optimization session and saved it intooptunapi/optunapi/config,
we are ready to run the Optuna-server.$uvicornserver:optunapiINFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)INFO:     Started reloader process [28720]INFO:     Started server process [28722]INFO:     Waiting for application startup.INFO:     Application startup complete.What does the commanduvicorn server:optunapimean?The commanduvicorn server:optunapirefers to:server: the fileserver.py(the Python "module") inoptunapi/optunapi.optunapi: the object created inside ofserver.pywith the lineoptunapi = FastAPI().Note that Uvicorn sets127.0.0.1and8000as default values for the server IP and port.
To change the defaults it's enough launching the previous command with the arguments--hostand--portfollowed by the chosen values.Trainer-clientThe optimization session is managed by an Optunastudy, initialized with the first client HTTP request,
or loaded and expanded by any other connecting machines. To refer to a particular optimization session a
client has to encode the name of the corresponding configuration file within its HTTP request.Consider the simple use-case provided by OptunAPI, where we want to find the minimum of a 2D-paraboloid:optunapi/tests/simple_client.py.
Since the provided configuration file is namedoptuna-test.yaml, then the GET request submitted by the client
to receive the hyperparameters set has to contain the string'optuna-test':importrequestsHOST='http://127.0.0.1:8000'read_hparams=requests.get(HOST+'/optunapi/hparams/optunapi-test')hp_req=read_hparams.json()TRIAL_ID=hp_req['trial_id']PARAMS=hp_req['params']What happens behind the scenes is that the above HTTP request calls anaskinstance to the Optunastudy, stored inoptunapi/optunapi/dbonce created and namedoptunapi-test.db. As already said, anaskinstance is atrialequipped with
a set of hyperparameters and the client can recover those values decoding the corresponding HTTP response.
In the example above,hp_reqis a dictionary containing, among others, the identifier number of the currenttrial(TRIAL_ID) and a dictionary for the hyperparameters values (PARAMS).Having accessed to the hyperparameters values, we can perform whatever learning algorithm one prefers and
evaluate the associated training score, that will be used asobjective valueto finish thetrialinstance.
This is done with a new GET request referring to the same optimization session (again,'optunapi-test'in the path)
and passingTRIAL_IDandSCOREas query parameters:importrequestsHOST='http://127.0.0.1:8000'send_score=requests.get(HOST+'/optunapi/score/optunapi-test?trial_id=TRIAL_ID&score=SCORE')score_req=send_score.json()BEST_TRIAL_ID=score_req['best_score_id']BEST_PARAMS=score_req['best_params']Each running client allows to refine the search for minima performed by the Optuna algorithms, focusing
on smaller and smaller space portion and enhancing the mapping of the hyperparameters space.Securing HTTP requestsOptunAPI is designed to be used within a VPN not directly opened to the public Internet. On the other hand,
opening the Optuna-server to Internet allows to exploit easily a wide variety of computing resources, from
on-premises machines to instances deriving from different cloud computing services (AWS, Azure, GCP, etc.).
Such design raises a security issue since anyone can submit a request to the server or catch its response,
opening the system to cyberattack.A possible solution to this issue relies on the SSH protocol. The idea is to set up the Optuna-server
as aprivate server(from the perspective ofREMOTE SERVER) not directly visible from the outside
(LOCAL CLIENT’s perspective). This configuration, schematically represented in the sketch below,
allows alocal clientto still access theprivate serverpassing through theremote serverauthenticating with SSH credentials.----------------------------------------------------------------------

                                |
    -------------+              |    +----------+               +---------
        LOCAL    |              |    |  REMOTE  |               | PRIVATE
        CLIENT   | <== SSH ========> |  SERVER  | <== local ==> | SERVER
    -------------+              |    +----------+               +---------
                                |
                             FIREWALL (only port 22 is open)

    ----------------------------------------------------------------------OptunAPI provides a very simple implementation of this scheme:optunapi/tests/secured_client.py.
It is based onsshtunneland allows to submit a HTTP request to theprivate serverafter having specifying our SSH credentials (ssh_username,ssh_pkey).importsshtunnelimportrequestswithsshtunnel.open_tunnel((REMOTE_SERVER_IP,22),ssh_username='mbarbetti',ssh_pkey='/home/mbarbetti/.ssh/id_rsa',remote_bind_address=(PRIVATE_SERVER_IP,PRIVATE_SERVER_PORT),local_bind_address=('127.0.0.1',10022))astunnel:ping_server=requests.get('http://localhost:10022/optunapi/ping')ping_msg=ping_server.json()print(ping_msg)How to run the server in this case?In this configuration the Optuna-server acts asprivate server,
then its IP and port are the ones declared within thewithstatement:$ uvicorn server:optunapi --host PRIVATE_SERVER_IP --port PRIVATE_SERVER_PORTLicenseThis project is licensed under the terms of the MIT license.