Package: meta-ml
META LearnMetaRL-basedEstimator usingTask-encodings forAutomated machineLearningMETA Learnis a deep learning approach to automated machine learning that
parameterizes the API of machine learning software as a sequence of actions to
select the hyperparameters of a machine learning pipeline in an end-to-end
fashion, from raw data representation, imputation, normalizing, feature
representation, and classification/regression. Currently thesklearn APIis the only supported ML framework.Why?As the diversity of data and machine learning use cases increases, we need
to accelerate and scale the process of training performant machine learning
systems. We'll need tools that are adaptable to specific problem domains,
datasets, and the (sometimes non-differentiable) performance metrics that we're
trying to optimize. Supervised learning of classification and regression tasks
given a task distribution of small to medium datasets provides an promising
jumping off platform for programmatically generating a reinforcement learning
environment for automated machine learning (AutoML).Installationinstallmetalearnlibrary:pip install -e .then you can run an experiment with themetalearncli.# run an experiment with default values
$ metalearn run experimentRunning an experiment with a configuration fileAlternatively, you can create an experiment configuration file to run
your experiment.# create experiment config file
$ metalearn create config my_experiment config/local --description "my experiment"

# output:
# wrote experiment config file to config/local/experiment_2018-37-25-21:37:11_my_experiment.ymledit the config fileparameterssection to the set of parameters
that you want to train on, then run the experiment with$ metalearn run from-config config/local/experiment_2018-37-25-21:37:11_my_experiment.ymlrun an experiment using floydhub:$ floyd_scripts/train-metalearn.sh path/to/config/file 'optional job run message'Relevant WorkThe Combined Algorithm Selection and Hyperparameter optimization
(CASH) problem is an important one to solve if we want to
effectively scale and deploy machine learning systems in real-world use cases,
which often deals with small (< 10 gb) to medium size (10 - 100 gb) data.CASH is the problem of searching through the space of all ML frameworks,
defined as an AlgorithmAand a set of relevant hyperparameterslambdaand proposing a set of models that will perform well given a dataset and
a task.In order to solve this problem, previous work likeautosklearnuses a Bayesian Optimization techniquesSMACwith an offline meta-
learning "warm-start" step using euclidean distance to reduce the search space
of ML frameworks. This meta-learning step was done by representing the datasets
with metadata features (e.g. number of features, skew, mean, variance, etc.) to
learn representations of the data space that perform well with respect to the ML
framework selection task.Neural Architecture Searchis another approach to the CASH
problem, where a Controller network proposes "child" neural net architectures
that are trained on a training set and evaluated on a validation set, using the
validation performanceRas a reinforcement learning reward signal to learn
the best architecture proposal policy.ContributionsThe contributions of the META Learn project are two-fold: it builds on the neural
architecture search paradigm by formalating the output space of the Controller
as a sequence of tokens conditioned on the space of possible executableframeworks. The scope of this project is to define aframework, expressed
as a set of hyperparameters, that can be evaluated by a machine learning
framework, like sklearn, which evaluates to an instantiated sklearnPipeline. Once defined, it can be fitted on a training
set and evaluated on a validation set of a particular datasetD.Following the Neural Architecture scheme, META Learn uses the REINFORCE algorithm
to compute the policy gradient used to update the Controller in order to learn a
policy for proposing goodframeworksthat are able to achieve high validation
set performance.The second contribution of this project is that it proposes a conditional
MLframeworkgenerator by extending the Controller network to have anencodernetwork that takes as input metadata about the datasetD(e.g. number of
instances, number of features). The output of theencodernetwork would be
fed into thedecodernetwork, which proposes an MLframework. Therefore,
we can condition the output of thedecodernetwork metadata onDto proposeframeworksthat depend on the encoded dataset representation.Training AlgorithmThe environment is a distribution ofksupervised learning tasks, consisting
of a pair(X, y)of features and targets, respectively. At the beginning of
an episode, the environment samples one task and foriiterations produces
sample splits(X_train, y_train, X_validation, y_validation)drawn from the
task dataset.TheMetaLearnControllerreceives the current task statesvia metafeatures
associated the task, e.g.# of training samples,# of features,target type,#of continuous features,#of categorical features, etc.Given the task state, the controller generates MLframeworksover a state
space of algorithms and hyperparameter values. The controller can be viewed as a
policy approximator, which selects actions based on some pre-definedAlgorithmSpace, representated as a direct acyclic graph where each node
contains a set of hyperparameter values to choose from. The controller traverses
this graph via a sequential decoder by selecting hyperparameters via softmax
classifiers, where certain actions may remove certain edges from the graph.
This enforces incompatible hyperparameter configurations.For example, the algorithm space for a possiblesklearnpipeline would
consist of the following components:categorical encoder (e.g. OneHotEncoder)categorical encoder hyperparametersimputer (e.g. SimpleImputer)imputer hyperparametersrescaler (e.g. StandardScaler)rescaler hyperparametersfeature processor (e.g. PCA)feature processor hyperparametersclassifier/regressor (e.g. LogisticRegression, LinearRegression)classifier/regressor hyperparametersWhen the controller reaches a terminal node in algorithm space, the environment
evalutes the selected MLframeworkand produces a validation score that the
controller uses as a reward signal. Validation performance is calibrated such
that a better score produces higher rewards. Using the REINFORCE policy
gradient method, the controller tries to find the optimal policy that
maximizes validation performance over the task distribution.AnalysesThe./analysissubfolder contains jupyter notebooks that visualize the
performance of the cash controller over time. Currently there are 5 analyses
in the projectanalysissubfolder:20180715_metalearn_controller_analysis.ipynb: a basic interactive analysis
of a single job's outputs.20180809_experiments_jobs_150_151.ipynb: analyzes baseline performance
based on correct action selection implementation. For more info on the bug,
seehere.20180902_experiments_entropy_jobs_166-175.ipynb: analyzes tuning experiment
after entropy coefficient was introduced in the loss function.20180930_experiments_regression_195.ipynb: analyzes a pure regression
metalearn controller.20181003_experiments_regression_197.ipynb: analyzes a pure regression
controller with two sets of entropy coef settings.20191111_metalearn_openml_cc18_benchmark.ipynb: evaluate the metalearn
controller's ability to learn from an internal RL algorithm implemented within
the RNN loop.