Meta-Modelling Meta-Learning
By: Thomas Hartmann
Tags: Meta Modeling, Automl, Artificial Intelligence, Machine Learning, Meta Learning

A Meta-Model for Machine learning

Generally, machine learning can be seen as a search problem, approximating an unknown underlying mapping function from inputs to outputs. Design choices, like the algorithm, hyperparameters and their variabilities, narrow or widen the scope of possible mapping functions, i.e. of the search space. This is similar across various machine learning algorithms (e.g. neural networks, gradient boosting, or linear regression). What distinguishes them from each other are the atomic building blocks.

If we take neural networks as an example, the building blocks of a neural network would be layers. Random forests, as another example, are composed of decision-trees.

So let’s define first a generic meta-model for any learning algorithm:

Meta-Model for Machine Learning

As it can be seen in the figure, on a high level, our learning meta-model consists of an objective, a learning algorithm, an optimizer, and data set metadata.

The objective specifies the goal of the learning algorithm. For example, if we consider the same data set containing historical prices for houses, we could perform a linear regression to predict the house prices or a classification to perform a clustering on similar houses. In other words, the objective is the loss function which defines our learning goals.

specifies the goal of the learning algorithm. For example, if we consider the same data set containing historical prices for houses, we could perform a linear regression to predict the house prices or a classification to perform a clustering on similar houses. In other words, the objective is the which defines our learning goals. The learning algorithm can be a neural network, random forest, etc., and the learning block respectively would be a layer, decision-tree, and so forth. Learning blocks can be composed to trees, e.g. to model layers in a neural network, and contain the learned state, e.g. matrices, weights, or trees.

can be a neural network, random forest, etc., and the respectively would be a layer, decision-tree, and so forth. Learning blocks can be composed to trees, e.g. to model layers in a neural network, and contain the learned state, e.g. matrices, weights, or trees. Each learning algorithm and its learning blocks respectively, have a set of hyperparameters , e.g. the number of hidden layers in case of neural networks.

, e.g. the number of hidden layers in case of neural networks. Hyperparameters as well as the learning blocks can have constraints . For instance, if parameters must be in a certain range. Another example, for a neural network layer, it must have the same number of inputs than the previous layer has outputs.

. For instance, if parameters must be in a certain range. Another example, for a neural network layer, it must have the same number of inputs than the previous layer has outputs. A learning algorithm has to specify one or several initialization methods , like uniform random distribution, gaussian distribution, which initializes the state of the learning algorithm and its learning blocks respectively. The initialization has itself hyperparameters, like the random seed or the standard deviation.

, like uniform random distribution, gaussian distribution, which initializes the state of the learning algorithm and its learning blocks respectively. The initialization has itself hyperparameters, like the random seed or the standard deviation. The optimizer optimizes the model towards the objective and is controlled by parameters, which can again have their own constraints, like the learning rate and regularization rate. Examples of optimization algorithms are gradient descent, stochastic gradient descent, or Adam.

optimizes the model towards the objective and is controlled by parameters, which can again have their own constraints, like the learning rate and regularization rate. Examples of optimization algorithms are gradient descent, stochastic gradient descent, or Adam. The data set metadata contains statistical information about the data set, like its size, dimension, etc. and its features. Features can be derived features or regular features

In an interesting article, Perez argues that an objective is also just a function and an optimizer just an algorithm and therefore both could be represented by a learning meta-model itself. Along these lines, one could argue that the optimizer can also be seen as another hyperparameter. However, it is important to note that for the same learning architecture, the accuracy, speed, and results can change significantly depending on the initialization method and optimizer hyperparameters. Therefore, we do think that it makes sense to keep the optimizer and objective as separate concepts.