Ensembles of Deep Networks. This article discusses the Hydra... | by Connor Shorten | Towards Data Science

Ensembles of models refers to the practice of combining predictions from multiple statistical models to form one final prediction. This opens up opportunity for diversity in the representational capacity of the model. This concept is analogous to anecdotes such as seeking the opinion of multiple doctors. Ensembles of models are a very common enhancement to traditional machine learning models, such as the upgrade of decision trees to random forests. In contrast to machine learning models, deep models take a long time to train and therefore it is not practical to form ensembles of deep models each trained from scratch. This article covers the novelties of a CNN ensemble design from Minetto et al. named “Hydra”. This design saves computation time by using a shared feature extractor network for models in the ensemble. Following is a link to the paper:
arxiv.org
Very quickly, the picture below summarizes the main idea of the Hydra:
Deep networks can be modularized and thought of as consisting of two components, a feature extractor and a classification network. In this image above, two powerful deep CNN architectures, ResNet and DenseNet, are used to extract features from images which are passed into multiple classification ‘heads’. Extracting features will result in lowering the dimensionality of the input from something like a 224 x 224 x 3 image into a 4096 x 1 vector. The vectors extracted from the Hydra’s body are used as input to the Hydra’s heads, thus 4096 x 1 vector into a vector of class predictions as big as the number of classes.
Training is done to fine-tune the weights in the Hydra’s heads such that they can be adapted for the task at hand. Another interesting detail of the Image above is the ImageNet weights. This is used as a Transfer Learning initialization approach for the weights of the Hydra’s body. These weights are then fine-tuned using the FMoW (Functional Map of the World) land classification dataset which Minetto et al. are training their network for. After fine-tuning these weights, the fully connected layers are chopped off and replaced by the Hydra’s heads. Hopefully after this explanation you can see why they have chosen to name it Hydra. A Hydra is some kind of multi-headed dragon and similar to the multi-headed dragon, each head of the ensembled convolutional neural network shares the same body.
Some of the interesting characteristics of doing this is that it promotes functional and data diversity in the model. Functional diversity can be achieved with adding novelties to the fully-connected layers in each Hydra head, perhaps even reshaping the feature extracted vector and convolving over it again. However, achieving data diversity is a more interesting paradigm that is also investigated in this paper.
Ensemble models usually achieve data diversity by partitioning the initial dataset, with say m instances, into k buckets each with m / k instances. The data from each of these buckets is used to fine-tune the weights on the Hydra’s heads. This approach can also be extended to incorporate augmented data and has many possibilities of construction. Data Augmentation techniques such as geometric transformations or color space manipulations can be used to create additionally diverse buckets of data to fine-tune the Hydra’s heads. The authors of the paper discuss the importance of using on-line data augmentation to save memory, as opposed to off-line augmentation. Off-line augmentation refers to augmenting the data and then writing each modified data instance to memory.
One final consideration to make when ensembling models in this way, is how to combine the prediction of the models. Naive solutions can range from majority voting where the output comes from the max class probability from each head or majority voting where the softmax outputs from each head are summed up and the max value from the resulting vector is the final prediction. This is referred to as the solution algorithm of the ensemble model. Their are many ways to be creative with this and test out different hypotheses, such as assigning higher weights to the Hydra’s heads which have the best classification performance on their own, or searching for a subset of heads which have the least correlated predictions.
Thank you for reading, I thought the paper was very interesting and recommend checking it out! Ensembles of models suffer from slower inference time, but are a promising solution for higher prediction accuracies. The Hydra architecture is an interesting and well-named method for ensembling Deep Neural Networks.
I am currently working on implementing the Hydra network in Keras and would like to share one detail I have come across that helped me get started. First, you need to train the weights of the feature extractor and save them using something simple like:
model.save('HydraBody.h5')
Then you will want to load these weights into each of the heads, this is facilitated by naming each layer of the HydraBody such as:
model.add(Conv2D(32, (3,3), name = 'Conv1')
Naming each of the layers is how you load the base weights into a model that doesn’t exactly match the architecture of the saved weights.
Thanks for reading!