Dynamically Adjusting Convolutional Neural Networks
By: Alex Gaskell
Tags: Machine Learning, Artificial Intelligence, Convolutional Network, Computer Vision

A neat hack to simplify working with CNNs with different input sizes!

Convolutional Neural Networks (CNNs) [1] are ubiquitous in the machine learning (ML) community and a key weapon in any ML researcher’s armoury.

However, working with CNN’s can be a headache when you are unsure of the size of your input (normally an image). This is the case when you are deciding on the resolution of an image or are working with images of different sizes which need cropping/padding. The problem occurs because the fully-connected layers require a fixed number of neurons which is a function of the input size (alongside the dimensions and number of convolutions used).

For example, I was recently working on a project detecting COVID-19 using audio clips of people breathing and coughing. Following audio processing convention, I first converted these audio files into an image called the log spectrogram which shows the sound wave frequencies against time of an audio clip (see Fig.1 for an example). However, the audio clips had different lengths so I had to crop and/or pad some samples, meaning the duration of audio clip effectively became a model hyper-parameter. To avoid considerable headaches and much manual fiddling, I devised a simple method to dynamically adjust the dimensions of my CNN. Read on for details plus a demo implementation!

Why can’t CNNs handle different input sizes?

Fig. 1: CNN schematic

Fig. 1 illustrates the issue with using CNNs for different input sizes. Using the COVID audio detection example above, if we use a 2s audio clip and pass it through the convolutional section of our CNN then our output is a vector of length 2. This forces the first layer of our fully connected component to have 2 neurons. If instead we use an 8s clip, this gives an 8x1 output, requiring 8 neurons in the first dense layer. Adjusting the input size therefore usually requires manual fiddling of the dimensions of the dense layers¹.

Dynamically-adjusting solution

An elegant solution is to detach the convolutional component from the dense component of your network. When instantiating your network, take a sample of input and forward pass it through the convolutional component only. The output here can then be used to obtain the number of neurons required in the first fully connected layer, so the fully connected component can be correctly instantiated. See the demo below:

Fig. 2: Naive implementation of CNN with hard-coded dense dimensions

Fig. 2 shows the naive CNN implementation. This simple network is configured for images of size 14x14. After passing through the convolutional layers, the output vector is 48 length. However, if we try the network with images of a different size, say 28x28, the network will fail during the forward pass. Line 9 is the root cause as it is hard-coded to have 48 input neurons but this will only work for 14x14 sized images. The solution it to make line 9 dynamic so it adjusts corresponding to the size of the input.

Fig.3: dynamic implementation which automatically adjusts its dimensions depending on the input size

Fig. 3 shows the dynamic implementation. My version separates the convolutional and dense components into parent and child classes. This method first instantiates the convolutional component so we can then run a forward pass on this section of the network and use the output dimensionality to derive the dimensions required in the the fully connected layer in the DynamicCNN . This is implemented via a helper method: get_dense_dim .

Using this method we can use differently sized images (e.g. 14x14 or 28x28) without having to adjust the network configuration. This will allow you to use different image sizes in your hyper-parameter sweep, thereby helping you to train better models.

This method should be used wisely, however, and probably only for relatively small changes in input size. The complexity of a fully connected layer is linear with respect to the input neurons, therefore increasing image size will slow down your model and make it larger. In addition, it might be sensible to adjust the dimensions of subsequent linear layers as otherwise increasing the input size will create a single very wide linear layer, resulting in a bottleneck in the network. An alternative solution to mine proposed here is to use PyTorch’s AdaptiveAveragePooling layer, with further details on this here.

I hope this tip has been of interest/use. As always, any thoughts/comments/feedback will be gratefully received. Thanks for reading and happy machine learning!