MobileNetV2: Inverted Residuals and Linear Bottlenecks | by Paul-Louis Pröve | Towards Data Science

In April 2017 a group of researchers from Google published a paper which introduced a neural network architecture that was optimized for mobile devices. They strived for a model that delivered high accuracy while keeping the parameters and mathematical operations as low as possible. This was much needed in order to bring deep neural networks to smartphones.
The architecture dubbed MobileNet revolves around the idea of using depthwise separable convolutions, which consist of a depthwise and a pointwise convolution after one another. If you’re a little fuzzy on the details of this operation feel free to check out my other article that explains this concept in detail. MobileNetV2 extends its predecessor with 2 main ideas.
Residual blocks connect the beginning and end of a convolutional block with a skip connection. By adding these two states the network has the opportunity of accessing earlier activations that weren’t modified in the convolutional block. This approach turned out to be essential in order to build networks of great depth.
When looking a little closer at the skip connection we notice that an original residual block follows a wide->narrow->wide approach concerning the number of channels. The input has a high number of channels, which are compressed with an inexpensive 1x1 convolution. That way the following 3x3 convolution has far fewer parameters. In order to add input and output in the end the number of channels is increased again using another 1x1 convolution. In Keras it would look like this:
def residual_block(x, squeeze=16, expand=64):  m = Conv2D(squeeze, (1,1), activation='relu')(x)  m = Conv2D(squeeze, (3,3), activation='relu')(m)  m = Conv2D(expand, (1,1), activation='relu')(m)  return Add()([m, x])
On the other hand, MobileNetV2 follows a narrow->wide->narrow approach. The first step widens the network using a 1x1 convolution because the following 3x3 depthwise convolution already greatly reduces the number of parameters. Afterwards another 1x1 convolution squeezes the network in order to match the initial number of channels.
In Keras it would look like this:
def inverted_residual_block(x, expand=64, squeeze=16):  m = Conv2D(expand, (1,1), activation='relu')(x)  m = DepthwiseConv2D((3,3), activation='relu')(m)  m = Conv2D(squeeze, (1,1), activation='relu')(m)  return Add()([m, x])
The authors describe this idea as an inverted residual block because skip connections exist between narrow parts of the network which is opposite of how an original residual connection works. When you run the two snippets above you will notice that the inverted block has far fewer parameters.
The reason we use non-linear activation functions in neural networks is that multiple matrix multiplications cannot be reduced to a single numerical operation. It allows us to build neural networks that have multiple layers. At the same time the activation function ReLU, which is commonly used in neural networks, discards values that are smaller than 0. This loss of information can be tackled by increasing the number of channels in order to increase the capacity of the network.
With inverted residual blocks we do the opposite and squeeze the layers where the skip connections are linked. This hurts the performance of the network. The authors introduced the idea of a linear bottleneck where the last convolution of a residual block has a linear output before it’s added to the initial activations. Putting this into code is super simple as we simply discard the last activation function of the convolutional block:
def inverted_linear_residual_block(x, expand=64, squeeze=16):  m = Conv2D(expand, (1,1), activation='relu')(x)  m = DepthwiseConv2D((3,3),  activation='relu')(m)  m = Conv2D(squeeze, (1,1))(m)  return Add()([m, x])
The snippet above shows the structure of a convolutional block that incorporates inverted residuals and linear bottlenecks. If you want to match MobileNetV2 as closely as possible there are two other pieces you need. The first aspect simply adds Batch Normalization behind every convolutional layer as you’re probably used to by now anyways.
The second addition is not quite as common. The authors use ReLU6 instead of ReLU, which limits the value of activations to a maximum of...well...6. The activation is linear as long as it’s between 0 and 6.
def relu(x):  return max(0, x)def relu6(x):  return min(max(0, x), 6)
This is helpful when you’re dealing with fixed point inference. It limits the information left of the decimal point to 3 bits, meaning we have a guaranteed precision right of the decimal point. This was also used in the orignal MobileNet paper. The final building block looks like this:
def bottleneck_block(x, expand=64, squeeze=16):  m = Conv2D(expand, (1,1))(x)  m = BatchNormalization()(m)  m = Activation('relu6')(m)  m = DepthwiseConv2D((3,3))(m)  m = BatchNormalization()(m)  m = Activation('relu6')(m)  m = Conv2D(squeeze, (1,1))(m)  m = BatchNormalization()(m)  return Add()([m, x])
Now that we understand the building block of MobileNetV2 we can take a look at the entire architecture. In the table you can see how the bottleneck blocks are arranged. t stands for expansion rate of the channels. As you can see they used a factor of 6 opposed to the 4 in our example. c represents the number of input channels and n how often the block is repeated. Lastly s tells us whether the first repetition of a block used a stride of 2 for the downsampling process. All in all it’s a very simple and common assembly of convolutional blocks.
Something that I’m particuarly happy about is the fact that MobileNetV2 provides a similar parameter efficiency to NASNet. NASNet is the current state of the art on several image recognition tasks. It’s building blocks are quite complex making it rather unintuitive why it works so well. The building block of NASNet were not designed by humans but by another neural network. Introducing a simple architecture like MobileNetV2 that shows a comparable efficiency makes me a little more confident that the next big architecture might be designed by a human being as well.