Understanding AlexNet: A Detailed Walkthrough
By: Azel Daniel
Tags: Machine Learning, Convolutional Network, Data Science, Artificial Intelligence

The 11 layers of AlexNet were:

Layer C1: Convolution Layer (96, 11×11) Layer S2: Max Pooling Layer (3×3) Layer C3: Convolution Layer (256, 5×5) Layer S4: Max Pooling Layer (3×3) Layer C5: Convolution Layer (384, 3×3) Layer C6: Convolution Layer (384, 3×3) Layer C7: Convolution Layer (256, 3×3) Layer S8: Max Pooling Layer (3×3) Layer F9: Fully-Connected Layer (4096) Layer F10: Fully-Connected Layer (4096) Layer F11: Fully-Connected Layer (1000)

Figure 6: AlexNet C1 — First Convolution Layer Diagram (image by author)

C1: First Convolution Layer

The first layer of AlexNet was a convolutional layer that accepted a (224×224×3) image tensor as its input. It performed a convolution operation using 96 (11×11) kernels with a stride of four and a padding of two. This produced a (55×55×96) output tensor that was then passed through a ReLu activation function then on to the next layer. The layer contained 34,944 trainable parameters.

Figure 7: AlexNet S2 — First Max Pooling Layer Diagram (image by author)

S2: First Max Pooling Layer

The second layer of AlexNet was a max-pooling layer that accepted output from the layer C1, a (55×55×96) tensor, as its input. It performed a zero-padded sub-sampling operation using a (3×3) kernel with a stride of two. This produced a (27×27×96) output tensor that was passed on to the next layer.

Figure 8: AlexNet C3 — Second Convolution Layer Diagram (image by author)

C3: Second Convolution Layer

The third layer of AlexNet was another convolutional layer that accepted output from the layer S2, a (27×27×96) tensor, as its input. It performed a convolution operation using 256 (5×5) kernels with a stride of one, and a padding of two. This operation produced a (27×27×256) output tensor that was then passed through a ReLu activation function, and then on to the next layer. The layer contained 614,656 trainable parameters, which summed to a total of 649,600 trainable parameters so far.

Figure 9: AlexNet S4 — Second Max Pooling Layer Diagram (image by author)

S4: Second Max Pooling Layer

The fourth layer of AlexNet was a max-pooling layer that accepted output from the layer C3, a (27×27×256) tensor, as its input. It performed a zero-padded sub-sampling operation using a (3×3) kernel with a stride of two, similar to layer S2. This produced a (13×13×256) output tensor that was then passed through a ReLU activation function, and then on to the next layer.

Figure 10: AlexNet C5 — Third Convolution Layer Diagram (image by author)

C5: Third Convolution Layer

The fifth layer of AlexNet was another convolutional layer that accepted output from the layer C5, a (13×13×256) tensor, as its input. It performed a convolution operation using 384 (3×3) kernels with a stride and padding of one. This produced a (13×13×384) output tensor that was then passed through a ReLu activation function, and then on to the next layer. The layer contained 885,120 trainable parameters, which summed to a total of 1,534,720 trainable parameters so far.

Figure 11: AlexNet C6 — Fourth Convolution Layer Diagram (image by author)

C6: Fourth Convolution Layer

The sixth layer of AlexNet was another convolutional layer that accepted output from the layer C5, a (13×13×384) tensor, as its input. It performed the same convolution operation as layer C5, which lead to the same output size. The output was also passed through a ReLu activation function. The layer contained 1,327,488 trainable parameters, which summed to a total of 2,862,208 trainable parameters so far.

Figure 12: AlexNet C7 — Fifth Convolution Layer Diagram (image by author)

C7: Fifth Convolution Layer

The seventh layer of AlexNet was another convolutional layer that accepted a (13×13×384) tensor from the layer C6 as its input. It performed a convolution operation using 256 (3×3) kernels with a stride and padding of 1. This produced a (13×13×256) output tensor. The output was also passed through a ReLu activation function. The layer contained 884,992 trainable parameters, which summed to a total of 3,747,200 trainable parameters so far.

Figure 13: AlexNet S8 — Third Max Pooling Layer Diagram (image by author)

S8: Third Max Pooling Layer

The eighth layer of AlexNet was a max pooling layer that accepted a (13×13×256) tensor from the layer C7 as its input. It performed a zero-padded sub-sampling operation using a (3×3) window region with a stride of two, similar to layer S2 and S4. This produced a (6×6×256) output tensor that was then passed through a ReLU activation function, and then on to the next layer.

Figure 14: AlexNet F9 — First Fully Connected Layer Diagram (image by author)

F9: First Fully Connected Layer

The ninth layer of AlexNet was a fully connected layer that accepted a flattened (6×6×256) tensor from the layer S8 as its input. It performed a weighted sum operation with an added bias term. This produced a (4096×1) output tensor that was then passed through a ReLu activation function, and on to the next layer. The layer contained 37,752,832 trainable parameters, which summed to a total of 41,500,032 trainable parameters so far.

Figure 15: AlexNet F10 — Second Fully Connected Layer Diagram (image by author)

F10: Second Fully Connected Layer

The tenth layer of AlexNet was another fully connected layer that accepted a (4096×1) tensor from the layer F9 as its input. It performed the same operation as layer F9 and produced the same (4096×1) output tensor that was then passed through a ReLu activation function, and then on to the next layer. The layer contained 16,781,312 trainable parameters, which summed to a total of 58,281,344 trainable parameters so far.

Figure 16: AlexNet F11 — Third Fully Connected Layer Diagram (image by author)

F11: Third Fully Connected Layer

The eleventh and final layer of the network was also a fully connected layer that accepted a (4096×1) tensor from the layer F10 as its input. It performed the same operation as layer F9 and F10 and produced a (1000×1) output tensor that was then passed through a softmax activation function. The layer contained 4,097,000 trainable parameters, which summed to a total of 62,378,344 trainable parameters overall. The output of the softmax activation function contained the predictions of the network.