Hierarchical-Split Block on Convolutional Neural Network
By: Nabil Madali
Tags: Computer Vision, Object Detection, Cnn, Machine Learning

​ Multi-scale features are very important for a large number of visual tasks. Many existing network structure improvements have considered the components of multi-scale information. [1] proposes a plug and play”Hierarchical-Split Block (HSB) to improve the performance of existing CNNs. HSB contains multiple Split and Concat operations, which together constitute the multi-scale feature extraction of the block; at the same time, HSB has better flexibility and efficiency.

Comparing the accuracy-latency of different improved versions of ResNet50 models, the circle area reflects the params size of different models, latency test on T4/FP32 and batchsize=1. The orange circles represent different improvement strategies to improve the ResNet model, the green circles represent the use of different attention strategies, and the red circles are the HS-ResNet we designed. In order to make the indicators more intuitive, we added ResNet101 and ResNet152 represented by blue circles.Source[1]

ResNet based on HSB components has achieved great performance improvements on multiple tasks.For example, on the ImageNet dataset, HS-ResNet50 achieved 81.28% of Top1 accuracy, which exceeds the previous ResNeSt proposed by Amazon. The following figure shows the accuracy and time-consuming comparison of different ResNets.

The objects in a picture may have different sizes, for example, sofas and cups are of different sizes, the necessary context information may occupy an area larger than the object itself. For example, we need to better determine whether a cup or a pen holder is on the table based on the information on the big table,for fine-precision classification and semantic segmentation, it is necessary to understand the part and observe information at different scales.

Alexnet stacks convolutions in order and obtains significant results compared to traditional methods. However, due to the limitation of network depth and convolution kernel size, alexnet has only a small receptive field.

stacks convolutions in order and obtains significant results compared to traditional methods. However, due to the limitation of network depth and convolution kernel size, alexnet has only a small receptive field. VGGnet increases the network depth and uses a smaller convolution kernel. A deeper network structure can expand the receptive field and extract features from a larger scale. By stacking more layers of large convolution kernels, it is easier to expand the receptive field. VGG has fewer parameters and stronger expressive ability than Alexnet. But they are all linear stacked convolutions, which can only express inflexible receptive fields.

increases the network depth and uses a smaller convolution kernel. A deeper network structure can expand the receptive field and extract features from a larger scale. By stacking more layers of large convolution kernels, it is easier to expand the receptive field. VGG has fewer parameters and stronger expressive ability than Alexnet. But they are all linear stacked convolutions, which can only express inflexible receptive fields. Googlenet uses parallel convolutions of different scales to enhance multi-scale expression capabilities. But it is limited by computing resources. Therefore, the strategy of multi-scale representation still fails to have a wider range of receptive fields.

uses parallel convolutions of different scales to enhance multi-scale expression capabilities. But it is limited by computing resources. Therefore, the strategy of multi-scale representation still fails to have a wider range of receptive fields. Resnet introduces short connections, so it has a deeper network structure while alleviating the problem of gradient disappearance. In the process of feature extraction, short connections allow different convolutions to be combined. Similarly, dense connections in densenet also enable the network to handle a wide range of scales.

[1] was inspired by res2net and Ghostnet network architecture,

Res2Net propose a novel building block for CNNs, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer.

Comparison between the bottleneck block and the proposed Res2Net module (the scale dimension s = 4).

A popular understanding is to replace the 3x3 convolution in the middle of the original resnet with the part on the right, which is at least directly connected without 3x3 convolution. It will go through at most 3 3x3 convolutions, so that you can understand the reason why there are more receptive fields than the original structure.

After 1x1 convolution, the feature map is divided into 4 parts. The first part of the circuit is very simple, x1 is not processed and passed directly to y1; the second part of the circuit, x2 is divided into two circuits after 3x3 convolution, one continues to propagate forward to y2, the other to x3, so the third one route gets the information of the second route,the third route, the fourth route, and so on.

An illustration of the convolutional layer and the proposed Ghost module for outputting the same number of feature maps. Φ represents the cheap operation.

GhostNet’s design idea is to reduce the redundancy of feature maps, and some feature maps can be obtained through convolution operations on existing feature maps . Therefore, it first obtains the yellow part through a convolution module, and then further generates a feature map in the yellow part, and finally the two are spliced ​​together.

Network architecture

The figure above shows the network structure diagram of the HSB model ,

A detailed view of Hierarchical-Split block with s=5. Split means equally splitting in the channel dimension. Conv means 3 × 3 standard convolution + Batch normalization+ ReLU. Concat means concatenating features in the channel dimension. This block only replaces the 3 × 3 convolution + Batch normalization + ReLU operation in the bottleneck of ResNet. The proposed Hierarchical-Split block can be taken as a plug-and-play component to upgrade existing convolutional neural networks.Source[1]

After the 1 × 1 convolution, they split the feature maps into s groups, denoted by xi , each xi will be fed into 3 × 3 convolutions, the output feature maps are denoted by yi ,the yi is split into two sub-groups, denoted y_{i,1} and y_{i,2}.

The y_{i,2} is concatenated with next group x_{i+1}, and then fed into a convulution operation,when finish operating all input feature groups, the channel dimension will be recovered by concatenating all yi,1, which each yi,1 has different channels.

The above HSB can be described as follows:

HSB contains two key operations Split and Concatenate:

Split is used to group features, and each group has the same number of channels.

Concatenate is used to fuse different features and enhance the information interaction of different groups.

Conclusion

[1] proposes a novel HSB module, which can extract multi-scale features efficiently. At the same time, the author builds ResNet based on HSB to improve HS-ResNet, which is used in multiple vision tasks such as image classification, target detection, SOTA performance is achieved in instance segmentation, semantic segmentation, etc.

The proposed HSB has a plug and play feature, it can be easily embedded in the existing network and improve performance, it is worth trying for all students.

References

1.Pengcheng Yuan, Shufei Lin, Cheng Cui, Yuning Du, Ruoyu Guo, Dongliang He, Errui Ding, Shumin Han.HS-ResNet: Hierarchical-Split Block on Convolutional Neural Network