Learning in the Frequency Domain
By: Nabil Madali
Tags: Computer Vision, Object Detection, Fourier Transform, Machine Learning

Traditional CNNs perform feature learning in the spatial domain of the image. Due to the limitation of video memory , the input image of CNN cannot be too large. The most common size is 224x224. The commonly used preprocessing and downsampling in CNN will be more crude to lose data information. [1] proposed a model based on DCT transformation , which aims to retain more original picture information through DCT transformation and reduce the communication bandwidth between CPU and GPU . The final experiment also proves the effectiveness of the model.

[1] proves that the input features in the frequency domain can be applied to all existing CNN models developed in the spatial domain with minimal modification. For example in ResNet-50, it only needs to delete the input CNN layer and keep the remaining residual block. The first residual layer is used as the input layer, and the number of input channels needs to be modified to fit the size of the DCT coefficient input. In this way, the modified model can maintain the parameter count and computational complexity similar to the original model.

Connecting the pre-processed input features in the frequency domain to ResNet-50. The three input layers (the dashed gray blocks) in a vanilla ResNet-50 are removed to admit the 56×56×64 DCT inputs. We take 64 channels as an example. This value can vary based on the channel selection. In learning-based channel selection, all 192 channels are analyzed for their importance to accuracy, based on which only a subset ( 192 channels) is used in the static selection approach.Source[1]

The preprocessing process is shown in the figure below.

The data pre-processing pipeline for learning in the frequency domain.Source[1]

The traditional preprocessing process and enhancement process in the same spatial field are essential, including image resizing, cropping and flipping . Then convert the image to YCbCr color space, which is the frequency domain . After that, the two-dimensional DCT coefficients of the same frequency are grouped into a channel to form a three-dimensional DCT cube. Through channel selection , a subset of the most influential channels is selected. The selected channels in the YCbCr color space are concat together to form a tensor. Finally, each channel is normalized by the mean and variance calculated from the training data set.

Since different channels of input features are at different frequencies, the author speculates that certain frequency channels are more beneficial for subsequent tasks , and deleting trivial frequency channels will not cause performance degradation. Therefore, the author proposes a learning-based channel selection mechanism. The module with this function is called the dynamic gate model, which is actually the SE-Block proposed in SE-Net .

The gate module that generates the binary decisions based on the features extracted by the SE-Block. The white color channels of Tensor 5 indicate the unselected channels.Source[1]

The specific processing flow is as follows :

First a Global Average Pool is used to form Tensor2.

1x1 convolution kernel is used to calculate Tensor3

They multiply Tensor3 through two trainable parameters to get Tensor4 with shape 1x1x2C where the two numbers in the last dimension describe the probability of being on and off for each frequency channel, respectively.

Tensor4 is point-wise multiplied to the input frequency channels to obtain Tensor 5

This module assigns a binary score to each channel. Input channels with a score of zero are separated from the network. Therefore, the size of the input data is reduced, thereby reducing the computational complexity of domain transformation and communication bandwidth requirements.

Conclusion

The proposed method of learning in the frequency domain leverages identical structures of the wellknown neural networks, such as ResNet-50, MobileNetV2, and Mask R-CNN, while accepting the frequency-domain information as the input. Experiment results show that learning in the frequency domain with static channel selection can achieve higher accuracy than the conventional spatial downsampling approach and meanwhile further reduce the input data size.

Specifically for ImageNet classification with the same input size, the proposed method achieves 1.60% and 0.63% top-1 accuracy improvements on ResNet-50 and MobileNetV2, respectively. Even with half input size, the proposed method still improves the top-1 accuracy on ResNet-50 by 1.42%. In addition, we observe a 0.8% average precision improvement on Mask R-CNN for instance segmentation on the COCO dataset.

References

1.Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, Fengbo Ren.Learning in the Frequency Domain,arXiv:2002.12416.