Research on adaptive local feature enhancement in convolutional neural networks

IET Image ProcessingVolume 14, Issue 16 p. 4306-4315 Research ArticleFree Access Research on adaptive local feature enhancement in convolutional neural networks Tongfeng Sun, Corresponding Author Tongfeng Sun suntf@cumt.edu.cn School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorChanglong Shao, Changlong Shao School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorHongmei Liao, Hongmei Liao School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorShifei Ding, Shifei Ding School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorXinzheng Xu, Xinzheng Xu School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this author Tongfeng Sun, Corresponding Author Tongfeng Sun suntf@cumt.edu.cn School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorChanglong Shao, Changlong Shao School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorHongmei Liao, Hongmei Liao School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorShifei Ding, Shifei Ding School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this authorXinzheng Xu, Xinzheng Xu School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu, People's Republic of China Mine Digitization Engineering Research Center of Ministry of Education of the People's Republic of China, Xuzhou, Jiangsu, People's Republic of ChinaSearch for more papers by this author First published: 23 February 2021 https://doi.org/10.1049/iet-ipr.2020.0591Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Local feature extraction is one of the key characteristics of convolutional neural networks (CNNs). This study proposes an adaptive local feature enhancement (ALFE) model with a low-frequency general appearance-enhancement operator and a high-frequency local detail enhancement operator to improve local features of CNNs. Through supervised training, the model could adaptively adjust enhancement parameters and achieve a global-local enhancement of training images and CNNs. The performance of ALFE was first preliminarily evaluated with a self-built CNN on CIFAR-10 data set in different conditions of image augmentation and feature pooling. CNNs with ALFE could increase the top-1 accuracy compared against CNNs in the same conditions, and nearly reach the same level of performance for different pooling approaches. With only two extra adjustable parameters, this model could effectively avoid overfitting, without affecting the convergence speed of CNN. In addition, the extra burden of network complexity could be neglected. Further, experiments of three existing popular CNNs (AlexNet, VGGNet and ResNet) with ALFE were carried out on dogs versus cats, Tiny ImageNet, and SVHN data sets, respectively. The results show that ALFE is feasible for the existing popular CNN models, improving their top-1 accuracies without changing their convergence speeds. 1 Introduction Convolutional neural networks (CNNs) have evolved as the main workhorse in solving computer vision tasks, including image classification [[1]], object recognition [[2]], image reconstruction [[3]], video tracking [[4]], etc. Over the years, many CNN models have been proposed, such as LeNet [[5]], AlexNet [[6]], GoogleNet [[7]], VGGNet [[8]] and ResNet [[9]]. Generally speaking, CNNs are mainly constructed by stacking interweaved layers of two types: convolutional layer and pooling layer. CNN models are largely consistent with the model of the mammal visual cortex proposed by Hubel and Wiesel [[10]]. Convolutional layers play a role of local feature extraction via a convolution operation, imitating human local receptive fields. Pooling layers reduce the dimension of input image data, increasing the generalisation of CNN. Local feature extraction and weight sharing are two key characteristics of CNNs. Local features are vital to the performance of CNNs. Therefore, the investigation of local feature processing, such as local feature enhancement, is beneficial to CNNs. Although CNNs can improve local features through adjusting convolution parameters, it is difficult to push convolution operators to enhancement operators and fit image enhancement by convolution layers because the adjustments of network parameters are uncontrollable subjectively. A local feature enhancement has always been a study focus on image enhancement. Image enhancement, in the traditional sense, is to enhance the edges, textures and other details of images so as to improve their visual effects [[11]-[13]]. Generally speaking, its implementation is based on images themselves, independent of other aspects of image processing, such as object segmentation, image classification, image reconstruction. Local features in CNNs are subject to image preprocessing, CNN topology, feature pooling, etc. Therefore, traditional image enhancement cannot solve the problem of local feature enhancement in CNNs. The local feature enhancement in CNNs has its own uniqueness. There are mainly two ways to enhance local features in CNNs: improving feature pooling and adding feature filters. Researchers have proposed many feature pooling methods, such as max pooling, and average pooling. Feature pooling is equivalent to a combination of a convolution filter operation and a subsampling operation. Max pooling plays an enhancement role, while average pooling plays a smoothing role. However, the main functions of feature pooling are to reduce dimensions and keep CNN invariant to data variation and perturbation rotation. Its enhancement effects on local features are limited. No pooling method can always keep superiority among various pooling methods on different data sets. In fact, local feature enhancement has not been explicitly discussed in many works of literature on feature pooling. There are few works on feature enhancement filters in CNNs. The research achievements mainly involve feature enhancement in specific fields, such as the recognition of character [[14]], fingerprint [[15]] and medical images [[16]-[18]], thus lacking wide adaptability. The feature enhancement in the works actually belongs to the category of image enhancement, which focuses on image enhancement based on the images themselves, without taking CNN into account. So far, there is still a lack of systematic research on feature enhancement in CNNs. It is difficult to design a local feature enhancement with high applicability in CNNs in that local features in CNN are subject to many factors, which are presented as follows: (i) Dataset: The images in different datasets may be greatly different in local features, and so are the contents and quality of the images in the datasets. Different data sets need different image enhancements. (ii) Data preprocessing: Data augmentation operations, including random rotation, flipping, cropping, and adjustment of saturation, brightness and contrast, etc. [[19], [20]], and some data normalisations [[21]] are often applied to training images as preprocessing approaches to improve the robustness of CNNs before the images are input into CNNs. Especially, data augmentation plays an important role in the training of CNNs. These operations all affect local features. (iii) CNN model and topology: CNNs have various models and topologies, whose influences on local features are unknown. And, convolution layers may contain different components, such as LRN [[6]], BN [[22]] which may also affect local features. (iv) Feature pooling: Although the feature pooling is a part of CNNs in topologies, it is listed out separately because it directly changes the data correlation among adjective pixels, which subsequently changes local features. Pooling approaches show different enhancement effects. Among the above factors, the first two are related to training images while the latter two to CNN. In the view of the fact that both images and CNN should be considered, the best way is to adaptively determine image enhancement with the training of CNN. This paper proposes an adaptive local feature enhancement (ALFE) model. The local feature enhancement is different from traditional image enhancement. The main contributions of this work are (i) The model takes low-frequency general appearance (GA) enhancement and high-frequency local detail (LD) enhancement into account. It is argued that low-frequency information and high-frequency information are both important in human visual perception. (ii) The model improves local features through only two extra trainable enhancement parameters, can avoid overfitting, and has wide adaptability. Based on the adaptive adjustment of the parameters, it achieves a global enhancement, allowing for both training images and CNN. The rest of this paper is organised as follows. Section 2 introduces the related works of CNNs and analyses the existing problems. Section 3 proposes an ALFE model and discusses the implementation of ALFE on CNN. In Section 4, experiments are carried out to validate the performance of the ALFE model. Furthermore, this section discusses the training of CNN+ALFE, providing two training approaches. Finally, conclusions are presented in Section 6. 2 Related works The investigation of local feature enhancement in CNNs is inspired by a fact that CNN is built on the extraction of local features. So far, local feature enhancement is mainly implemented through feature pooling and feature enhancement filters. There are many works about the improvement of feature pooling, which essentially contribute to the extraction of local features. The forms of feature pooling include max-pooling [[23]], average pooling [[24]], stochastic pooling [[25]], and learning pooling [[26]]. The first three are hand-crafted pooling operations, and the last one is a learning pooling operation. A feature pooling operation is equivalent to the combination of a convolution operation and a subsampling operation. For example, the max-pooling picks the max among input activations at ( p , q ) within a local region R i , j as its output, which is equivalent to a combination of a max convolution operation and a subsampling operation. In max pooling, high-frequency signals, e.g. texture information, are enhanced while the low-frequency signals, e.g. general shape and background, are depressed. The main problem with the hand-crafted pooling operations is that they cannot adaptively adjust according to input data, and are not guaranteed to minimise the classification error during the training stage. Learning pooling uses a linear filter to aggregate the features within local pooling regions. It adjusts its pooling parameters with CNN training. The problem with it is that it would generate too many trainable parameters and leads to overfitting. Furthermore, we argue that the main functions of feature pooling are to reduce the dimension, avoid over-fitting, etc., and its enhancement effects are limited because its equivalent convolution operator has limitations. For example, simply selecting the maximum value in max-pooling or average value in average pooling may not be suitable for image feature enhancement in actual applications. This paper does not investigate feature enhancement from the perspective of pooling; instead, it only takes feature pooling as a dimension reduction method. With feature enhancement filters, most current research works can be categorised as traditional image enhancement. In the works, feature enhancement is not the research focus and is just simply involved in a specific task, thus lacking in-depth analysis. Zhu et al. [[14]] and Jang et al. [[15]] adopted contrast enhancement in natural scene character recognition and fingerprint spoof detection based on CNN, respectively. Zhu et al. applied a sine function-based histogram transformation to enhance bimodal histograms of character images. It pushed the histogram peaks of foreground and background away from each other while keeping the greyscale range. Jang et al. used histogram equalisation to increase the contrast of images. Histogram equalisation makes the probability of the histogram of an image uniform and attempts to transform the spatial histogram of an image to closely match a uniform distribution. Basheera et al. [[16]] adopted Gaussian filters, acting as an enhancement layer, to improve CNN-based Alzheimer's disease classification methods. Vardhana et al. [[17]] diagnosed diseases using CNN image classification, and presented an edge detection layer to detect the biomedical images. Edge enhancement can highlight object structures, boundaries and textures, facilitating the identification of different types of objects and the delineation of their distribution ranges. Nahid et al. [[18]] proposed histopathological breast-image classification with multiscale retinex enhancement by CNN. Instead of using the raw images directly, they normalised the image by applying the multiscale retinex algorithm to improve the local contrast and illumination variation. Nogueira et al. [[27]] discussed the fingerprint image enhancement in fingerprint liveness detection using CNNs, including image local normalisation, orientation estimation, ridge frequency estimation, Gabor filtering and image binarisation/thinning. Oloyede et al. [[28]] utilised a grey transfer function to manipulate each pixel, and enhanced face images in the face classification using CNN. It was argued that an effective transformation function should be used that is capable of efficiently mapping the intensity values of the raw input image so that it produces an improved output image. Consequently, an evaluation function was designed to automatically select the optimal enhancement parameters. The main problems with feature enhancement filters are as follows: (i) Only the input images are taken into account, thus lacking high applicability for actual practices. As discussed in Section 1, local features in CNN are also related to image preprocessing, CNN topology, feature pooling, etc. (ii) Image enhancement pays more attention to high-frequency signals while ignoring low-frequency signals. This paper argues that low-frequency signals are also conductive to data classification, especially, category identification. (iii) There is no general enhancement method for CNNs. So far, image enhancement is mainly designed for specific images, such as character, fingerprint and medical images. 3 Proposed method 3.1 Low-frequency and high-frequency enhancements What should be enhanced in the local features of CNNs? According to visual perception, human beings classify objects according to two kinds of features: GA characteristics and LD characteristics [[29]-[31]]. GA characteristics represent the shapes of image object and background, and are highly related to low-frequency contents. LD characteristics refer to image edge, texture, corner etc., which are related to high-frequency contents. Coincidentally, image contents can be roughly divided into low-frequency and high-frequency contents. Image enhancement in the traditional sense mainly refers to high-frequency enhancement [[14]-[18], [32]]. However, in image classification, low-frequency characteristics are also beneficial for image classification. Consequently, both the low-frequency and high-frequency enhancements are considered in this paper. For example, Fig. 1 shows some images from two data sets CIFAR-10 [[33]] and dogs versus cats [[34]]: Fig. 1a shows the ten classes in CIFAR-10, as well as two random images from each class; Fig. 1b shows two classes in dogs and cats, as well as four random images from each class. Since dogs versus cats contain higher-resolution images with fewer classes compared with CIFAR-10, it is easier to be interpreted and analysed. Accordingly, the images in this data set are chosen for further analysis. Fig. 1Open in figure viewerPowerPoint Images from CIFAR-10 and dogs versus cats (a) Images from CIFAR-10, (b) Images from dogs versus cats Both low-frequency GA and high-frequency LD characteristics contribute to the classifications of the images. Fig. 2 shows the diagrammatic sketches of a cat and a dog. They are obtained by cropping two images in dogs versus cats and marking the objects in black. According to the GA sketches, it is easy to classify the two sketches: Fig. 2a is a dog and Fig. 2b is a cat. From the perspective of visual perception, GA depicts the general object appearance and components, conductive to segmenting the object from the background. It is heavily related to low-frequency contents in images. Sometimes an image can be classified only according to some part of the object in the image. For example, the two images can be classified according to the heads in the rectangles. Fig. 3 shows two raw images in dogs and cats and two patches are marked by rectangle boxes from the two images, respectively. It is evident that the two images can be distinguished based on the patches with high probabilities. The two image patches contain the objects’ texture characteristics, which are also helpful to image classification. The texture characteristics depict the local details of images and are heavily related to high-frequency contents in images. Fig. 2Open in figure viewerPowerPoint Image classification based on GA shapes (a) Sketch of a dog image, (b) Sketch of a cat image Fig. 3Open in figure viewerPowerPoint Image classification based on LD textures (a) Dog image, (b) Cat image GA and LD characteristics may take different roles in different data sets. To keep high applicability in various conditions, the image enhancement in CNN should take both GA and LD enhancements, namely low-frequency and high-frequency enhancements, into consideration. 3.2 Local enhancement operators In order to be feasible for various CNNs and data sets, this paper proposes an adaptive local enhancement (ALFE) model which adaptively adjusts its enhancement parameters through end-to-end training, considering that the proper tendency and intensity of enhancement are unknown in advance. Sample data and local feature maps in CNNs are two-dimensional images. We choose spatial enhancement operators to enhance local features in CNNs because the computation complexity of spatial enhancement is low, and its output can be directly imported into CNNs. Fully-adjustable operators have high representation capabilities, but they are similar to common convolutional operators. It is difficult to push the common convolution operators to enhancement operators through the training of CNNs. If a fully-adjustable operator is used as an image filter, it will act as a single-channel convolutional layer, causing overfitting. Therefore, the operators, which, consisting of two parts, are defined as follows: F = U + a M (1) where the first part U denotes a preserving unit to keep raw image contents; the second part a M stands for a local enhancement matrix. a is an adjustable parameter, and M is a basic enhancement unit. Whether a is positive or negative denotes the tendency of enhancement, and its absolute value denotes the intensity of enhancement. The enhancement operator can return to the state of zero enhancement when a is equal to zero. Here the enhancement operator is represented as a spatial filter. It is implemented through a single-channel local convolution on images (input images or local feature images) which means that it can be embedded into CNNs and adaptively determined together with the training of CNNs. As is discussed in Section 3.1, local enhancements in CNNs include a low-frequency enhancement and a high-frequency enhancement. Therefore, M is labelled as M l or M h , F as F l or F h , and α as α l or α h , to indicate their different roles. I is used to label the image to be enhanced; similarly I l and I h are used to denote low-frequency enhanced, and high-frequency enhanced images, respectively. When α > 0, the operator plays a role of local feature enhancement; when α < 0, the operator plays a role of local feature reduction; when α = 0, only the preserving unit U in (1) is left, indicating a zero enhancement. The following in this section are the designs of U, M l and M h , which are in the form of matrixes. 3.2.1 Preserving unit U The size of the preserving unit U corresponds to that of a local convolution window in CNN. The size is supposed to be 5 × 5. Then, as an example, U is defined in Fig. 4. Its central element is 1 and other elements are 0. A convolution operation based on U is to preserve the raw data in I. Fig. 4Open in figure viewerPowerPoint Preserving unit matrix 3.2.2 Basic low-frequency enhancement unit M l The basic low-frequency enhancement unit M l is to enhance low-frequency GA features. The contrast among different components will be enhanced. For example, the background and object in an image will display a higher contrast. The elements in M l represent the weights of adjective points of a point ( x , y ), and the weights are closely related to the correlations between adjective image pixels. Given that the correlations are further related to their spatial distances, it is supposed that the weight parameters in M l obey a Gaussian distribution. However, instead of using a Gaussian filter, a low-frequency enhancement operator M l is produced as shown in Fig. 5 with a 5 × 5 window as an example. The left subimage shows a two-dimension matrix from a Gaussian distribution with a mean equal to 0 and variance equal to 2.0. Through setting the central element to zero, a basic low-frequency enhancement matrix can be obtained, as shown in the right subimage. Fig. 5Open in figure viewerPowerPoint Production of low-frequency enhancement unit matrix. The left subimage shows a Gaussian matrix with mean equal to 0 and variance equal to 2.0, and the right shows the produced low-frequency enhancement unit matrix The enhanced image I l , ( x , y ) at a point ( x , y ) is a weighting sum as I l , ( x , y ) = I ( x , y ) + α l ∑ i , j ∈ R ( x , y ) I ( i , j ) × ω ( i , j ) (2) where R ( x , y ) refers to the local convention windows of the point ( x , y ). It consists of two parts: the first part is produced by the preserving matrix U; the second part is produced by the low-frequency enhancement unit matrix M l . The basic low-frequency enhancement unit is a low-frequency filter, enhancing low-frequency local feature. According to (2), setting the central element of M l to zero helps to enhance the correlations between the point ( x , y ) and its adjective points in I l , reducing data independence. Meanwhile, the local enhancement operator has little influence on high-frequency signals, which can be greatly reflected by the differences between adjective pixels. High-frequency signals are only affected by the second part after the image signals are enhanced. It can be inferred that the changes of the differences are so minor that high-frequency signals are not affected approximately. Through adjusting a l , M l optimises the contrast among local regions to improve image classification. 3.2.3 Basic high-frequency enhancement unit M h The role of basic high-frequency enhancement unit M h is to enhance high-frequency LD features. The contrast among adjective pixels would be enhanced. The elements in M h represent the weights of adjective points of a point. M h , as an example with a 5 × 5 window, is produced as shown in Fig. 6. It can be obtained with the following equation: M h = U − M G (3) where U is a preserving unit matrix, and M G is a Gaussian matrix. It can be seen that the elements except the central in M h are opposite to those in M G , and the sum of all the elements in M h is equal to zero. Fig. 6Open in figure viewerPowerPoint Production of high-frequency enhancement unit matrix. The left subimage shows a Gaussian matrix with mean equal to 0 and variance equal to 2.0, and the right shows the produced high-frequency enhancement unit matrix The value I h , ( x , y ) at a point ( x , y ) can be shown as follows: I h , ( x , y ) = I ( x , y ) + α h ∑ i , j ∈ R ( x , y ) , ( i , j ) ≠ ( x , y ) I ( x , y ) − I ( i , j ) × ω ( i , j ) (4) It consists of two parts: the first part is produced by the preserving matrix U; the second part is produced by high-frequency enhancement unit matrix M h , enhancing the differences among neighbouring pixels. According to (4), the local enhancement operator has little influence on low-frequency signals. Low-frequency signals are only affected by the second part after the image signals are enhanced. If low-frequency signals in a local region are equal, they are not affected according to (6). Through adjusting a h , M h optimises image details to improve image classification. 3.3 Local feature enhancement in CNNs As discussed in Section 3.2, operators F l and F h work independently to a large extent, enhancing low-frequency and high-frequency signals in images. Therefore, it can be designed as two stacked enhancement layers: a low-frequency enhancement layer and a high-frequency enhancement layer. Each enhancement layer includes a single-channel enhancement transform. In actual applications of CNNs, raw input data are usually signal-channel grey images or three-channel colour images (RGB images). With colour images, the three channels are enhanced by the same enhancement transforms. Namely, each enhancement layer is extended from one channel to three channels via copying. There are one input layer, and multiple conventional and pooling layers in CNNs. It seems that local feature enhancement can be added to each layer. However, more local feature enhancement parameters are likely to cause overfitting. The low-frequency enhancement layer and high-frequency enhancement layer are added following the input layer, which will be convenient for the enhancement based on both input images and CNNs. The two layers have two placement orders: low-frequency + high-frequency enhancement layers and high-frequency + low-frequency enhancement layers. Theoretically, low-frequency and high-frequency enhancement operators work independently, and the two orders are approximately equivalent. It is noted that the experiments on the two orders produced similar results and consequently only the results of the first order are shown for the conciseness of the paper. Enhancement parameters a l and a h are adjusted as CNN parameters in CNN training. The initial values of enhancement parameters are set to zero or random values near to zero. In data forward propagation during the training of CNN+ALFE (CNN with ALFE), the local feature enhancement effects are transferred to the follow-up layers with data stream, optimising local features in the follow-up layers. In error backpropagation, the training errors in each CNN layer are propagated backward to enhancement layers for the adjustment of the trainable parame