Residual‐wider convolutional neural network for image recognition

IET Image ProcessingVolume 14, Issue 16 p. 4385-4391 Research ArticleFree Access Residual-wider convolutional neural network for image recognition Guoqiang Li, Guoqiang Li Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this authorWenhua Chen, Corresponding Author Wenhua Chen linfengyu128@163.com Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this authorChao Mu, Chao Mu Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this author Guoqiang Li, Guoqiang Li Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this authorWenhua Chen, Corresponding Author Wenhua Chen linfengyu128@163.com Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this authorChao Mu, Chao Mu Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, 066004 People's Republic of China Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004 People's Republic of ChinaSearch for more papers by this author First published: 23 February 2021 https://doi.org/10.1049/iet-ipr.2020.0726AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Recent works show that the performance of convolutional neural networks (CNNs) could be improved by making the network wider and introducing residual connections. For example, the Inception architecture is one of the classical models. However, the structure of the series of Inception is complex, and there are more convolution layers and existing redundant image feature information. In order to further improve the performance and observability of CNNs, a novel type of network structure based on two modules, wider module and residual module, is proposed to make full advantage of the information of the image and learn more abundant features in this study, which is called residual-wider network (R-WN). The structure of R-WN is easy to understand and adopts modular design method. Experiential results demonstrate that the proposed R-WN with optimal structure and much fewer convolution layers could achieve much better image recognition results on four classical open data sets (MNIST, CIFAR10, SVHN, and Oxford Flowers 17). 1 Introduction In recent years, due to the active studies and achievements regarding artificial intelligence (AI), AI technologies based on deep learning [[1]] have been successfully applied in speech recognition [[2]], natural language processing [[3]] and computer vision. The deep convolutional neural networks (CNNs) have outperformed state-of-the-art algorithms in many visual recognition tasks, such as image classification [[4]], image retrieval [[5]], object detection [[6]], semantic segmentation [[7]], and so on. The classical CNNs evolve from VGG [[8]], Inception [[9]-[11]], residual network (ResNet) [[12]] to DenseNet [[13]]. In the meantime, many other CNNs have emerged. AlexNet [[14]] was the first attempt of applying a deep convolutional network on image processing and the winner of the 2012 ImageNet competition, which was widely focused and used. VGG-Nets were used as the basic network of localisation tasks in the 2014 ImageNet competition and won the first prize. The depth is regarded as the vital factor for the performance of VGG-Nets. Network-In-Network utilised multilayer perceptron (a combination of multiple full connection layers and non-linear function) to improve image classification performance, which was different from traditional neural networks. GoogLeNet, a deeper and wider convolutional network was presented by Google in the 2014 ILSVRC. It is known as 'Inception', which tries to introduce more sparsity and more optimal locality into the convolutional layers (the number of neurons). Such inception structure highly expedites the proceeding of many computer vision issues. Motivated by the success of GoogLeNet, several variants of Inception emerged for boosting further enhancement [[15]]. The above researches indicate that the depth and width are the core factors for representing network performance. However, it becomes increasingly difficult to train deeper and deeper networks. To solve this problem, Srivastava and co-authors proposed Highway Networks, which used a learned gating mechanism to make information flow across several layers. Subsequently, He and co-authors proposed ResNet, which used an identity shortcut connection to realise residual learning. Not long ago, DenseNet pushed this method to the extreme by connecting each layer to every other layer in a feed-forward fashion. As ResNet becomes more popular in the research community, many variants of residual networks emerge. ResNeXt [[16]] introduced a hyper-parameter called cardinality, which provided a new way to adjust the capacity of models. Deep networks usually require several weeks to be trained. To overcome the shortcoming of time-consuming, Huang et al. [[17]] proposed stochastic depth residual networks (SD ResNets), which not only improved the classification performance but also decreased drastically the training time. However, SD ResNets randomly dropped some layers during the training procedure but the whole network was used in the evaluation. What this means is that some of the connections in ResNet may be redundant. In our opinion, we agree that the trained CNNs could be more accurate and efficient when there is enough depth or width. Motivated by the remarkable success of ResNet and Inception-ResNet-v2 [[11]], we draw on the idea of the residual module and wider module to propose a new architecture called residual-wider network (R-WN) that not only presents a novel layout of skip connections but also decreases the convolution layers, the structure of R-WN is given as Fig. 1. In R-WN, the residual connections are applied by us between sub-blocks (e.g. A1, B1, and C1) as a method to solve the issue of gradient vanishing, the elaborate connections in the inside of every sub-block is introduced to preserve the feed-forward nature. A wider module that contains two branches with filters of different sizes for seeing various receptive fields is proposed, and then features are combined by summation (element-wise addition) just as ResNet before they are fed into another layer. R-WN is tested on four highly competitive benchmark data sets (MNIST, CIFAR10, SVHN, and Oxford Flowers 17) and compared with state-of-the-art methods, such as Inception-ResNet-v2, DenseNet, and so on. Experimental results show that our method with fewer parallel structures and fewer layers do not harm the performance of the network but get an evident promotion. Fig. 1Open in figure viewerPowerPoint Overall arrangement of R-WN, wider modules (A, B, and C) are shown in Fig. 2b The rest of this paper is organised as follows. To begin with, the related research projects are introduced in Section 2. Furthermore, Section 3 shows image classification experiments and the result analysis. Finally, the paper is concluded in Section 4. 2 Modular design of R-WN R-WN contains three main parts, traditional CNNs, residual connections based on the residual network for training deep architectures easily and wider module optimised from Inception-ResNet-v2 for making full use of multi-scale image information and accelerates the speed of convergence. 2.1 Architecture of R-WN We assume that there are n blocks in R-WN, the connections of adjacent blocks and the inside connections of every block are given as Fig. 1. There are four skip connections in each block of R-WN, one skips connection outside of wider modules (except for the last block) and three inside wider modules. Two branches with kernels of different sizes in every wider module of R-WN are introduced to increase the width of the network, the confluence of information of each branch adopts the operation of element-wise addition, which is also used in ResNet. ReLu is used as an activation function after each wider module, but it is removed after each convolution layer and each element-wise addition operation in the residual module. Besides, dropout is used by us after each block to avoid overfitting. At the end of the last block, a global average pooling, a fully connected layer, and a softmax are added to generate the final classification results. In a word, R-WN is designed efficiently, which can not only handle the problems of overfitting, vanishing gradient and network saturation, but also accelerate the speed of convergence and make the classification results better. 2.2 Convolutional neural network Generally speaking, CNN is a kind of feed-forward neural network with deep structure and convolution calculation, which is one of the representative algorithms of deep learning. The CNNs architecture basically consists of several convolutional layers, pooling layers and optionally fully connected layers. In our proposed network, the module of CNNs is extended by us to include the following layers: Input layer: It holds the raw pixels of the image with h × w × c (e.g. 32 × 32 × 3), where h is the height of the image, w is the width and c is the channel number (e.g. c = 3 for RGB images). Batch normalisation (BN) [[15]]: It is used after each convolutional layer in order to allow the network to use a larger learning rate for training and accelerate the network training speed, as well as to avoid the risk of overfitting and divergence. Rectified linear unit (ReLu) [[18]]: ReLu is an activation function ( f x = max 0 , x ) that is the most commonly used in the output of convolution layers. It shortens the learning cycle and avoids network saturation. Global average pooling: Which is used to pooling the average value of the feature graph at the last layer to form a feature point and constitute the final feature vector, is mainly used to solve the problem about fully connected layers. It can reduce computational parameters, and bring more sparseness and robustness to the network. Besides, there is no parameter to optimise in the global average pooling, thus overfitting would be avoided at this layer. 2.3 Residual module The deep residual learning framework [[12]] is essential for avoiding the degradation problem. Suppose the desired underlying mapping of the CNNs is H x , where x is the input image. It has been proved that a residual function F x = H x − x is easier to optimise rather than approximating original mapping H x . Therefore, a residual block (as shown in Fig. 2a) is defined as F x + x, which is performed by an element-wise addition. In R-WN, the residual module is introduced by us, however, ReLu is not adopted in the residual module, namely, the ReLu is removed after each convolution layer and each element-wise addition operation. In short, we do not use the complete residual block of ResNet in our proposed R-WN, but the idea is supported by us. Fig. 2Open in figure viewerPowerPoint Residual block and wider module (a) Residual block, (b) Wider module From our perspective, the problems of gradient disappearance and gradient explosion during training deeper networks can be solved by introducing skip connections like ResNet. In addition, it helps the network to learn the new information of the previous layer and the training speed of the model is significantly improved. 2.4 Wider module The wider module plays an extremely important role in the whole network and improves the performance of the network. One sample of the wider module is shown in Fig. 2b. There are two branches in the wider module and skip connection is introduced for one of them. The filter sizes are 1 × 1 and 3 × 3 and the output filter banks of the last two layers of both branches are combined and added into one output vector. Formally, x denotes the input of the wider module. In order to easily describe two branches, x l and x r are introduced to denote the inputs of left and right structure, respectively, where x l = x r = x, and y as the output of one wider module. Therefore, the mathematics model of a wider module could be described as (1)–(5): x l 1 = f l 1 x l = f l 1 x (1) x r 1 = f r 1 x r = f r 1 x (2) x r 2 = f r 2 x r 1 = f r 2 f r 1 x r = f r 2 f r 1 x (3) x r 3 = f r 3 x r 1 + x r 2 = f r 3 f r 1 x r + f r 2 f r 1 x = f r 3 f r 1 x + f r 2 f r 1 x (4) y = x l 1 + x r 3 (5) where the operation of composite function f ∗ can be BN, ReLU, or convolution (conv). f r 1 ∗ , f r 2 ∗ , and f r 3 ∗ denote the operation of the first layer, the second layer, and the final layer of right structures, respectively. f l 1 ∗ is the operation of the first layer of left. x r 1 , x r 2 , and x r 3 denote the outputs of the first layer, second layer, and the final layer of right structures, respectively. x l 1 is the output of the first layer of left. From the formulas (1)–(5), we can observe that the image information x is extracted features many times by wider module, which realises the fusion and reuse of multi-scale feature information. Suppose the error of the output of the last layer of the module is θ, and the gradient propagation formula is as (6): ∂ θ ∂ x = ∂ θ ∂ y ⋅ ∂ y ∂ x = ∂ θ ∂ y ⋅ ∂ x l 1 ∂ x + ∂ x r 3 ∂ x = ∂ θ ∂ y ⋅ ∂ f l 1 x ∂ x + ∂ f r 3 f r 1 x + f r 2 f r 1 x ∂ x = ∂ θ ∂ y ⋅ ∂ f l 1 x ∂ x + ∂ f r 3 f r 1 x + f r 2 f r 1 x ∂ x ⋅ ∂ f r 1 x ∂ x + ∂ f r 2 f r 1 x ∂ x ⋅ ∂ f r 1 x ∂ x = ∂ θ ∂ y ⋅ ∂ f l 1 x ∂ x + ∂ f r 3 f r 1 x + f r 2 f r 1 x ∂ x ⋅ ∂ f r 1 x ∂ x ⋅ 1 + ∂ f r 2 f r 1 x ∂ x (6) As can be seen from the formula (6), there is no continued product phenomenon, thus avoiding the gradient disappeared or explosion to a certain extent. For the entire R-WN network, assuming that the m i represents the output image of each module, where the subscript i represents the number of modules, which is taken as the multiple of 3, and F ∗ represents all the operations of each module on the input image, the mathematical modelling formulas for the entire model are expressed as (7)–(15): m 1 = x l 1 + x r 3 = f l 1 x + f r 3 f r 1 x + f r 2 f r 1 x = F 1 x (7) m 2 = m 1 + F 2 m 1 (8) m 3 = F 3 m 2 (9) m 4 = F 4 m 3 (10) m 5 = m 4 + F 5 m 4 (11) ⋮ m 3 i − 1 = m 3 i − 2 + F 3 i − 1 m 3 i − 1 (12) ⋮ m i − 2 = F i − 2 m i − 3 (13) m i − 1 = F i − 1 m i − 2 (14) m i = F i m i − 3 (15) It can be found that the essence of formulas (7)–(15) is a composite function, a series of convolution operation, feature fusion operation and others are used for the input image, which achieves the end-to-end training. The gradient propagation formulas of entire R-WN network is similar to the formula of a single wider module and is not repeated. 2.5 Theory analysis The structures of modules of R-WN and Inception-ResNet-v2 are shown in Fig. 3, you can observe that the module of Inception-ResNet-v2 contains many branches and the structure is relatively complex. While R-WN optimises its structure and parameter settings by reducing the number of branches and convolution operations, but continues to hold the richness of receptive fields ( 1 × 1, 3 × 3, and 5 × 5). The receptive field represents the image features of the CNNs which can see the area of input images, that is, how much the output of the feature is affected by the pixels in the area of the receptive field. Different sizes of receptive fields can extract different image features and the richness of receptive fields can help improve the image classification performance. Fig. 4 shows different sizes of receptive fields involved in the two networks. The formulas of receptive field calculation are represented as (16) and (17): receptive field = output field size − 1 × stride − 2 × padding + kernel size (16) receptive field size = output field size − 1 + kernel size (17) stride and padding are not considered when calculating receptive fields for computational convenience, because both of them are invariants. As a result, formula (16) is simplified to (17), where kernel size indicates the convolution kernel size, output field size represents the output feature map size. Finally, three different sizes of receptive fields, 1 × 1, 3 × 3, and 5 × 5 are obtained. The module of Inception-ResNet-v2 employs a number of 1 × 1 convolution kernels, but R-WN has halved them. We believe that the richness of receptive fields can maintain the performance of the network to a certain degree, however, the number of convolution kernels is in a secondary position. The experimental results of Section 3 can demonstrate that some 1 × 1 convolution operations are redundant and prove the effectiveness of R-WN. Fig. 3Open in figure viewerPowerPoint Module of Inception-ResNet-v2 and the module of R-WN (a) Module of Inception-ResNet-v2, (b) Module of R-WN Fig. 4Open in figure viewerPowerPoint Kinds of receptive fields (a) Receptive field: 1 × 1, (b) Receptive field: 3 × 3, (c) Receptive field: 3 × 3, (d) Receptive field: 5 × 5 3 Experiments We train and test R-WN on four open data sets (MNIST, CIFAR10, SVHN, and Oxford Flowers 17) and compared with successful deep learning networks, such as Inception-ResNet-v2 and DenseNet. The four data sets are processed in the same way in every experiment in order to ensure a fair comparison, respectively. Besides, the set of hyper-parameters obeys the rules of comparison in each experiment. For all experiments, R-WN with n = 3 (37 convolution layers) was selected as the basic framework. Table 1 provides details of each block and each layer in R-WN. On the one hand, the performance of R-WN with n = 3 is much better. On the other hand, R-WN is optimised based on Inception-ResNet-v2 and the number of convolution layers of Inception-ResNet-v2 is 45, which 37-layer R-WN is better to compare with Inception-ResNet-v2 for the approximate number of convolution layers. We give the experimental details and report the quantitative image classification results with analysis on four popular data sets in this section. Table 1. Architecture of R-WN, n is set to 3, sub-block represents the wider modules A, B and C in R-WN # Layer Wider module Output size input 7 × 7 , 64 — 7 × 7 / 8 × 8 , 64 block 1 sub-block 1 3 × 3 , 256 + 1 × 1 , 64 3 × 3 , 64 3 × 3 , 256 × 3 7 × 7 / 8 × 8 , 256 block 2 sub-block 2 3 × 3 , 512 + 1 × 1 , 128 3 × 3 , 128 3 × 3 , 512 × 3 7 × 7 / 8 × 8 , 512 block 3 sub-block 3 3 × 3 , 1024 + 1 × 1 , 256 3 × 3 , 256 3 × 3 , 1024 × 3 7 × 7 / 8 × 8 , 1024 output softmax — 1 × 1 , 10 / 17 3.1 Data sets MNIST: The MNIST database [[19]] comes from NIST's Special Database 3 (SD-3) and Special Database 1 (SD-1), which contains binary images of handwritten numbers from 0 to 9. They are collected by high school students and staffs of the United States Census Bureau. A total of 30,000 patterns from SD-3 and 30,000 patterns from SD-1 are purposefully gathered in the training set with 60,000 samples. The test set including 10,000 samples, 5000 of which are selected from SD-3 and the remainder from SD-1. The MNIST data set contains grey-scale images with 28 × 28 dimension. CIFAR10: The CIFAR10 data set [[20]] consists of coloured natural images with 32 × 32 pixels, which is drawn from ten classes and split into 50,000 training and 10,000 testing images. We adopt a standard data augmentation scheme (mirroring/shifting) that is widely used for this data set [[12]]. Street view house numbers: The SVHN [[21]] data set is composed of 32 × 32 colour images of house numbers collected by Google Street View. There are two formats in the data set we consider the second format, which is divided into 73,257 images as the training set, 26,032 ones as the testing set and 531,131 ones as an extra set. The task of this data set is to classify the digit located at the centre of the image other digits may appear in the background, which are not the target to detect. Oxford Flowers 17: Oxford Flowers 17 data set is a selection of 17 flowers in the UK selected by Visual Geometry Group at Oxford University, consisting of 224 × 224 pixels of colour images, each with 80 images and 1360 images for the entire data set. 3.2 Training All networks are trained using weight decay of 10−4 and a Nesterov momentum of 0.99. The weight initialisation introduced by [[12]] is adopted by us. A dropout layer is added in R-WN after each block (except the second block) and set the dropout rate to 0.2. We run all the experiments on a single Nvidia TITAN Xp GPU in Tensorflow, which is a very popular deep-learning frame. On CIFAR10, we use a random gradient descent optimisation algorithm with momentum. The batch size is 64 and epochs are 200. The initial learning rate is set to 0.001 and would be divided by 10 at 50 and 75% of the whole number of the total training epoches. On MNIST, the batch size is 128 and we set epochs to 50. The initial learning rate is 10−4 and is divided by 10 at 50% and 75% of the total training epoches. The optimisation algorithm is the same as CIFAR10. For SVHN, the optimisation algorithm is Adam and the initial learning rate is set to 10−4. The batch size is 40 and the epochs are 20. Besides, we apply the method of local contrast normalisation as [[22]] to preprocess the data set. On Oxford Flowers 17 data set, the batch size is 32 and we set epochs to 50. The initial learning rate is set to 0.1 and the optimisation algorithm is Adam. However, we do not intend to achieve the state-of-the-art results which also utilise other techniques such as ensemble, randomised input order and sampling methodologies. Therefore, the simple architecture of the proposed network is employed by us, as well as the one used in these experiments. Verify the validity of R-WN through a series of contrast experiments. 3.3 Image classification results and analysis The test results of R-WN and other networks are reported in Table 2, the focus is on the analysis of R-WN and Inception-ResNet-v2 since R-WN is improved on the foundation of Inception-ResNet-v2. What is noteworthy in Table 2 is that R-WN with 37 layers outperforms Inception-ResNet-v2 with 45 layers on three data sets (MNIST, CIFAR10, and SVHN), the accuracies of R-WN increases by 0.03%, 1.06%, 0.25%, respectively. It is an exciting result for such a challenging classification task that the fewer of the convolution layers, the better result of classification recognition. In addition, take SVHN for example, we present the curves graph in Fig. 5, which records the course of losses of R-WN and Inception-ResNet-v2 during the training procedure. It is observed that the loss of R-WN is lower than Inception-ResNet-v2 from early stages, which the speed of convergence is also faster than Inception-ResNet-v2. At the same time, it verifies the correctness of theoretical analysis in Section 2.5 and the rationality optimisation of R-WN, that is, the part 1 × 1 convolution operations of the module of Inception-ResNet-v2 are redundant and the improvement of R-WN is effective. In order to enhance the persuasion of the experimental results, the comparison curves of losses of R-WN and five advanced CNNs are drawn in Fig. 6, and we can observe obviously that the convergence rate of R-WN is the fastest, as well as the drop in loss, is most pronounced, which further proving the above conclusion. Table 2. Test accuracies (%) of models on four data sets, we mark the overall best results in boldface to highlight them Model Layers MNIST CIFAR10 SVHN Oxford flowers 17 ResNet-34 34 99.32 85.75 89.4 78.36 ResNet-50 50 98.99 79.87 87.6 48.56 DenseNet (k = 12) 76 98.26 82.20 72.32 — DenseNet (k = 12) 106 99.28 86.04 86.04 80.29 Xception [[23]] 42 98.87 86.19 87.37 — Inception-ResNet-v2 45 99.38 86.18 92.05 — R-WN (ours) 37 99.41 87.24 92.30 82.21 Fig. 5Open in figure viewerPowerPoint Loss of two models on SVHN Fig. 6Open in figure viewerPowerPoint Loss of many models on SVHN The performance of R-WN is best on many data sets, as shown in Fig. 7, showing the test accuracies of R-WN and other models on four data sets, which indicates that R-WN is not designed for a particular data set, but it has a strong generalisation capability. Fig. 7Open in figure viewerPowerPoint Test accuracy of R-WN compared with other models on four data sets It is deserving to be mentioned that many networks, like DenseNet, could have had better performance with more training steps, but the number of steps is set the same for all models in every experiment for an equal comparison. By contrast, on the MNIST data set, although the test accuracy of R-WN is the highest compared with other networks, the gap of percentage between R-WN and Inception-ResNet-v2 is small, only increases by 0.03%. We think that the biggest reason may be that MNIST data set is relatively small and the shallower network can achieve high accuracy, the present test accuracy is so high that it is difficult to improve it significantly. As mentioned earlier, R-WN can achieve much higher accuracy using other optimisation techniques, which are out of the scope of this paper and not the main goal. All in all, it is proved that the improvement and optimisation of R-WN are effective through theoretical analysis and experiments, and R-WN has a better performance compared with Inception-ResNet-v2. The operations of halving the number of 1 × 1 convolution, holding the richness of receptive fields ( 1 × 1, 3 × 3, and 5 × 5) and reducing the number of branches not only improve the test accuracy of image classification of R-WN but also accelerates the speed of convergence. As the receptive field of 1 × 1 convolution kernel is small, though which can further compress the network and reduce the dimension, the most important information of feature maps will be lost by adopting many 1 × 1 convolutions, and convolution operation in the computer is more complex with a large proportion of resources when extracting image features. In conclusion, compared with the number of convolution kernels, the richness of the receptive field plays a more important role in the expressiveness of networks. 4 Conclusion In this paper, we propose a new R-WN, which improves the performance significantly on MNIST, CIFAR10, SVHN, and Oxford Flowers 17 data sets for image classification. To begin with, there are two parallel structures and 37 convolutional layers in R-WN, which are fewer than the most of popular CNNs in the present. Furthermore, experimental results show that, compared with other models, R-WN with less layers and fewer branches not only achieves much better test accuracy and generalisation ability but also speeds up convergence processes, which has testified its effectiveness. In future work, testing the image recognition effect of R-WN on ImageNet data set will be the main task. 5 Acknowledgments This project was supported by the National Natural Science Foundation of China (Grant No. 61403331), the Program for the Top Young Talents of Higher Learning Institutions of Hebei (Grant No. BJ2017033), the Natural Science Foundation of Hebei Province (Grant No. F2016203427), and the China Postdoctoral Science Foundation (Grant No. 2015M571280). 6 References [1]LeCun, Y., Bengio, Y., Hinton, G.: 'Deep learning', Nature, 2015, 521, pp. 436– 444 [2]Bisio, I., Garibotto, C., Grattarola, A., et al.: 'Smart and robust speaker recognition for context-aware invehicle applications', IEEE Trans. Veh. Technol. (IEEE), 2018, 67, (9), pp. 8808– 8821 [3]Zhang, Y., Zhang, P., Yan, Y.: 'Language model score regularization for speech recognition', IET Image Process., 2019, 28, (3), pp. 604– 609 [4]Zhou, Z., Shin, J., Zhang, L., et al.: 'Fine-tuning CNNs for biomedical image analysis: actively and incrementally'. Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 7340– 7349 [5]Qi, Y., Zhang, S., Qin, L., et al.: 'Hedging deep features for visual tracking', IEEE Trans. Pattern Anal. Mach. Intell. (IEEE), 2018, 1, (1), p. 99 [6]Xie, D., Zhang, L., Bai, L., et al.: 'Deep learning in visual computing and signal processing'. Applied Computational Intelligence and Soft Computing (ACISC), 2017, pp. 1– 13, https://doi.org/10.1155/2017/1320780 [7]Zhang, L., Chai, R., Arefan, D., et al.: 'Deep learning method for tumor segmentation in breast DCE-MRI'. Medical Imaging, The Int. Society For Optical Engineering (SPIE), San Diego, CA, USA, 2019 [8]Zhao, S., Zhao, X., Ding, G., et al.: 'Unsuper-vised domain adaptation for learning discrete probability distributions of image emotions'. Association for Computing Machinery Multimedia Conf. (ACM), New York, NY, USA, 2018, pp. 1319– 1327 [9]Szegedy, C., Ioffe, S., Vanhoucke, V., et al.: 'Inception-v4, Inception-ResNet and the impact of residual connections on learning'. Association for the Advance of Artificial Intelligence (AAAI), San Francisco, CA, USA, 2017 [10]Szegedy, C., Liu, W., Jia, Y., et al.: 'Going deeper with convolutions'. Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 1– 9 [11]Szegedy, C., Vanhoucke, V., Ioffe, S., et al.: 'Rethinking the inception architecture for computer vision'. Proc. of the Institute of Electrical and Electronics Engineers Conf. on Computer Vision and Pattern Recognition (IEEE CVPR), Las Vegas, NV, USA, 2016, pp. 2818– 2826 [12]He, K., Zhang, X., Ren, S., et al.: 'Deep residual learning for image recognition'. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016 [13]Huang, G., Liu, Z., Weinberger, K.Q., et al.: 'Densely connected convolutional networks'. Proc. Institute of Electrical and Electronics Engineers Conf. on Computer Vision and Pattern Recognition (IEEE CVPR), Honolulu, HI, USA, 2017, pp. 2261– 2269 [14]Krizhevsky, A., Sutskever, I., Hinton, G.E.: 'Imagenet classifification with deep CNNs'. Proc. of Conf. and Workshop on Neural Information Processing Systems (NIPS), Vancouver, Canada, 2012, pp. 1097– 1105 [15]Ioffe, S., Szegedy, C.: 'Batch normalization: accelerating deep network training by reducing internal covariate shift'. Int. Conf. on Machine Learning (ICML), Miami, FL, USA, 2015 [16]Xie, S., Girshick, R., Dollár, P., et al.: 'Aggregated residual transformations for deep neural networks'. Proc. Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5987– 5995 [17]Huang, G., Sun, Y., Liu, Z., et al.: 'Weinberger: deep networks with stochastic depth'. Proc. of European Conf. on Computer Vision (ECCV), Amsterdam, Netherlands, 2016, pp. 646– 661 [18]Glorot, X., Bordes, A., Bengio, Y.: 'Deep sparse rectififier neural networks'. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA, 2011 [19]LeCun, Y., Jackel, L.D., Bottou, L., et al.: 'Comparison of learning algorithms for handwritten digit recognition'. Int. Conf. on Artifificial Neural Networks (ANN), Bratislava, Slovakia, 1995, pp. 53– 60 [20]Krizhevsky, A., Hinton, G.: ' Learning multiple layers of features from tiny images'. Technical report, University of Toronto, Toronto, 2009 [21]Netzer, Y., Wang, T., Coates, A., et al.: 'Reading digits in natural images with unsupervised feature learning'. Deep Learning and Unsupervised Feature Learning Workshop, Conf. and Workshop on Neural Information Processing Systems (NIPS), Granada, Spain, 2011 [22]Goodfellow, I.J., Bulatov, Y., Ibarz, J., et al.: 'Multi-digit number recognition from street view imagery using deep CNNs', Comput. Sci. Conf. on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA, 2013, pp. 1– 13 [23]Chollet, F.: 'Xception: deep learning with depthwise separable convolutions'. Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017 Volume14, Issue16December 2020Pages 4385-4391 FiguresReferencesRelatedInformation