Data-oriented Scene Recognition

1 INTRODUCTION . Since the development of AlexNet ( Krizhevsky et al. , 2012 ) , a number of variations of deep Convolutional Neural Networks ( CNNs ) emerged . Motivated by the success of studies ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2016 ; He et al. , 2016a ) , the networks become deeper by adding more convolutional layers to improve performance for the targeted problems . On the other hand , arguments on the benefits of increasing network width gained support from researchers . Zagoruyko et al . ( Zagoruyko & Komodakis , 2016 ) presented wider deep residual networks that significantly improve the performance over ResNet ( He et al. , 2016a ) . The wider deep network achieved state-ofthe-art performance on ImageNet ( Deng et al. , 2009 ) and CIFAR ( Krizhevsky et al. , 2009 ) , showing consistently better accuracy than its ResNet counterparts . Xie et al . ( Xie et al. , 2017 ) proposed a multi-branch architecture called ResNeXt by widening the residual blocks and cooperating group convolution . In contrast to ResNet , ResNeXt ( Xie et al. , 2017 ) demonstrated a performance boost on a larger ImageNet-5K set and the COCO object detection dataset ( Lin et al. , 2014 ) . ResNeSt ( Zhang et al. , 2020 ) preserved the wider network layout and multi-branch strategy and introduced a modulated architecture to improve the feature learning process . The proposed ResNeSt networks further improved the performance on the ImageNet dataset . Despite the benefits brought by increased depth and width ( i.e. , number of channels ) , side effects such as vanishing gradient and requirement of large number of training examples make it difficult to employ a deep network for learning from complex scenery images . Is a deeper network more suitable for extracting features from scenery images for better understanding complex views ? Or shall we employ more channels to achieve improved performance ? To answer these questions , we need to have a better understanding of the functions of network layers and channels . A number of studies have been conducted in the past years . Lu et al . ( Lu et al. , 2017 ) argued that an integration of both depth and width provides a better understanding of the expressive power of neural networks . Tan and Le ( Tan & Le , 2019 ) showed that it is critical to balance the network depth and width by maintaining a constant depth/width ratio and demonstrated the effectiveness of their approach on ResNet and MobileNet . Besides manually designed networks , deep Neural Architecture Search were proposed to optimize the network depth and width ( Zoph & Le , 2017 ; Guo et al. , 2020 ) . However , most methods , if not all , were developed and evaluated using ImageNet ( Deng et al. , 2009 ) and CIFAR ( Krizhevsky et al. , 2009 ) . These data sets commonly depict an object near the center of the image , and the label tells what the object is . That is , such an image is mostly dominated by one object and , hence , is referred to as “ object-centric ” . On the other hand , a scene image presents a complex view consists of multiple objects and background clutters . This inadvertent data bias could potentially lead to the ignorance of the characteristics of different data . The features that are crucial for object recognition dominantly affect the design of the CNNs . Deep networks with more layers use a variety of receptive fields to extract distinctive scale features whereas networks with more channels capture fine-grained patterns ( Tan & Le , 2019 ) . However , in many applications , both types of information that are crucial for accurately recognizing an image , but the prominence difference between them is seldom explored . This paper attempts to bring out a new perspective on neural network design : the data itself has the preference . Learning the overall spatial layout is crucial to recognize the entire scene , thus scene recognition favors the networks that can better learn the spatial information . For the object-centric images , typical examples only consist of one single object , the spatial layout does not contribute much to the semantic meaning of the image . As the differences between certain object categories are subtle , the detailed patterns and textures of objects are likely to be more representative . The networks that emphasize learning various features can better fit the requirement of object recognition tasks . By considering the distinct characteristics of scene images , our hypothesis is that for scene recognition , learning spatial-wise information improves the performance of CNNs in a more effective manner compared to learning channel-wise information . To evaluate this hypothesis , we conducted comprehensive experiments and our results show scene images gain clear benefits from deepening the network , and the performance change caused by altering the width is marginal . We further proposed a Deep-Narrow Network , which increases the depth of the network as well as decreased the width of the network . We design a Lossless Pooling component and use it in our Deep-Narrow Network to extract spatial features . 2 RELATED WORK . Deep Networks Network depth has played an integral role in the success of CNNs . With the increase of depth , the network can better approximate the target function with richer feature hierarchies , which enables the boost of performance . The success of VGGNet ( Simonyan & Zisserman , 2015 ) and Inception ( Szegedy et al. , 2015 ; 2016 ; 2017 ) on ILSVRC competition further reinforced the significance of the depth . ResNet ( He et al. , 2016b ) , which is a continuance work of deeper networks , revolutionized the possible depth of deep networks by introducing the concept of residual learning and identity mapping into CNNs . ResNet ’ s effective methodology enables the network to be extremely deep and demonstrated improved performance in image recognition tasks . Wide Networks Network width has also been suggested as an essential parameter in the design of deep networks . Wide ResNet ( Zagoruyko & Komodakis , 2016 ) introduced an additional factor to control the width of the ResNet . The experimental results showed that the widening of the network might provide a more effective way to improve performance compared to making ResNet deeper . Xception ( Chollet , 2017 ) can be considered as an extreme Inception architecture , which exploits the idea of depth-wise separable convolution . Xception modified the original inception block by making it wider . This wider structure has also demonstrated improved performance . ResNeXt ( Xie et al. , 2017 ) introduced a new term : cardinality to increase the width of ResNet and won the 2016 ILSVRC classification task . With the success of ResNeXt , it is widely accepted that widen the deep network is an effective way to boost model performance . A most recent wider network : ResNeSt ( Zhang et al. , 2020 ) preserved the wide architecture of ResNeXt and achieved superior performance on image and object recognition tasks . Effects of Depth and Width Although depth and width are proven to be essential parameters in network architecture design , the effect of depth and width , i.e. , what do deep and wide networks learn remains seldom explored . Most of the existing literature focus on the effect of width and depth separately or the trade-off between depth and width in the network design ( Lu et al. , 2017 ) . Tan and Le ( Tan & Le , 2019 ) claimed that deep networks can make use of a larger receptive field while wide networks can better capture fine-grained features . Nguyen et al . ( Nguyen et al. , 2021 ) explored the effects of width and depth and found a characteristic structure named block structure . They demonstrated that for different models , the block structure is unique , but the representations outside the block structure trends to be similar despite the setting of depth and width . In our paper , we analyze the effect of depth and width in CNNs from the perspective of image characteristics . 3 EXPERIMENTAL STUDY . 3.1 DATA SETS AND EXPERIMENTAL SETTINGS . To understand the impact of network structure to the data set and ultimately the applications , we use ImageNet 2012 ( Deng et al. , 2009 ) and Places Standard dataset ( Zhou et al. , 2017 ) as our evaluation data sets . ImageNet 2012 is the benchmark object recognition data set that consists of 1,000 classes and 1.28 million training images . An image in ImageNet 2012 usually contains a single object that is highly distinctive from the background . Place365 Standard dataset is designed for scene recognition and contains 1.8 million training images of 365 classes . The images in Places365 Standard datasets present more complex scenery images . We train deep network models and compute the single-crop ( 224 × 224 pixels ) top-1 and top-5 accuracy based on the application of the models to the validation set . We train each model for 100 epochs on eight Tesla V100 GPUs with 32 images per GPU ( the batch size is 256 ) . All models are trained using synchronous SGD ( Stochastic Gradient Descent ) with a Nesterov momentum of 0.9 and a weight decay of 0.0001 . The learning rate is set to 0.1 and is reduced by a factor of 10 in every 30 epochs . In the training of ResNet and its variants , we follow the settings in ( He et al. , 2016b ) . 3.2 COMPARISON ON DIFFERENT DATA SETS . We conducted our comparison study using ResNet and its variants on Places365 and ImageNet data sets . The results are reported in Tables 1 ( varying network depth ) and 2 ( varying network width ) . By increasing the network depth from 50 to 101 , i.e. , ResNet-50 and ResNet-101 , we obtained a performance improvement of 1.40 % and 2.32 % on Place365 and ImageNet data sets in terms of Top-1 accuracy . Theoretically , if the widening of the network is more effective to improve the performance as stated in the previous literature , we should expect more accuracy increase when we double the width of the networks . However , doubling the width leads to a top-1 accuracy increase of 3.28 % on ImageNet , but only 0.94 % on Place365 . More surprisingly , for ResNeXt , which also doubled the network width , the relative performance increase on ImageNet is 2.34 % in terms of top-1 accuracy , but the number is only 0.14 % on Place365 . The trend that the model performance on ImageNet is more sensitive to width change compared with Place365 is also true when we decrease network depth or narrow down the width . When the network depth decreased from 50 to 18 , ImageNet suffered a 7.23 % relative top-1 accuracy decrease , and for Place365 it is 2.71 % . , that is , the top-1 performance drop on ImageNet is around 2.7 times of the top-1 performance drop on Places . But when we decreased the width of ResNet-50 to half of the original size , the number changed to 4.7 . The statement that widening of the network might provide a more effective way to improve performance is biased towards ImageNet ( object-centric data ) and ignores the characteristics of scenery images . 4 ANALYSIS OF THE IMPACT OF DEPTH AND WIDTH OF DEEP NETWORKS . 4.1 COMPLEXITY OF IMAGES . Our first hypothesis is the performance difference is caused by the complexity of scenery images . This hypothesis is aroused by the distinct complexity difference between scene images and object- centric images : object-centric images always only contain one major object which occupies a large portion of the view , scene images always consist of multiple objects and background clutters . Figure 1 shows two samples from benchmark object-centric data set ( ImageNet ) and benchmark scene data set ( Place365 ) , respectively . Figure 1 ( a ) is labeled as “ bald eagle ” , in which the eagle stands in the center of the view and occupies a large portion of the entire image ; figure 1 ( b ) is labeled as “ forest-broadleaf ” , the entire view consists of not only a bird but also tree branches and leaves . As the correct recognition of scenery images relies on multiple components , a scene image is typically considered more complex than an object-centric image . To evaluate this hypothesis , we introduced another data set that is also widely believed to be “ complex ” : the fine-grained image classification data set . Fine-grained classification is considered to be a more complex task as the classes in the data set can only be discriminated by local and subtle differences . CUB-200-2011 is a data set consists of 200 different species of birds , which serves as a benchmark data set for fine-grained classification tasks . Figure 1 shows two samples in the CUB200-2011 data set . In Figure 1 , black-footed albatross ( c ) and sooty albatross ( d ) are considered to be two different categories in classification . The two albatrosses are similar in appearance , and the differentiating of them is challenging due to the subtle traits that characterize the different species are not straightforward . We conducted comparison experiments on two “ complex ” data sets ( Places365 and CUB-200-2011 ) and one “ simple ” data set . The results are shown in Table 3 . Using the benchmark ResNet-50 as the backbones , we observed that on CUB-200-2011 , the relative top -1 accuracy increased by 1.81 % when doubling the width and dropped 4.28 % when we narrowed the width to half of the original . This performance change caused by altering the width is much acute compared to the result on Place365 ( 0.94 % and 1.11 % , doubling and halving the width respectively ) under the same settings , which demonstrated that a wider network is able to effectively enhance the recognition of “ complex ” fine-grained features . This result does not agree with our first hypothesis : if the performance difference is originated from the complexity of the data , we should observe a moderate performance change on CUB-200-2011 data set along with the changing of network width . The results proved that the performance variance on different data set is not the consequence of the complexity ( rich fine-grained details ) of the data .

Summary: The paper describes experiments with networks with different depth and width in object-centred datasets (ImageNet), scene datasets (Places), fine-grained (CUB). The authors claim to have found empirical evidence in favour of having deep and width adjusted to the dataset. Also, a lossless pooling module is presented.