Data-oriented Scene Recognition

1 INTRODUCTION . Since the development of AlexNet ( Krizhevsky et al. , 2012 ) , a number of variations of deep Convolutional Neural Networks ( CNNs ) emerged . Motivated by the success of studies ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2016 ; He et al. , 2016a ) , the networks become deeper by adding more convolutional layers to improve performance for the targeted problems . On the other hand , arguments on the benefits of increasing network width gained support from researchers . Zagoruyko et al . ( Zagoruyko & Komodakis , 2016 ) presented wider deep residual networks that significantly improve the performance over ResNet ( He et al. , 2016a ) . The wider deep network achieved state-ofthe-art performance on ImageNet ( Deng et al. , 2009 ) and CIFAR ( Krizhevsky et al. , 2009 ) , showing consistently better accuracy than its ResNet counterparts . Xie et al . ( Xie et al. , 2017 ) proposed a multi-branch architecture called ResNeXt by widening the residual blocks and cooperating group convolution . In contrast to ResNet , ResNeXt ( Xie et al. , 2017 ) demonstrated a performance boost on a larger ImageNet-5K set and the COCO object detection dataset ( Lin et al. , 2014 ) . ResNeSt ( Zhang et al. , 2020 ) preserved the wider network layout and multi-branch strategy and introduced a modulated architecture to improve the feature learning process . The proposed ResNeSt networks further improved the performance on the ImageNet dataset . Despite the benefits brought by increased depth and width ( i.e. , number of channels ) , side effects such as vanishing gradient and requirement of large number of training examples make it difficult to employ a deep network for learning from complex scenery images . Is a deeper network more suitable for extracting features from scenery images for better understanding complex views ? Or shall we employ more channels to achieve improved performance ? To answer these questions , we need to have a better understanding of the functions of network layers and channels . A number of studies have been conducted in the past years . Lu et al . ( Lu et al. , 2017 ) argued that an integration of both depth and width provides a better understanding of the expressive power of neural networks . Tan and Le ( Tan & Le , 2019 ) showed that it is critical to balance the network depth and width by maintaining a constant depth/width ratio and demonstrated the effectiveness of their approach on ResNet and MobileNet . Besides manually designed networks , deep Neural Architecture Search were proposed to optimize the network depth and width ( Zoph & Le , 2017 ; Guo et al. , 2020 ) . However , most methods , if not all , were developed and evaluated using ImageNet ( Deng et al. , 2009 ) and CIFAR ( Krizhevsky et al. , 2009 ) . These data sets commonly depict an object near the center of the image , and the label tells what the object is . That is , such an image is mostly dominated by one object and , hence , is referred to as “ object-centric ” . On the other hand , a scene image presents a complex view consists of multiple objects and background clutters . This inadvertent data bias could potentially lead to the ignorance of the characteristics of different data . The features that are crucial for object recognition dominantly affect the design of the CNNs . Deep networks with more layers use a variety of receptive fields to extract distinctive scale features whereas networks with more channels capture fine-grained patterns ( Tan & Le , 2019 ) . However , in many applications , both types of information that are crucial for accurately recognizing an image , but the prominence difference between them is seldom explored . This paper attempts to bring out a new perspective on neural network design : the data itself has the preference . Learning the overall spatial layout is crucial to recognize the entire scene , thus scene recognition favors the networks that can better learn the spatial information . For the object-centric images , typical examples only consist of one single object , the spatial layout does not contribute much to the semantic meaning of the image . As the differences between certain object categories are subtle , the detailed patterns and textures of objects are likely to be more representative . The networks that emphasize learning various features can better fit the requirement of object recognition tasks . By considering the distinct characteristics of scene images , our hypothesis is that for scene recognition , learning spatial-wise information improves the performance of CNNs in a more effective manner compared to learning channel-wise information . To evaluate this hypothesis , we conducted comprehensive experiments and our results show scene images gain clear benefits from deepening the network , and the performance change caused by altering the width is marginal . We further proposed a Deep-Narrow Network , which increases the depth of the network as well as decreased the width of the network . We design a Lossless Pooling component and use it in our Deep-Narrow Network to extract spatial features . 2 RELATED WORK . Deep Networks Network depth has played an integral role in the success of CNNs . With the increase of depth , the network can better approximate the target function with richer feature hierarchies , which enables the boost of performance . The success of VGGNet ( Simonyan & Zisserman , 2015 ) and Inception ( Szegedy et al. , 2015 ; 2016 ; 2017 ) on ILSVRC competition further reinforced the significance of the depth . ResNet ( He et al. , 2016b ) , which is a continuance work of deeper networks , revolutionized the possible depth of deep networks by introducing the concept of residual learning and identity mapping into CNNs . ResNet ’ s effective methodology enables the network to be extremely deep and demonstrated improved performance in image recognition tasks . Wide Networks Network width has also been suggested as an essential parameter in the design of deep networks . Wide ResNet ( Zagoruyko & Komodakis , 2016 ) introduced an additional factor to control the width of the ResNet . The experimental results showed that the widening of the network might provide a more effective way to improve performance compared to making ResNet deeper . Xception ( Chollet , 2017 ) can be considered as an extreme Inception architecture , which exploits the idea of depth-wise separable convolution . Xception modified the original inception block by making it wider . This wider structure has also demonstrated improved performance . ResNeXt ( Xie et al. , 2017 ) introduced a new term : cardinality to increase the width of ResNet and won the 2016 ILSVRC classification task . With the success of ResNeXt , it is widely accepted that widen the deep network is an effective way to boost model performance . A most recent wider network : ResNeSt ( Zhang et al. , 2020 ) preserved the wide architecture of ResNeXt and achieved superior performance on image and object recognition tasks . Effects of Depth and Width Although depth and width are proven to be essential parameters in network architecture design , the effect of depth and width , i.e. , what do deep and wide networks learn remains seldom explored . Most of the existing literature focus on the effect of width and depth separately or the trade-off between depth and width in the network design ( Lu et al. , 2017 ) . Tan and Le ( Tan & Le , 2019 ) claimed that deep networks can make use of a larger receptive field while wide networks can better capture fine-grained features . Nguyen et al . ( Nguyen et al. , 2021 ) explored the effects of width and depth and found a characteristic structure named block structure . They demonstrated that for different models , the block structure is unique , but the representations outside the block structure trends to be similar despite the setting of depth and width . In our paper , we analyze the effect of depth and width in CNNs from the perspective of image characteristics . 3 EXPERIMENTAL STUDY . 3.1 DATA SETS AND EXPERIMENTAL SETTINGS . To understand the impact of network structure to the data set and ultimately the applications , we use ImageNet 2012 ( Deng et al. , 2009 ) and Places Standard dataset ( Zhou et al. , 2017 ) as our evaluation data sets . ImageNet 2012 is the benchmark object recognition data set that consists of 1,000 classes and 1.28 million training images . An image in ImageNet 2012 usually contains a single object that is highly distinctive from the background . Place365 Standard dataset is designed for scene recognition and contains 1.8 million training images of 365 classes . The images in Places365 Standard datasets present more complex scenery images . We train deep network models and compute the single-crop ( 224 × 224 pixels ) top-1 and top-5 accuracy based on the application of the models to the validation set . We train each model for 100 epochs on eight Tesla V100 GPUs with 32 images per GPU ( the batch size is 256 ) . All models are trained using synchronous SGD ( Stochastic Gradient Descent ) with a Nesterov momentum of 0.9 and a weight decay of 0.0001 . The learning rate is set to 0.1 and is reduced by a factor of 10 in every 30 epochs . In the training of ResNet and its variants , we follow the settings in ( He et al. , 2016b ) . 3.2 COMPARISON ON DIFFERENT DATA SETS . We conducted our comparison study using ResNet and its variants on Places365 and ImageNet data sets . The results are reported in Tables 1 ( varying network depth ) and 2 ( varying network width ) . By increasing the network depth from 50 to 101 , i.e. , ResNet-50 and ResNet-101 , we obtained a performance improvement of 1.40 % and 2.32 % on Place365 and ImageNet data sets in terms of Top-1 accuracy . Theoretically , if the widening of the network is more effective to improve the performance as stated in the previous literature , we should expect more accuracy increase when we double the width of the networks . However , doubling the width leads to a top-1 accuracy increase of 3.28 % on ImageNet , but only 0.94 % on Place365 . More surprisingly , for ResNeXt , which also doubled the network width , the relative performance increase on ImageNet is 2.34 % in terms of top-1 accuracy , but the number is only 0.14 % on Place365 . The trend that the model performance on ImageNet is more sensitive to width change compared with Place365 is also true when we decrease network depth or narrow down the width . When the network depth decreased from 50 to 18 , ImageNet suffered a 7.23 % relative top-1 accuracy decrease , and for Place365 it is 2.71 % . , that is , the top-1 performance drop on ImageNet is around 2.7 times of the top-1 performance drop on Places . But when we decreased the width of ResNet-50 to half of the original size , the number changed to 4.7 . The statement that widening of the network might provide a more effective way to improve performance is biased towards ImageNet ( object-centric data ) and ignores the characteristics of scenery images . 4 ANALYSIS OF THE IMPACT OF DEPTH AND WIDTH OF DEEP NETWORKS . 4.1 COMPLEXITY OF IMAGES . Our first hypothesis is the performance difference is caused by the complexity of scenery images . This hypothesis is aroused by the distinct complexity difference between scene images and object- centric images : object-centric images always only contain one major object which occupies a large portion of the view , scene images always consist of multiple objects and background clutters . Figure 1 shows two samples from benchmark object-centric data set ( ImageNet ) and benchmark scene data set ( Place365 ) , respectively . Figure 1 ( a ) is labeled as “ bald eagle ” , in which the eagle stands in the center of the view and occupies a large portion of the entire image ; figure 1 ( b ) is labeled as “ forest-broadleaf ” , the entire view consists of not only a bird but also tree branches and leaves . As the correct recognition of scenery images relies on multiple components , a scene image is typically considered more complex than an object-centric image . To evaluate this hypothesis , we introduced another data set that is also widely believed to be “ complex ” : the fine-grained image classification data set . Fine-grained classification is considered to be a more complex task as the classes in the data set can only be discriminated by local and subtle differences . CUB-200-2011 is a data set consists of 200 different species of birds , which serves as a benchmark data set for fine-grained classification tasks . Figure 1 shows two samples in the CUB200-2011 data set . In Figure 1 , black-footed albatross ( c ) and sooty albatross ( d ) are considered to be two different categories in classification . The two albatrosses are similar in appearance , and the differentiating of them is challenging due to the subtle traits that characterize the different species are not straightforward . We conducted comparison experiments on two “ complex ” data sets ( Places365 and CUB-200-2011 ) and one “ simple ” data set . The results are shown in Table 3 . Using the benchmark ResNet-50 as the backbones , we observed that on CUB-200-2011 , the relative top -1 accuracy increased by 1.81 % when doubling the width and dropped 4.28 % when we narrowed the width to half of the original . This performance change caused by altering the width is much acute compared to the result on Place365 ( 0.94 % and 1.11 % , doubling and halving the width respectively ) under the same settings , which demonstrated that a wider network is able to effectively enhance the recognition of “ complex ” fine-grained features . This result does not agree with our first hypothesis : if the performance difference is originated from the complexity of the data , we should observe a moderate performance change on CUB-200-2011 data set along with the changing of network width . The results proved that the performance variance on different data set is not the consequence of the complexity ( rich fine-grained details ) of the data .

Summary: This work raises an interesting concern about the data type used in the validation of CNN architectures being a critical component in the design decisions of these architectures.   The authors perform and present a study to analyze if commonly used design strategies (deeper and wider CNNs bring benefits in performance) apply similarly regardless of the type of data used (in particular, the study is focused on object-centric image classification, validated with the ImageNet benchmark, vs scene image classification, i.e. scene recognition, validated with the Places365 benchmark).   The presented study concludes the authors hypothesis that the scene-recognition task has different requirements. It needs more focus on spatial features than object-centric image classification, and the authors propose an alternative CNN architecture design, narrower and with a novel  pooling strategy to preserve spatial information, which achieves similar performance than baseline networks with much less computational resources.