See More for Scene: Pairwise Consistency Learning for Scene Classification

1 Introduction . Image classification is one of the fundamental tasks of computer vision , and attracts a lot of attention as a popular task for facilitating the improvement of convolution neural network ( CNN ) [ 16 , 23 , 26 , 11 , 14 ] . As a valuable classification subtask , scene classification has its own characteristics which still needs further study . In comparison with objects , scenes are more complex and have many differences . One of main differences is that the characteristics of scenes are basically distributed over the whole image , while those of objects are confined within a clear boundary . This phenomenon inspires a core idea of scene classification , “ seeing ” comprehensive and informative regions in the image . Most scene classification methods aim to extract unspecific [ 9 , 25 , 6 ] or specific regions [ 29 , 33 , 3 ] , then aggregate them via statistical models [ 9 , 6 , 29 ] or relation modeling approaches [ 25 , 3 ] . The main challenge is that the extraction and aggregation are partially independent from the main CNN backbone . It induces some issues as follows . 1 ) some sort of incompatibility caused by the separated 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ways to extract and represent regions . 2 ) inevitable computational consumption from extra region extraction and aggregation processes . It is less studied to investigate the inherent properties of CNN and make them adaptive to the needs of scene classification in a more organic and effective way . In this paper , we explore a new way to understand scene images and the CNN models for scene classification in terms of the focus area . It is defined as the region consisting of pixels with large aggregated activation values in the feature maps . From this perspective , we obtain some observations that reveal the differences in the characteristics of CNNs between scene and object classification , which also correlates well with the properties of these two kinds of images , i.e. , scenes are more complex with richer objects . One of the interesting findings is that some properties of training scheme could drive the development of large focus area of the corresponding CNN model , which raises a question about how to optimize the training strategy by considering the scene characteristics . In order to take advantages of the scene characteristics , we propose a new learning scheme with a tailored loss for scene classification . Our goal is to inspire the CNN models to focus on more regions in the feature maps . However , the supervision of potentially useful regions is difficult to be obtained . In alternative , we attempt to erase the already activated regions ( from the images ) and require the consistent outputs , thus , the CNN models themselves could expand the focus area with further optimizing . This is also motivated by our preliminary observations that a scene image can retain its semantic meaning even when some regions are masked . Specifically , the modified image is generated by partially erasing the input images with the guidance of model preference . The tailored loss is implemented as pairwise consistency of predictions , which can leverage the category-relevance information . The principle behind our proposed method is the adversarial learning mechanism under the setting of pairwise consistency . Compared to the original image allows the unconfined exploration , the modified image is used for discovering additional parts which are previously ignored in the original branch . The consistency constraint with the shared parameters unifies these two modes of region explorations , leveraging the underlying opposing operations to perform a more comprehensive and robust information extraction process . In addition , instance-level pairwise consistency shares the excellent quality of knowledge distillation , and yields the unchanged models with superior performances . Experiments on one of the largest scene datasets , Places365 [ 38 ] , demonstrate the effectiveness of the proposed method with various CNNs by showing the significant performance improvements . Some statistical analyses about the focus area indicate that our proposed method can enhance the capability of exploring larger focus area in CNNs and fit the characteristics of scene images . We also evaluate our method on ImageNet [ 15 ] , and the inferior result can be the evidence to reveal the inner working mechanism that is specifically designed for scene images . 2 Related work . Scene classification . In the era of CNN , many works [ 9 , 5 , 7 , 18 , 3 , 31 ] explore various region extraction and encoding methods based on CNN representations to improve scene classification . They generally need lots of extra overhead compared to a single CNN classifier and are not easy to apply on large scale datasets . A few researches focus on specific deep models designed for scene classification , including Dictionary Learning CNN [ 19 ] , spatial unstructured layer [ 10 ] , and contextual relation learning [ 40 ] . However , we aim to investigate the inherent properties of scene CNN model , and propose an advanced training method to enhance it without any change of architecture . Comparisons of scene and object networks . A growing number of works have been proposed to understand the discrepancy between these two kinds of networks , especially in terms of the learned representations . In [ 39 ] , the mean image method is used to visualize the units of CNN layers , and visually exhibits the relevance between the unit structures and the differences of the training data in a qualitative way . Subsequently , a series of works tried to align the units of the CNN with some semantic concepts by measuring the empirical receptive fields [ 36 ] or using the feature activation as the segmentation results [ 1 ] , and found that the networks trained to recognize scenes generally have more interpretable units related to certain concepts , especially objects . In addition , Herranz et al . [ 13 ] investigated the representations of CNNs trained on ImageNet and Places365 by comparing the object scale ranges in these two datasets , and suggested the transfer performances could be improved when choosing the suitable image sizes . In contrast , we analyze the feature maps by estimating the aggregated activation values and providing the new perspective of the focus area . Moreover , the training schemes are also introduced and compared in terms of the focus area for better understanding of the difference in the learned CNN models . Erasing methods in CNN . Erasing regions or pixels in images or feature maps is an effective method in weakly supervised object localization [ 24 , 28 , 32 , 4 ] / segmentation [ 17 ] , due to the observation that the models often only concentrate on small and most discriminative regions of object when supervised by only image-level labels . Our idea is inspired by this but study further the properties of CNN from the perspective of the focus area , which changes previous empirical conclusion based on some examples to a robust and reliable statistical analysis . In addition , we explore the erasing method with a tailored loss and extending them into classification task with superior performances , which enriches the understanding of the intrinsic correlations between the erasing method and image classification . 3 Comparisons of scene and object networks via the focus area . A large number of previous image classification works propose various technical changes of CNN , and evaluate them on ImageNet [ 15 ] , which is an object-centric dataset . Despite the proposed large-scale scene-centric datasets , Places205 [ 39 ] and Places365 [ 38 ] , there is still a lack of researches on them , due to the underlying assumption that effective models on object classification could be generalized to scene classification . However , there are always some obvious differences between object and scene images . In this case , scene classification should be investigated individually compared to object classification . Here , we provide a new perspective to analyze the differences between scene and object networks by estimating the focus area of model on the input images . When the CNN models are trained , it is possible to find out the regions , which the model concentrates on and extracts information from , to make final decision . Various techniques have been developed to achieve this goal by using occluded images [ 39 ] , gradient information [ 22 ] , or architecture tweaks [ 2 ] . The success of these attempts motivates us to analyze the distributions of focus area of models trained on scene and object datasets , and figure out the possible similarities and differences between them . Based on the feature maps from last convolution layer , we adopt a simple adaptive thresholding method to discover the focus area of model on an input image . The channel-wise average pooling are performed on the feature maps to generate a heatmap . Then the mean value of this heatmap is used as the threshold to segment it into a binary map . Finally , the focus area is composed of the pixels with positive value in the binary map . The underlying assumption is that the feature maps can serve as an indicator to point out the regions which models desired [ 30 , 27 ] , and the pixels with high activation values in the feature maps have more valuable information [ 1 ] . The two variables , considered in our analyses about focus area , are dataset and training scheme . In terms of dataset , ImageNet and Places365 are used as object-centric and scene-centric datasets , respectively . Additionally , we introduce two training schemes , including simple and complex ways . The main difference between them is the crop percent in the input processing operations . The former fixes the percent to 76 % ( 224×224 / 256×256 ) , while the latter chooses a random percent in range [ 8 % , 100 % ] and is widely used in modern CNN training process . The detailed description of these training schemes could be found in appendix . Given a focus area , we compute the coverage ratio of it to the whole image as an analytical metric . As the focus area on a single image may be too specific and not informative , we gather the validation datasets of ImageNet and Places365 to form an evaluation set for analyzing the distribution of the coverage ratio . We present the classification performance and the visualization of the distribution of the coverage ratio across datasets and training schemes . Referring to the box plots in Figure 1 , it is clearly shown that the model learned on Places365 tend to focus more parts of the input image than ImageNet when comparing the median and mean of the data distribution , and the result holds for both training schemes . It is likely that the model trained on Places365 learns the mechanism of comprehensive exploration and activation as a necessary condition for recognizing a scene . Recalling the definition of the focus area , the pixels in it all have the aggregated activation values than the mean value , which indicates that they either have large activation in a few channels or be frequently activated in many channels . Some evidences [ 8 , 35 ] have shown that each channel ( or unit ) in feature maps could be regarded as a part or a full of a detector for a semantic concept , and the concept regions emerge as the pixels with large activation values . In this way , the focus area roughly represents the group of detected semantic concepts in each image , suggesting that the size of the focus area can be accepted as a quantitative index of the regions with semantic concepts . By this correlation , the preference for large focus area seems to be a natural consequence of the scene attributes ( containing more semantic concepts ) . Another intriguing finding is the effect of training scheme . In contrast with complex training , simple way presents the capability of enhancing the region exploration of model and yielding large focus area on more examples , as shown in Figure 1 . We also show the classification performances of two training schemes on ImageNet and Places365 in Table 1 . It concludes that simple training is a competitive choice for Places365 , while it works worse on ImageNet . We infer that complex training adopts the cropped patches with various scales and yields more diverse training examples , which is better for object classification . However , the similar performances of them on Places365 indicate that the ability of simple training to explore regions is useful for improving the performances , and could bridge the gap caused by the stronger data augmentation . This performance difference also raises question about the optimal training method considering scene characteristics , and motivates us to design a new learning framework as the effective response .

Summary: In this work, the authors enlarge effective receptive fields (ERF) of CNNs for scene image classification. To this goal, the authors propose a new learning scheme and a tailored loss. The proposed learning target is to make the outputs of an input image and its modified version to be consistent. The tailored loss is implemented as pairwise consistency of prediction to leverage category-relevance information. Experiments on Places365 partly verify the effectivenss of the proposed methdods.