Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: To explore the potential recognition ability of CNN for scenes, this paper presents a scene recognition framework based on the analysis of effective receptive field (ERF). This paper first analyzes the correlation between ERF and CNN models of different tasks. Then the authors propose a framework to erase the useful regions while emphasizing other regions and expanding receptive field during training. Experiments are conducted on Places365 and ImageNet. Results show that the proposed method is more effective for scenes than objects.
main_review: Image classification is a classical and fundamental task in computer vision. Many excellent models have been proposed for object classification (on ImageNet), but the typical researches of scene classification are relatively few. Therefore, it’s interesting to design typical models to exploit the characteristic of scene classification rather than following the same architecture of object recognition.

One interesting insight provided by this paper is that a scene image could be correctly classified even when some regions are covered. And based on this observation, the authors propose a framework to erase the “easier” regions so more other regions could be emphasized during training. Also, the receptive field could be expanded.

Some concerns:
-ImageNet dataset is not introduced. 
-As the reviewer claimed in Line 61, “while is not applicable to object classification,” is this an advantage of the proposed method or a disadvantage? And what’s the reason for this.
-In Table 1, which factor is more important to the performance, the number of parameters, or RF size? Also, it seems the gains on Top-1 are larger than Top-5. What’s the reason?
-There are many related results in different tables. Therefore, the differences between them also should be clarified. For instance, ResNet in Table 1 is 54.36, in Table 4 is 54.12. What are the differences between the two settings? 

Minor issues: 
- Line 30, “as follows.”-> “as follows:”
- It’s better to avoid repeated sentences, such as Line 50 and 177.
limitations_and_societal_impact: Yes, the authors have discussed the limitations in section 4.2, such as no change in inference and high cost in training.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigated the difference between object classification and scene classification in terms of the effective receptive field. This paper further proposed an erasing scheme guided by the class activation map (CAM) to help enlarge the effective receptive field of classification models during training. Both the original image and the erased image are used to produce the traditional cross-entropy training signal. Also, a KL-divergence term is used to ensure the pairwise consistency between the original image and the erased image. Experiments show minor improvements over baseline methods.
main_review: Originality:
Limited novelty. This paper is the first to investigate the difference between object classification and scene classification in terms of the effective receptive field. But how effective receptive field affects the performance of scene classification is not well studied, neither theoretically nor empirically. The proposed erasing-based training scheme has been used in other lines of research such as multimodal community. Hence the novelty of the proposed approach is limited.

Quality: 
Some of the main claims of this paper are simply based on intuition. More theoretical or empirical analysis is required.
1)	About the histograms of impact maps in Figure 2. First to speak, it is confusing to use different numbers of bins. Why is there a difference between 90 bins and 70 bins? Also, some of the charts don’t agree with the claim that ERF of models trained on scene images is larger, e.g. Row 1 Column 3. Lastly, the AUCs of Figure 2 are not consistent, why so?
2)	This paper is based on an intuition that larger ERF will improve the performance of scene classification. But this correlation is well studied. For example, how and to what extent will larger ERF improve performance?
3)	The correlation between the performance gain of the erasing-based training scheme and the enlarged ERF is not studied. This paper merely intuitively provided this chain of causation: erasing scheme during training -> more activated ReLU units -> larger ERF. More comprehensive analysis is required to draw a connection between ERF and scene classification.


Clarity:
This paper is generally easy to follow. However, Section 4.2 about the CAM-guided erasing scheme is quite confusing. How to multiply classification weights with CAM?

Significance:
Moderately significant. The proposed method outperforms various baselines. But the improvements are minor.

Main Weakness:
As mentioned earlier, the erasing-based training scheme is of limited novelty. More analysis is required to draw connections between larger ERF and better scene classification performance.

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors enlarge effective receptive fields (ERF) of CNNs for scene image classification. To this goal, the authors propose a new learning scheme and a tailored loss. The proposed learning target is to make the outputs of an input image and its modified version to be consistent. The tailored loss is implemented as pairwise consistency of prediction to leverage category-relevance information. Experiments on Places365 partly verify the effectivenss of the proposed methdods.
main_review: In this work, the authors should address the following concerns.

1.Contributions

In fact, the conceptions of ERF and pairwise learning are not new in current CV field, especically in image classification. Section 3 only list the existing works on ERF and most of the theories are from existing works. The authors only use Histogram to compute the distribution information. In  my view, this is not a contribution, but only a computing method. Besides, the proposed pairwise learning is more like a data augmentation method with erasing modules. In fact, the whole training method is very similar to existing weakly-supervised classification or segmentation methods. In the network structure, the CAM and region erasing methods have already been used in previous works. Thus, the techinical contributions are very limited, considering most of the used modules are from existing works. If there are any insigtful contributions, the authors can highlight them and compare with previous works.

2. Motivations

As far as we known, large ERF is helpful for many CV tasks. The authors aims to enlarge the ERF. However, the basic idea is just using existing methods. What are your essential insights? In my view, there are many methods to modify the image to enlarge ERF, for example recurrent cropping, multi-scale, pyramid structure,etc. How to ensure the effectiveness of the proposed histogram computation and erasing design? Besdies, the basic module is still CAM without any changes.

3.Experiments

First, through the whole experiments, I find that there are no comparisons with recent scene classification methods. In fact, the authors only compare with basic methods with different backbone networks as shown in Tab.1. The performance is not comparable to current SOTA results. Second, there are no full analysis on the key modules. For example, how about using ResNet-101/152 backbone? Why very different ERFs (435 vs 899) show very similar results in Tab.2? If the claims are correct, is it consistent to the experiments? Besides, there are no significant differences in performance of Tab. 6, considering the CPC and IPC. Third, there are no visual comparisions to verify the proposed modules, especially CPC, IPC, Erasing effects, etc. In overall, experiments are not very convincing for the claims.
limitations_and_societal_impact: The authors have not adequately addressed the limitations and potential negative societal impact of their work.  I suggest the authors add more computation comparisons such as FLOPS/SPEED. Besides, the proposed method should show some failture examples in classification and analysed why the model can predict the true labels under larger ERF.
review_title: null

Meta-review: This paper presents a systematic study on the potential recognition ability of CNN for scenes in terms of the effective receptive field (ERF).  It is interesting to see that the proposed model is based on exploiting the characteristic of scene classification rather than following the same architecture of object recognition. Moreover, all the reviewers agree that the proposed method provides a key contribution to the fundamental CV tasks.  


Decision: Accept (Poster)