Data-oriented Scene Recognition

Abstract: Most deep learning backbones are evaluated on ImageNet. Using scenery images as an example, we conducted extensive experiments to demonstrate the widely accepted principles in network design may result in dramatic performance differences when the data is altered. Exploratory experiments are engaged to explain the underlining cause of the differences. Based on our observation, this paper presents a novel network design methodology: data-oriented network design. In other words, instead of designing universal backbones, the scheming of the networks should treat the characteristics of data as a crucial component. We further proposed a Deep-Narrow Network and Lossless Pooling module, which improved the scene recognition performance using less than half of the computational resources compared to the benchmark network architecture ResNets. 

Reviews and author responses:

The paper suggests that for certain kinds of data, width is much less important to a network's performance than depth. This assumption, the authors claim, is a result of overfitting or over-reliance on ImageNet as a benchmark. Experimental results show width being much more important to the task of object classification than scene recognition. To investigate this finding, the authors propose and evaluate two different hypotheses, and then propose a deep and narrow architecture based on the experimental results. The paper finds an interesting trend between sensitivity of architectures to width and depth, and the type of data being trained on. The proposal of hypotheses and attempt to disprove these hypotheses through experiments is also interesting.

However, it's unclear to me whether or not these trends hold in general. Only two datasets and one class of architectures were used to establish this trend. Furthermore, these correlations seem tenuous at best to me and the arguments are much too "hand-wavy." Intuitively, I could entertain that width contributes to "more fine-grained features." But depth being related to "spatial information" because of growing receptive field doesn't make any sense. I may be wrong, but I think even Resnet18 has a receptive field that is larger that the standard 224x224 size images in ImageNet. Thus, increasing depth really shouldn't give any benefits to "spatial information processing." If anything, depth is commonly related to feature "complexity" (for example, going deeper in a network we find neurons that excite on edges, curves, and face detectors).

In addition, the presented architecture lacks novelty. It seems to be a Resnet, but with a different width and depth. This amounts to hyperparameter search. There are also problems with the proposed "lossless pooling" layer. Most critically is that the lossless pooling layer is essentially a fixed 3x3 convolution with 1/4 in the corner of the kernel and zeroes elsewhere. This means that this operation, if it is important to the downstream prediction, should simply be learned by the convolutional layers, rendering this layer useless. The maxpooling of Resnets (which is incorrectly diagramed in Figure 5) does not suffer from this problem because it cannot be parameterized by a convolution. In addition, the maxpooling operation only happens once in the Resnet architectures, so it's not clear to me how often lossless pooling is being used in the proposed "deep-narrow" architecture.

Finally, the writing of the paper would benefit greatly from a revision. The exposition of ideas is somewhat disorganized, leaving the reader confused. Terms are ill-defined and hand-wavy and the reader is often left guessing until well into a section (for example, in section 4.2, it is not even clear we're considering classification until paragraph 4).  The paper presents an interesting empirical trend, conducts experiments in an attempt to understand the trend, and proposes and architecture based on these trends. The presentation of the ideas is disorganized, the existence of the empirical trend is debatable, and the proposed architecture lacks novelty. For this reason, I cannot recommend acceptance of the paper in its current state.

This work raises an interesting concern about the data type used in the validation of CNN architectures being a critical component in the design decisions of these architectures. 

The authors perform and present a study to analyze if commonly used design strategies (deeper and wider CNNs bring benefits in performance) apply similarly regardless of the type of data used (in particular, the study is focused on object-centric image classification, validated with the ImageNet benchmark, vs scene image classification, i.e. scene recognition, validated with the Places365 benchmark). 

The presented study concludes the authors hypothesis that the scene-recognition task has different requirements. It needs more focus on spatial features than object-centric image classification, and the authors propose an alternative CNN architecture design, narrower and with a novel  pooling strategy to preserve spatial information, which achieves similar performance than baseline networks with much less computational resources.
 Strengths:
+ the raised concern is very interesting. Typically image classification architectures are validated on object-centric datasets (mostly ImageNet), and the design decisions concluded may not be valid for other types of datasets (such as scenes).
+ the paper is well organised and clearly states the motivation and the contributions presented
+ the related work makes a nice summary of the image classification architectures and the most significant design steps generally accepted to be beneficial for the final performance.
+ authors propose an alternative network design (deep-narrow + lossless pooling networks) that achieves similar accuracy than the baseline architecture (ResNet), slightly smaller loss of accuracy in Place-recognition data than in object-centric data. The most significant benefit is the saving of computational resources (although this is consistent regardless of the type of datasets).



Weaknesses.
The main weakness I find is about the experimental validation. It  needs additional details to facilitate repeatibility and should be more thorough to bring more solid conclusions:

- About the differences between results obtained for ImageNet and Places365 in the first experiment, there is a possibility that they could be originated by other reasons that those discussed, such as number of classes or certain training issues.  In the experimental validation, it’s not only the type of data that is different, but also the number of classes differs (1000 vs 365 vs 200)). Some explicit discussion on this would be interesting for a more complete analysis. Having to discriminate more or less classes could bring similar issues or slightly bias the results obtained in the first experiment (sec 4.1.).  

- Actually, In the second experiment (sec 4.2), a random set of 100 clases (which ones exactly?) is selected to make the comparisons more fair. Why not to run several “samplings” and then average the results? This would avoid bias towards the sampled classes on the only execution. This bias could be causing the differences.  Besides, maybe the first experiment (sec 4.1.) could also follow a similar strategy to balance number of classes?

- The insights highlighted about the proposed architecture (Deep-Narrow + Lossless pooling) are not very convincing, in the sense that discussion on results in Table 4 and 5 focus on the drop of performance using ImageNet, which is not that much larger than performance in Places365 (around 1% vs 0.1%) to make a strong statement. What seems a significant change, I would say, is the change in computational resources with respect to the baseline, but this is similar regardless of the data type used.  I find the tendency of different effect on Places and ImageNet data can be interesting to mention, but the overall strong conclusion I see is that in both cases the Deep-Narrow architecture obtains almost the same quality of results with half of the resources. For completion, it would be nice to see Table 6 results also with ImageNet if possible.


Typos and smaller fixes/issues:
- page 1, last paragraph: “complex view THAT consists of  …”
- Table 1: ImagiNet —> ImageNet
- Table 1 and 2: “number of parameters based on …”
- page 7, first paragraph: “effective and effective option”? 
- page 7, actuary —> accuracy

- Fig. 4, I would please label the axis within the plot. I find it’s easier for the reader to have the info there
- When describing in the text the increases in Table 5 I think there is an inconsistency when saying +0.59% -0.48% with respect to the numbers shown in the Table 5. I currently tend to reject. The raised concerns about bias to the validation data used for image classification CNNs are interesting. However, the experimental design to validate the hypothesis posed is not thorough enough to extract solid conclusions. Currently the more clear conclusion is about decisions towards saving on computational resources, compared to insights on the central topic, i.e., differences regarding the data type.

The paper describes experiments with networks with different depth and width in object-centred datasets (ImageNet), scene datasets (Places), fine-grained (CUB). The authors claim to have found empirical evidence in favour of having deep and width adjusted to the dataset. Also, a lossless pooling module is presented. Strengths:
- The paper tries to establish a relationship between CNN's depth, width and characteristics of image datasets in terms of number of objects and granularity, which is welcome
- The paper contains several experiments in different datasets
- Experiments with low/high pass images are interesting ideas
- It presents a novel method to reduce lateral dimensionality of feature maps, by reshaping the tensor instead of summarizing (average) or selecting (max) data.

Weaknesses:
- The empirical results are not sufficient to support the claims. When inspecting the results, the impacts of changing width and depth of networks are quite similar in all datasets. There are differences, but all datasets showed increase or decrease in accuracy at the same time for practically all experiments. The fact that a given dataset suffered more than others in a particular experiment may not be linked directly to the conclusions, but could come from other causes
- The proposed pooling method is not actually a pooling in the sense of subsampling, but a rearrangement/reshaping of the feature maps tensor. Its result, when compared to the regular approach, is not significantly different.
- The experiments with high/low pass filters is interesting, but insufficient to allow new insights on the problem. It seems strange to me that the accuracy start at a low value (5 % for low pass and 20% for high pass) for a small kernel, and significantly increases for stronger filters, up to 30x30! If I could guess, I would say there is something wrong. Also, I could not draw the same conclusions as the authors from this experiment, since the curves are quite similar when comparing Fig4 a and b. Places suffer more from low pass filter, but the accuracy is already so low that it may not really indicate something useful. Although the paper presented some effort in the study of width x depth related to different image datasets, there are no empirical evidence to support the claims with a reasonable confidence. Also, the proposed pooling do not appear to make a positive effect on learning. For those reasons, I believe the paper should be rejected since it does not bring any new insights apart from what is already known in the literature about those topics.

The paper analyzes the performance of the ResNet architectures on the ImageNet and Places365 and based on the information found it proposes a new architectures is shows improved performance The paper starts by analyzing the performance of the ResNet architectures on the standard benchmarks of ImageNet and Places365. The critical issue here is that the reported performance is for the baseline ResNet versions, while, currently, the top performance on these benchmarks is dramatically higher  (https://paperswithcode.com/sota/image-classification-on-imagenet). For instance one may note that "Fixing the train-test resolution discrepancy: FixEfficientNet" report higher performance with 9m params. 
The paper follows by proposing the narrower structure. It is rather unexpected as it goes in the opposite direction of the "wide" resNet, which is a widely accepted contribution. 
The paper  shows that by narrowing, the performance of the Resnet is improved. The paper makes a critical point in its argumentation based on the numerical evaluation. Yet the evaluation should take into some of the more recent contributions which improve the standard performance and show that in such a condition, the narrowing is efficient. 
Overall the issues are
 - for this paper, the numerical results are critical. The prior art taken into account should be more recent. The paper might be fine, if it would have been in the same time with WideResNet (2016), but there have been passed 5 years.
- wideresnet should not miss from comparison
- given that "deep narrow networks" has 12 M params, the comparison should include architectures with 12 M params such as ResNet 18, the appropriate versions of Dense, SeNet, etc. Also some of these should be used in the context of recent improvement such as FixEfficientNet, MetaLabels, Self Training, etc. 
 - Alternatively, the paper might show that the proposed improvement can be efficiently combined with other improvements and the performance can be one of the top  The innovation is rather incremental (make the networks narrower) and the supporting evidence is not convincing. The paper has to improve in at least one of the directions

Meta-review: All reviewers recommended reject. No responses from the authors.

Decision: Reject