How much Position Information Do Convolutional Neural Networks Encode?

Abstract: In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs.

Reviews and author responses:

This paper studied the problem of the encoded position information in convolution neural networks. The hypothesis is that CNN can implicitly learn to encode the position information. The author tests the hypothesis with lots of experiments to show how and where the position information is encoded.

Clarity:
This paper is interesting for me. It tries to understand the encoded position information that is easily ignored by researchers. I like adequate experiments with learned position information and position illustrations.

Experiments:
1. The paper mainly discussed the zero-padding and found it is the source of position information. How about other padding modes like constant-padding, reflection-padding, and replication-padding?

2. The partial convolution-based padding method [1] (padded regions are masked out) shows that its recognition accuracy is higher than the traditional zero-padding approach. Can you help investigate where the position information comes from for this case?

[1] Partial Convolution based Padding, https://arxiv.org/pdf/1811.11718.pdf.


Some of my concerns are well addressed by the author thus I upgrade my score.


We thank Reviewer 3 for the detailed feedback and we will further explain the question raised in the comment. 

We think the first question is about initialization, (cold or hot start). We also thought about a longer training procedure for the PosENet because it was trained from scratch. But we found that the training loss does not decrease after the first several iterations, the weight becomes saturated quickly. The training loss of the PosENet converges at 0.084 after the first epoch. Also, all the test losses (on natural images PASCAL-S or synthetic images BLACK, WHITE or NOISE) are the same as the training loss. This suggests that the prediction may be completely independent of the content of images.

We believe the reason behind this is that zero-padding delivers obvious boundary information, the transition between the zeros padded and the content. As discussed in the answer to Reviewer 1, we believe not all padding strategies can deliver this position information.


Many thanks for your review and we appreciate your insightful feedback.

In our paper we discussed the implicit effect of the widely used zero-padding mechanism in CNNs. We believe the strong position information is encoded by the value transition near the boundary, zero to non-zero values. Intuitively, we believe other padding strategies, e.g. reflection or replication padding, are not able to deliver this clear position information.

We compared the effect of Circular padding implemented in Pytorch with the commonly used zero-padding on the Horizontal (H) setting using VGG16, First row of Table 1 (VGG). The training loss of zero-padding starts from 0.045 and drops to 0.03 in the end. While the loss for circular-padding begins at 0.065 and ends at 0.056, much higher than zero-padding. The results of circular-padding on the PASCAL-S dataset are (SPC 0.381, MAE 0.224). Note that this result is similar to the setting of VGG w/o padding, Table 4 (VGG w/o padding on H). This further validates our hypothesis that the position information is delivered by the value transition of zero-padding.

For the conv-padding paper, according to Equations (4) and (5), their method essentially still applies zero-padding, which means the position information should be encoded. Their method is actually weighing the output of the convolution based on how many zeros are padded, r(i, j).


We really appreciate your review and  we’re glad to hear you are pleased with the paper!

Please let us further clarify the implementation details. We did not remove any pooling layers except the last average pooling layer in the ResNet, which was designed to compress the output in order to feed to a Fully Connected (FC) layer. The pooling layers within each network (convolutional part, sometimes called backbone) have been retained because the weight was trained based on that structure design. It is commonplace to replace the FC layers with conv layers as in most dense labeling tasks.


The paper investigates to what degree Convolutional Neural Networks (CNNs) learn to encode positional information.
Rather interesting finding is the not only they do encode this information, but that it is to a large degree function of the padding commonly used in the CNN architectures.

The problem the paper is looking at is well motivated, the experiments are nicely designed and it includes comprehensive ablation study.
Previous and related work seems to be well referenced.
The main idea of introducing the PosENet to predict the gradient map is neat, and allows for interesting experiments (e.g. what layers most strongly encode the positional information).

I really enjoyed the paper, the overall quality is high and does not seem to be rushed (no obvious typos or mistakes in the figures/tables).
I believe this should be an accept.

Q:
I can understand why you removed the pooling layers, but did you try to run some of your experiments with these as well? How were the numbers effected?

This paper studies whether and how position information is encoded in CNNs. On top of VGG and ResNet, it constructs an additional PosENet to recover position information. By analyzing how well PosENet recovers position information, this paper provides several interesting findings: CNNs indeeds encode position information and zero-padding is surprisingly important here.

[Pros]

1. I enjoy reading this paper: probing CNNs is not easy, but it designs experiments in an intuitive way and rigorously performs ablation studies and analysis.
2. The observations and findings are interesting and helpful to the community.

[Cons]

1. A weakness of this paper is that it ignores the impact of training process while probing PosENet: In Table 1, VGG/ResNet perform much better than PosENet, but it could be because VGG/ResNet is easier to train (kind of fine-tuning PosENet only) than PosENet. Would be nice to show the training curve and train PosENet longer.
2. Zero-padding seems to play a surprisingly important role in encoding position information (Table 5), but it is still unclear why it is so important and how it helps.

Overall, I think this is a good paper.


Meta-review: This paper analyzes the weights associated with filters in CNNs and finds that they encode positional information (i.e. near the edges of the image).  A detailed discussion and analysis is performed, which shows where this positional information comes from.  

The reviewers were happy with your paper and found it to be quite interesting.  The reviewers felt your paper addressed an important (and surprising!) issue not previously recognized in CNNs.

Decision: accept-spotlight