How much Position Information Do Convolutional Neural Networks Encode?
Authors: Md Amirul Islam*
Sen Jia*
Neil D. B. Bruce
Venue:  (2020)
Keywords: network understanding
absolute position information

Abstract: In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs.

Published as a conference paper at ICLR 2020
How MUCH POSITION INFORMATION Do CONVOLUTIONAL NEURAL NETWORKS ENCODE?
Md Amirul Islam 1.1.2 Sen Jia Neil D. B. Bruce1.2
Ryerson University, Canada
2 Vector Institute for Artificial Intelligence, Canada
ami rul @scs. ryerson.ca.
sen. Sttarryeroon.oom ca, brace@ryerson.ca
ABSTRACT
In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues where positional information is derived from in deep CNNs.
INTRODUCTION
Convolutional Neural Networks (CNNs) have achieved state-of-the-arr results in many computer vi-
sion tasks, c.g. object classification (Simonyan & Zisserman 2014; He et al. 2016) and detection
(Redmon et al. 2016; Ren et al.) 22155), face recognition (Taigman al. [2014). semantic segmenta-
tion (Long et al. 2015 Chen et al. [2018: 8 Noh et al. 2015 1055][ssan Islam et al. 2017) and saliency detection
(Cornia al. [018]] Li et al., [014]] Jia & Bruce, 2019: Islam et al.] 2018). However, CNNs have
faced some criticism in the context of deep learning for the lack of interpretability (Lipton) 2018).
The classic CNN model is considered to be spatially-agnosti and therefore capsule (Sabour et al., 2017) or recurrent networks (Visin et al. 2015) have been utilized to model relative spatial relation- ships within learned feature layers. It unclear if CNNs capture any absolute spatial information which is important in resition-depender tection). As shown in Fig. [ the regions determined be most salient (Jia & Bruce 2018) tend to be near the center of an image. While detecting saliency on a cropped version of the images, the most salient region shifts even though the visual features have not been changed. This somewhat surprising, given the limited spatial extent of CNN filters through which the image is interpreted. In this paper, we examine the role of absolute position information by performing a series of random- izution lesty with the hypothesis that CNNs might indeed learn to encode position information as cue for decision making. Our experiments reveal that position information is implicitly learned from the commonly used padding operation pero-ppddinn). Zero-padding is widely used for keeping the same dimensionality when applying convolution. However, its hidden effect in representation learn- ing has been long omitted. This work helps to better understand the nature of the learned features in CNNs and highlights an important observation and fruitful direction for future investigation.
tasks (e.g. semantic segmentation and salient object de-
Previous works try to visualize learned feature maps demystify how CNNs work. simple idea is to compute losses and pass these backwards to the input space to generate a pattern image that can maximize the activation of given unit (Hinton et al. 2006 Erhan et al., 2009). However, it is very difficult to model such relationships when the number of layers grows. Recent work deconvolutional network (Zeiler et al. 2011) is leveraged to map learned features back to the input space and their
(Zeiler & Fergus| 2014) presents a non-paramttric method for visualization.
"Equal contribution

---

Published as a conference paper at ICLR 2020
Figure Sample predictions for salient regions for input images (left), and a slightly cropped version (right). Cropping results in shift in position rightward of features relative to the centre. It notable that this has significant impact on output and decision of regions deemed salient despite no explicit position encoding and modest change to position in the input.
results reveal what types of patterns a feature map actually learns. Another work (Selvaraju et al, 2017) proposes to combine pixel-level gradients with weighted class activation mapping to locate the region which maximizes class-specific activation. As an alternative to visualization strategies, an empirical study (Zhang et al., (016) has shown that simple network can achieve loss on noisy labels. We share the similar idea of applying a randomization test to study the CNN learned features. However, work differs from existing approaches in that these techniques only present interesting visualizations or understanding, but fail to shed any light on spatial relationships encoded by CNN model.
training
In summary, CNNs have emerged as a way of dealing with the prohibitive number of weights that would come with fully connected end-to-end network. A trade-off resulting from this is that kernels and their learned weights only have visibility of small subset of the image. This would seem imply solutions where networks rely more on cues such as texture and color rather than shape (Baker et al., 2018). Nevertheless, position information provides a powerful cue for where objects might appear an image (e.g. birds in the sky). sufficiently on such cues that they implicitly encode spatial position along with the features they represent. is our hypothesis that deep neural networks succeed part by learning both what and where things are. This paper tests this hypothesis, and provides convincing evidence that CNNs do indeed rely on and learn information about spatial positioning in the image to a much greater extent than one might expect.
conceivable that networks might rely
2 POSITION INFORMATION IN CNNS
CNNs naturally try to extract fine-level high spatial-frequency details (e.g. edges, texture, lines) in the early convolutional stages while at the deepest layers encoding the network produces the richest possible category specific features representation (Simonyan & Zisserman 2014]] He et al.
2016 (Badrinarayanan et al. (017). In this paper, we propose hypothesis that position information implicitly encoded within the extracted feature maps and plays important role in classifying, detecting or segmenting objects from a visual scene. We therefore aim to prove this hypothesis by predicting position information from different CNN archetypes in an end-to-end manner. In the following sections, we first introduce the problem definition followed by brief discussion of our proposed position encoding network.
Problem Formulation: Given an input image Im 1x our goal is to predict gradient- like position information mask f,e Ax where each pixel value defines the absolute coordinates of an pixel from left left-tright or top-bbotoom We generate gradient-like like masks Gpos h (Sec. 2.2 for supervision our experiments, with weights of the base CNN archetypes being fixed.
2.1 POSITION ENCODING NETWORK
Our Position Encoding Network (PosENet) (See Fig. ( consists of two key components: a feed- forward convolutional encoder network fene and a simple position encoding module, denoted as fpem- The encoder network extracts features different levels of abstraction, from shallower to deeper layers. The position encoding module takes multi-scale features from the encoder network input and predicts the absolute position information the end.
Encoder: We use ResNet and VGG based architectures to build encoder networks (fone) by re- moving the average pooling layer and the layer that assigns categories. As shown in Fig. the
2

---

Published as a conference paper at ICLR 2020
Lom 099
6pes
Natural
Black
White
Noisc
12
Postion Erooding Module
VS
Figure 3:
Sample images and generated
Figure 2: Illustration of PosENet architecture.
gradient-like ground-trun position maps.
encoder module consists of five feature extractor blocks denoted by fg. tracted multi-scale features from bottom to top layers of the canonical network are denoted by (Spost fposr f foor I fooor Spoe). We summarize the key operations follows:
The
Spos (W. Imm
(I)
where Wa denotes weights that frozen. denotes the convolution operation. Note that in probing the encoding network, only the position encoding module fpem is trained to focus on extracting position information while the encoder network is forced to maintain their existing weights.
Position Encoding Module: The position encoding module takes multi-scale features (post ... _ from fene as input and generates the desired position map fp thorough a trans- formation function Tpos- The transformation function Tpos first applies bi-linear interpolation operation on the feature maps to have the same spatial dimension resulting in feature map fpos. Once we have the same spatial dimension for multi-scale features, we concatenate them together followed by sequence of l k convolution operations. In our experiments, we vary the value of between {1, 5, 7) and most experiments are carried out with single convolutional layer in the position encoding module fpem. The key operations can be summarized follows:
pos pas ... +
spos) fp (We pas
f
(2)
where W
pos is the trainable weights attached with the transformation function Tpos-
The main objective the encoding module validate whether position information is implicitly learned when trained on categorical labels. Additionally, the position encoding module models the relationship between hidden position information and the gradient like ground-truth mask. The output is expected to be random if there is no position information encoded in the features maps and vice versa (ignoring any guidance from image content).
2.2 SYNTHETIC DATA AND TRUTH GENERATION
To validate the existence of position information in network, implement randomization test by assigning a normalized gradient-like position map as ground-truth shown in Fig. We first generate gradient-like masks in Horizontal (H) and vertical (V) directions. Similarly, apply Gaussian filter to design another type of ground-truth map, Gaussian distribution (G) The key motivation of generating these three patterns is validate if the model can learn absolute position on one or two axes. Additionally, We also create two types of repeated patterns, horizontal and vertical stripes, (HS, VS). Regardless of the direction, the position information in the multi-level features is likely to be modelled through transformation by the encoding module fpem. Our design of gradient ground-trui truth can be considered as type of random label because there no correlation between the input image and the ground-truth with respect to position. Since the extraction of position information is independent the content of images, we can choose any image datasets. Meanwhile, we also build synthetic images to validate our hypothesis.
2.3 TRAINING THE NETWORK
As we implicitly aim to encode the position information from a pretrained network, we freeze the encoder network fene in all of our experiments. Our position encoding module fpem generates the
We the gradient denote pixel intensities instead of the gradient in hack propagation.

---

Published as a conference paper at ICLR 2020
position map fo of interest. During training, for given input image Im RXxxxx and associated ground-truth position map goss we apply the supervisory signal on f by upsampling the size of Gios. Then, we define a pixel-wise mean squared enror loss to measure the difference between predicted and ground-truth position maps as follows:
1
J 6,
(1 - )²)
(3)
2n
where x E IR" and E IR" (n denotes the spatial resolution) are the vectorized predicted position and ground-truth map respectively. and y; refer to pixel of fp and Ojss respectively.
3 EXPERIMENTS
3.1 DATASET AND EVALUATION METRICS
Datasets: We use the DUT-S dataset (Wang et 2017) our training set, which contains 10, 533 images for training. Following the common training protocol used in (Zhang et al. 2017 Liu et al. 2018). we train the model on the training set of DUT-S and evaluate the existence of position formation on the natural images of the PASCAL-S (Li et al., 2014) dataset. The synthetic images (white, black and Gaussian noise) are also used as described the common setting used in saliency detection just to make sure that there is no overlap between the training and test sets. However, any images can be used in our experiments given that the position information is relatively content independent.
Section 2.2 Note that we follow
As position encoding measurement is new direction, there is no universal metric. We use two different natural choices for metrics (Spearmen Correlation (SPC) and Mean Absoute Error (MAE)) to measure the position encoding performance. The SPC is defined as the Spearman's 's correlation between the ground-truth and the predicted position map. For ease of in- terpretation, we keep the SPC score within range [-1 1]. MAE is the average pixel-wise difference between the predicted position map and the ground-truth gradient position map.
Evaluation Metrics:
3.2 IMPLEMENTATION DETAILS
We initialize the architecture with network pretrained for the ImageNet classification task. The new layers in the position encoding branch are initialized with xavier initialization (Glorot & Bengio 2010). We train the networks using stochastic gradient descent for 155 epochs with momentum of 0.9, and weight decay of 1e-4. We resize each image to fixed size of 224 224 during training and inference. Since the spatial extent of multi-level features are different, we align all the feature maps to a size of 28 x 28, We report experimental results for the following baselines that are described as follows: VGG indicates PosENet is based the features extracted from the VGGI6 model. Similarly, ResNet represents the combination of ResNet-152 and PosENet. PosENet alone denotes only the PosENet model is applied to learn position information directly from the input image. H, V. G. HS and VS represent the five different ground-truth patterns, horizontal and vertical gradients, 2D Gaussian distribution, horizontal and vertical stripes respectively.
3.3 EXISTENCE OF POSITION INFORMATION
We first conduct experiments to validate the exis- tence of position information encoded in pretrained model. Following the same protocol, we train the VGG and ResNet based networks each type of the ground-truth and report the experimental results in T We also report results when we only train PosENet without using any pretrained model to justify that the position information is not driven from prior knowledge of objects. Our experiments do not focus on achieving higher performance on the metrics but instead validate how much position information CNN model encodes how easily PosENet can extract this informa- tion. Note that, we only use one convolutional layer with a kernel size of 3 x without any padding in the PosENet for this experiment.
Position Information in Pretrained Models:
As shown in Table I. PosENet (VGG and ResNet) can easily extract position information from the pretrained CNN models, especially the ResNet based PosENet model. However, training PosENet
4

---

Published as a conference paper at ICLR 2020
PASCAL-S
Black
White
Noise
Model
SPC
MAE
SPC
MAE
SPC
MAE
SPC
MAE
PosENet
.122
.251
.
.251
.0
.251
.001
.251
H
VGG
.742
.149
.751
.164
.873
.157
.51
.173
ResNet
.933
.084
.987
.800
.994
.078
.973
PosENet
.131
.288
.0
.251
.0
.251
.053
.250
VGG
.816
.129
.846
.146
.927
.138
.771
.150
ResNet
.951
.083
.978
069
.979
.072
.968
.074
PosENet
-0011
.233
.166
.0
.186
.034
.214
VGG
.814
.109
.842
.123
.898
.116
.762
.129
ResNet
.966
.070
.953
.068
.964
.044
.971
.055
PosENet
.712
-0055
.704
.704
.023
HS
VGG
.055
.556
.532
.583
.576
.574
.375
.573
ResNet
.344
.528
.566
.188
.562
.515
.471
.530
PosENet
.006
.723
.081
.709
.081
.709
.018
.714
VS
VGG
.374
.567
.538
.575
.437
.578
.526
.566
ResNet
.520
.537
.523
.593
.514
.523
.545
Table I: Quantitative comparison of different networks in terms of SPC and MAE across different image types.
Input
GT
PosENet
VGG
ResNet
Figure 4: Qualitative results of PosENet based networks corresponding to different ground-truth patterns.
(PosENet) separately achieves much lower scores across different patterns and source images. This result implies that it is very difficult to extract position information from the input image alone. PosENet can extract position information consistent with the ground-truth position map only when coupled with deep encoder network. As mentioned prior, the generated ground-truth map be considered as a type of randomization test given that the correlation with input has been ignored (Zhang et al.) 2016). Nevertheless, the high performance the test truth patterns reveals that the model is not blindly overfitting to the noise and instead is extracting true position information. However, we observe low performance on the repeated patterns (HS and VS) compared to other patterns due to the model complexity and specifically the lack of correlation between ground-truth and absolute position (last two rows of Table T. The pattern can be seen as one quarter of sine wave whereas the striped patterns (HS and VS) can be considered as repeated periods of a sine wave which requires deeper comprehension.
different ground-
The qualitative results for several architectures across different patterns are shown in Fig. We can see the correlation between the predicted and the ground-truth position maps corresponding to H, G and HS patterns, which further reveals the existence of position information in these networks. The quantitative and qualitative results strongly validate our hypothesis that position information is implicitly encoded in every architecture without any explicit supervision towards this objective.
Moreover, PosENet alone shows no capacity to output a gradient map based on the synthetic data. We further explore the effect of image semantics in Sec. 4.1 It interesting note the performance gap among different architectures specifically the ResNet based models achieve higher performance than the VGG16 based models. The reason behind this could be the of different convolutional

---

Published as a conference paper at ICLR 2020
PosENet
VGG
PosENet
VGG
Layers
Kernel
SPC
MAE
SPC
MAE
SPC
MAE
SPC
MAE
Layer
.012
.251
.742
.149
1
.013
.251
.542
.196
H
2 Layers
.056
.250
.797
.128
H
x 3
.012
.251
.742
.149
3 Layers
.055
.250
.830
.117
x 7
.060
.250
.288
.120
Layer
-001
.109
.017
.188
.724
.127
G
2 Layers
.067
.187
.828
.105
G
x 3
-0001
.233
.814
.109
3 Layers
.126
.186
.835
.104
x 7
.068
.187
.816
.111
Layer
-0001
.556
-004
.628
.317
.576
HS
2 Layers
-006
.628
.483
.538
HS
x 3
-000
.723
.405
.556
3 Layers
.003
.628
.491
.540
7 7
.002
.628
.487
.532
(b)
Table 2: Quantitative comparison on the PASCAL-S dataset in terms of SPC and MAE with varying (a) number of layers and (b) kernel sizes. Note that (a) the kernel size is fixed x 3 but different numbers of layers are used in the PosENet. (b) Number of layers is fixed to one but we use different kernel sizes the PosENet.
kernels in the architecture or the degree of prior knowledge of the semantic content. We show an ablation study in the next experiment for further investigation. For the rest of this paper, we only focus on the natural images, PASCAL-S dataset, and three representative patterns, H, G and HS.
3.4 ANALYZING POSENET
In this section, we conduct ablation studies to examine the role of the proposed position encoding network by highlighting two key design choices. (1) the role of varying kernel size in the position encoding module and (2) stack length of convolutional layers we add to extract position information from the multi-level features.
Impact of Stacked Layers: Experimental results in Table show the existence of position informa- tion learned from an object classification task. In this experiment, we change the design of PosENet to examine if it is possible to extract hidden position information more accurately. The PosENet used the prior experiment (Table [ has only one convolutional layer with kernel size of 3 x 3. Here, apply a stack of convolutional layers of varying length to the PosENet and report the ex- perimental results Table (a). Even though the stack size varied, we aim to retain a relatively simple PosENet only allow efficient readout of positional information. As shown in Table 2 keep the kernel size fixed at 3 x 3 while stacking multiple layers. Applying more layers in the PosENet can improve the readout of position information for all the networks. One reason could be that stacking multiple convolutional filters allows the network to have a larger effective receptive field, for example two 3 x 3 convolution layers are spatially equal one x 5 convolution layer (Simonyan & Zisserman, 2014). An alternative possibility is that positional information may be represented a manner that requires more than first order inference (e.g. linear readout).
Impact of varying Kernel Sizes: We further validate PosENet by using only one convolutional layer with different kernel sizes and report the experimental results in Table From Table (b)), (b), we can see that the larger kernel sizes are likely to capture more position information compared to smaller sizes. This finding implies that the position information may distributed spatially within layers and in feature space as larger receptive field can better resolve position information.
We further show the visual impact of varying number of layers and kernel sizes to learn position information in Fig.
3.5 WHIBRE THE POSITION INFORMATION STORED?
Our previous experiments reveal that the position information is encoded in a pretrained CNN model. is also interesting to see whether position information is equally distributed across the layers. In this experiment,
train PosENet each of the extracted features. flas- f foos- flas f separately using VGG16 to examine which layer encodes more position information. Similar to only apply one kernel in F Ppom to obtain the position map.
Sec. 3.3

---

Published as a conference paper at ICLR 2020
Figure 5: The effect of more Layers (Top row) and varying Kernel Size (bottom row) applied in the PoseNet. Order (left - right): GT (G), PosENet (L=1, KS=1), PosENet (L=2, KS=3), PosENet (L=3, KS=7), VGG (L=1, KS=1), VGG (L=2, KS=3), VGG (L=3, KS=7).
Method
fpsx
SPC
MAE
pdx
pos
pox
.101
.249
.344
.255
H
VGG
.472
.003
.610
.181
.657
.177
V
.742
.19
.182
.404
.168
G
VGG
.588
.166
.653
.138
.693
.135
.814
.109
Table 3: Performance of VGG on natural images with a varying extent of the reach of different feed-forward blocks.
As shown in Table the VGG based PosENet with top f features achieves higher performance compared the bottom flos features. This may partially result of more feature maps being ex- tracted from deeper as opposed to shallower layers, 512 vs 64 respectively. However, it is likely indicative stronger encoding of the positional information in the deepest layers of the network where this information is shared by high-level semantics. We further investigate this effect for VGG16 where the top two layers (ft... and f have the same number of features. More inter- estingly, foos achieves better results than f This comparison suggests that the deeper feature contains more position information, which validates the common belief that top level visual features are associated with global features.
4 WHERE DOES POSITION INFORMATION COME FROM?
We believe that the padding near the border delivers position information to learn. Zero-padding is widely used in convolutional layers maintain the same spatial dimensions for the input and output, with number of zeros added at the beginning and at the end of both axes, horizontal and vertical. To validate this, remove all the padding mechanisms implemented within VGG16 but still initialize the model with the ImageNet pretrained weights. Note that we perform this experiment only using VGG based PosENet since removing padding on ResNet models will lead to inconsistent sizes of skip connections. We first test the effect of zero-padding used in VGG, no padding used in PosENet. As we can see from Table the VGG16 model without zero-padding achieves much lower perfor- mance than the default setting (padding=1) on the natural images. Similarly, we introduce position information to the PosENet by applying zero-padding. PosENet with padding=1 (concatenating one zero around the frame) achieves higher performance than the original (padding=O). When we set padding=2, the role of position information is more obvious. This also validates our experiment in Section 3.3 that shows PosENet is unable to extract noticeable position information because padding was applied, and the information is encoded from a pretrained CNN model. This why we did not apply vero-padding in PosENet in our previous experiments. Moreover, we aim explore
7

---

Published as a conference paper at ICLR 2020
G
HS
Model
SPC
MAE
SPC
MAE
SPC
MAE
PosENet
.012
.251
-.001
.233
-0001
.712
PosENet with padding=1
.274
.239
.205
.184
.148
.608
PosENet with padding=2
.397
.223
.380
.177
.214
.595
VGG16
.742
.149
.814
.109
.405
.556
VGGI6 w/o. padding
.381
.223
.359
.174
.011
.628
Table 4: Quantitative comparison subject to padding in the convolution layers used in PosENet and VGG (w/o and with zero padding) on natural images.
Figure 6: The effect of zero-padding on Gaussian pattern. Left to right: GT (G), Pad=O (.286, .186),
Pad=1 (.227, .10),, Pad=2 (.473, 169), VGG Pad=1 (.928, .885), VGG Pad=0(.405, 170).
how much position information is encoded in the pretrained model instead of directly combining with the PosENet. Fig. F ifllustraate the impact of prooppadding encoding position information subject to padding using Gaussian pattern.
4.1 CASE STUDY
Recall that the position information is considered be content independent but our results in Ta- [ show that semantics within an image may affect the position map. To visualize the impact of semantics, we compute the content loss heat map using the following equation:
1g pos
fP)) + 1(9 - fr') + 1(9%. f:)1
where f% fr. and fg are the predicted position maps from horizontal, vertical and Gaussian patterns respectively.
As shown in Figure the heatmaps of PosENet have larger content loss around the corners. While the loss maps of VGG and ResNet correlate more with the semantic content. Especially for ResNet, the deeper understanding of semantic content leads to stronger interference in generating smooth gradient. The highest losses are from the face, person, cat, airplane and vase respectively (from left to right). This visualization can be alternative method to show which regions model focuses especially in the case of ResNet.
Figure 7: Error heat maps of PosENet (1st row), VGG (2nd row) and ResNet (3rd row).
8

---

Published as a conference paper at ICLR 2020
4.2
DRIVEN POSITION INFORMATION
Saliency Detection: We further validate our findings in the position-dependent tasks (semantic segmentation and salient object defection (SOD)). First, we train the VGG network with and with- out zero-padding from scratch to validate if the position information delivered by zero-padding is critical for detecting salient regions. For these experiments, we use the publicly available MSRA dataset (Cheng et al. 2015) as our SOD training set and evaluate on three other datasets (ECSSD, PASCAL-S, and -MRON.,, From Table (a), we can see that VGG without padding achieves much worse results on both of the metrics (F-measur) and MAE) which further validates our findings that zero-padding is the key source of position information.
Semantic Segmentation: We also validate the impact of zero- pero-padding on the semantic segmentation task. We train the VGG16 network with and without zero padding on the training set of PASCAL VOC 2012 dataset and evaluate the validation set. Similar to SOD, the model with zero padding significantly outperforms the model with no padding.
ECSSD
PASCAL-S
DUT-OMRON
Model
Model
mIoU (%)
MAE
Fm
MAE
MAB
VGG w/o padding
12.3
VGG w/ padding
.36
.48
.32
.48
.25
VGG
23.1
VGG
.78
.17
.66
.21
.63
.18
(b)
(a)
Table 5: VGG models with and w/o zero-padding for (a) SOD and (b) semantic segmentation.
We believe that CNN models pretrained on these two tasks can learn more position information than classification task. To validate this hypothesis, we take the VGG model pretrained on ImageNet our baseline. Meanwhile, we train two VGG models on the tasks of semantic segmentation and saliency detection from scratch, denoted as VGG-SS and VGG-SOD respectively. Then we finetune these three VGG models following the protocol used in Section 3.3 From Table 66 we can see that the VGG-SS and VGG-SOD models outperform VGG by large margin. These experiments further reveal that the zero pard-pgdiing strategy plays an important role in a observation that has been inngiig in neural network solutions to vision problems.
task, an
PASCAL
BLACK
WHITE
NOISE
Model
SPC
MAE
SPC
MAE
SPC
MAE
SPC
MAE
VGG
.742
.149
.751
.164
.873
.157
.591
.173
VGG-SOD
.969
.055
.857
.099
.938
.087
.965
.060
VGG-SS
.982
.038
.990
.030
.985
.032
.955
.033
VGG
.814
.109
.842
.123
.898
.116
.762
.129
VGG-SOD
.948
.067
.904
.086
.907
.085
.912
.077
VGG-SS
.971
.055
.984
.050
.989
.046
.982
.051
VGG
.405
.556
.532
.583
.576
.574
.375
.573
HS
VGG-SOD
.667
.476
.699
.506
.709
.482
.668
.489
VGG-SS
.810
.430
.802
.426
.810
.426
.789
.428
Table 6: Comparison of VGG models pretrained for classification, SOD, and semantic semanticssemmentatiiii segmentation.
5 CONCLUSION
In this paper we explore the hypothesis that absolute position information implicitly encoded in convolutional neural networks. Experiments reveal that positional information is available to a strong degree. More detailed experiments show that larger receptive fields or non-linear readout of positional information further augments the readout of absolute position, which is already very strong from a trivial single layer 3 x PosENet. Experiments also reveal that this recovery is pos- sible when no semantic cues are present and interference from semantic information suggests joint encoding of what (semantic features) and where (absolute position). Results point to zero padding anchor from which spatial information is derived and eventually propagated over
and borders the whole image as spatial abstraction occurs. These results demonstrate a fundamental property of CNNs that unknown to date, and for which much further exploration is warranted.

---

Published as a conference paper at ICLR 2020
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for scene segmentation. TPAMI, 2017.
Nicholas Baker, Hongjing Lu, Gennady Frlikhman, and Philip J. Kellman. Deep convolutional networks do not classify based global object shape. PLOOS Computational Biology, 2018.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crís. TPAMI, 2018.
M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S. Hu. Global contrast based salient region detection. TPAMI, 2015.
M. Cornia, L.. Baraldi, G. Serra, and R. Cucchiara. Predicting human eye fixations via an lstm-based saliency attentive model. TIP, 2018.
Dumitru Erhan, Yoshua Bengio. Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 2009.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Simon Osindero, Max Welling, and Yee-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 2006.
Md Amirul Islam, Mrigank Rochan, Neil DB Bruce, and Yang Wang. Gated feedback refinement network for dense image labeling. In CVPR, 2017.
Md Amirul Islam, Mahmoud Kalash, and Neil DB Bruce. Revisiting salient object detection: Si- multaneous detection, ranking, and subitizing multiple salient objects. In CVPR, 2018.
Sen Jia and Neil DB Bruce. Eml-net: An expandable multi-layer network for saliency prediction. arXiv preprint arXiv: 1805.0104 2018.
Sen Jia and Neil DB Bruce. Richer and deeper supervision network for salient object detection. arXiv preprint arXiv: 1901.0090 2019.
Y. Li, X. Hou, C. Koch, M. Rehg, and A. L. Yuille. The secrets of salient object segmentation. In CVPR, 2014.
Zachary C Lipton. The mythos of model interpretability. Quene, 2018.
Nian Liu, Junwei Han, and for saliency detection. In CVPR, 2018.
Yang. Picanet: Learning pixel-wise contextual attention
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. In /CCV, 2015.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In NIPS, 2017.
10

---

Published as a conference paper at ICLR 2020
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- local- ization. In 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014.
Visin, K Kastner, K Cho, M Matteucci, Courville, and Bengio. A recurrent neural network based alternative convolutional networks. arXiv preprint arXiv: 1505.0033, 2015.
Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In CVPR, 2017.
M. D. Zeiler, G. W. Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV, 2011.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 16X1v::611.03581 2016.
Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan. Amulet: Aggregating multi-level convolutional features for salient object detection. In ICCV, 2017.


Reviews:
[{"review_id": "r1xDPX_yqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies whether and how position information is encoded in CNNs. On top of VGG and ResNet, it constructs an additional PosENet to recover position information. By analyzing how well PosENet recovers position information, this paper provides several interesting findings: CNNs indeeds encode position information and zero-padding is surprisingly important here.\n\n[Pros]\n\n1. I enjoy reading this paper: probing CNNs is not easy, but it designs experiments in an intuitive way and rigorously performs ablation studies and analysis.\n2. The observations and findings are interesting and helpful to the community.\n\n[Cons]\n\n1. A weakness of this paper is that it ignores the impact of training process while probing PosENet: In Table 1, VGG/ResNet perform much better than PosENet, but it could be because VGG/ResNet is easier to train (kind of fine-tuning PosENet only) than PosENet. Would be nice to show the training curve and train PosENet longer.\n2. Zero-padding seems to play a surprisingly important role in encoding position information (Table 5), but it is still unclear why it is so important and how it helps.\n\nOverall, I think this is a good paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklQXVNkcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied the problem of the encoded position information in convolution neural networks. The hypothesis is that CNN can implicitly learn to encode the position information. The author tests the hypothesis with lots of experiments to show how and where the position information is encoded.\n\nClarity:\nThis paper is interesting for me. It tries to understand the encoded position information that is easily ignored by researchers. I like adequate experiments with learned position information and position illustrations.\n\nExperiments:\n1. The paper mainly discussed the zero-padding and found it is the source of position information. How about other padding modes like constant-padding, reflection-padding, and replication-padding?\n\n2. The partial convolution-based padding method [1] (padded regions are masked out) shows that its recognition accuracy is higher than the traditional zero-padding approach. Can you help investigate where the position information comes from for this case?\n\n[1] Partial Convolution based Padding, https://arxiv.org/pdf/1811.11718.pdf.\n\n\nSome of my concerns are well addressed by the author thus I upgrade my score.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklLFSzRtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates to what degree Convolutional Neural Networks (CNNs) learn to encode positional information.\nRather interesting finding is the not only they do encode this information, but that it is to a large degree function of the padding commonly used in the CNN architectures.\n\nThe problem the paper is looking at is well motivated, the experiments are nicely designed and it includes comprehensive ablation study.\nPrevious and related work seems to be well referenced.\nThe main idea of introducing the PosENet to predict the gradient map is neat, and allows for interesting experiments (e.g. what layers most strongly encode the positional information).\n\nI really enjoyed the paper, the overall quality is high and does not seem to be rushed (no obvious typos or mistakes in the figures/tables).\nI believe this should be an accept.\n\nQ:\nI can understand why you removed the pooling layers, but did you try to run some of your experiments with these as well? How were the numbers effected?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the weights associated with filters in CNNs and finds that they encode positional information (i.e. near the edges of the image).  A detailed discussion and analysis is performed, which shows where this positional information comes from.  

The reviewers were happy with your paper and found it to be quite interesting.  The reviewers felt your paper addressed an important (and surprising!) issue not previously recognized in CNNs.
Decision: Accept (Spotlight)