Unveiling The Mask of Position-Information Pattern Through the Mist of Image Features
Authors: Chieh Hubert Lin
Hung-Yu Tseng
Hsin-Ying Lee
Maneesh Kumar Singh
Ming-Hsuan Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: positional information
position encoding
padding
CNN

Abstract: Recent studies have shown that paddings in convolutional neural networks encode absolute position information which can negatively affect the model performance for certain tasks. However, existing metrics for quantifying the strength of positional information remain unreliable and frequently lead to erroneous results. To address this issue, we propose novel metrics for measuring and visualizing the encoded positional information. We formally define the encoded information as Position-information Pattern from Padding (PPP) and conduct a series of experiments to study its properties as well as its formation. The proposed metrics measure the presence of positional information more reliably than the existing metrics based on PosENet and tests in F-Conv. We also demonstrate that for any extant (and proposed) padding schemes, PPP is primarily a learning artifact and is less dependent on the characteristics of the underlying padding schemes.

Under review as a conference paper at ICLR 2023
UNVEILING THE MASK OF PATTERN THROUGH THE MIST OF IMAGE FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies have shown that paddings in convolutional neural networks encode absolute position information which can negatively affect the model performance for certain tasks. However, existing metries for quantifying the strength of po- sitional information remain unreliable and frequently lead to erroneous results. To address this issue, we propose novel metries for measuring and visualizing the encoded positional information. We formally define the encoded information as Position-in intiiiotiiooo Pattern from Padding (PPP) and conduct a series of ex- periments to study its properties well as its formation. The proposed metrics measure the presence of positional information more reliably than the existing metrics based on PosENet and tests in F-Conv. We also demonstrate that for any extant (and proposed) padding schemes, PPP is primarily a learning artifact and is less dependent on the characteristics of the underlying padding schemes.
INTRODUCTION
Padding, of the most fundamental components in neural network architectures, has received much less attention than other modules in the literature. In convolutional neural networks (CNNs), zero padding is frequently used perhaps due its simplicity and low computational costs. This design preference remains almost unchanged in the past decade. Recent studies (Islam et al., 2020; Islam al., 2021b; Kayhan Gemert, 2020; Innamorati et al., 2020) show that padding can implicitly provide a network model with positional information. Such positional information can cause unwanted side-effects by interfering and affecting other sources of position-sensitive cues (e.g., explicit coordinate inputs (Lin et al., 2022; Alsallakh et al., 2021a; Xu et al., 2021; Ntavelis et al., 2022: Choi et al., 2021), embeddings (Ge et al., 2022), or boundary conditions of the model (Innamorati et al., 2020; Alguacil et al., 2021; Islam et al., 2021a)). Furthermore, padding may lead to several unintended behaviors (Lin et al., 2022: Xu et al., 2021; Ntavelis et al., 2022; Choi al., 2021), degrade model performance (Ge et al., 2022; Alguacil al., 2021; Islam al., 2021a), or sometimes create blind spots (Alsallakh et al., 2021a). Meanwhile, simply ignoring the padding pixels (known no-padding or valid-padding) leads to the foveal effect (Alsallakh et al., 2021b; Luo et al., 2016) that causes a model to become less attentive to the features on the image border. These observations motivate us to thoroughly analyze the phenomenon of positional encoding including the effect of commonly used padding schemes.
Conducting such a study requires reliable metries detect the presence of positional information introduced by padding, and more importantly, quantify its strength consistently. We observe that the existing methods for detecting and quantifying the strength of positional information yield inconsistent results. In Section 3, we revisit two closely related evaluation methods, PosENet (Islam* et al., 2020) and F-Conv (Kayhan & Gemert, 2020). Our extensive experiments demonstrate that (a) metrics based on PosENet are unreliable with an unacceptably high variance, and (b) the Border Handling Variants (BHV) test in F-Cony suffers from unaware confounding variables in its design, leading to unreliable test results.
In addition, we observe all commonly-used padding schemes actually encode consistent patterns underneath the highly dynamic model features. However, such pattern is rather obscure, noisy, and visually imperceptible for most paddings (except zeros-paddes peros-padding), which makes recognizing and analyzing it difficult. Fortunately, we show that such patterns can be consistently revealed with sufficient number of samples by defining an optimal padding scheme (see Section 2.1 and Figure 1).
The source codes and data collection scripts will be made publicly available.

---

Under review as a conference paper at ICLR 2023
Expectation of Channel-Averaged Image Features
PPP
1 Sample
10 Samples
100 Samples
480 Samples
480 Samples
Imperceptible
Recognizable
Figure 1: iiitiio-informattion Pattern from Padding (PPP). We propose method that can consistently and effectively extract PPPs through the distributional difference between optimally- padded (gray-scale surfaces) and algorithmically-pdae show that the two distributions become distinguishable as the number sample increases. Following the procedure in Section 2.2, extract clear view of PPP with the expectation of the pair- wise differences between aptimally-padded and engoritmiically-pddeeee visualization in tilted view (first row) and top view (second row). The colors represent the magnitude (blue/cold/weak to green/warm/stnnn at each pixel. The features are extracted at the 3rd layer of interest (Appendix A) from randn-padded (Section 2.4) ResNet50 pretrained on ImageNet.
features (colored surfaces). The results
features. We render each
We accordingly propose a new evaluation paradigm and develop a method consistently detect the presence of the inattiooniiformationn Pattern from Padding (PPP), which is persistent pattern embedded the model features to retain positional information. We present two metrics to measure the response of PPP from the signal-to-noise perspective and demonstrate its robustness and low deviation among different settings, each with multiple trials of training.
To weaken the effect of PPP, in Section 2.4, design a padding scheme with built-in stochasticity, making it difficult for the model to consistently construct such biases. However, our experiments show that the models can still circumvent the stochasticity and end up consistently constructing PPPs. These results suggest that a model likely constructs PPPs purposely to facilitate its training, rather than falsely accidentally learning some filters that respond to padding features.
With reliable PPP metrics, we conduct a series of experiments to analyze the characteristics of PPP in Section 4.1. Specifically, analyze the formation of PPP throughout each model training process in Section 4.3. The results show PPPs are formed expeditiously at the early stage of model training, slowly but steadily strengthen through time, and eventually shaped in clear and complete patterns. These results show that a model intentionally develops and reinforces PPPs to facilitate its learning process. Moreover, we observe the PPPs of all pretrained networks are significantly stronger than those in their initial states. This indicates an unbiased training procedure is of great importance in resolving the critical failures caused by PPP in numerous vision tasks (Alsallakh et al., 2021 a; Xu et al., 2021; Ge et al., 2022; Alguacil et al., 2021).
OBSERVATIONS AND METHODOLOGY
In this section, we first define symbols for expressing the functionality of paddings and define the optimal-padding scheme. We then give formal definition of Position-information Pattern from Padding (PPP) and utilize the qquimal-padding scheme to develop propose method to capture PPP and measure its response with two metrics.
2.1 OPTIMAL PADDING
The process of capturing image from the real world can be simplified into two steps: (a) 3D information of the environment is first projected onto an infinitely large 2D plane, and then (b) the camera determines resolution well field-of-view to form digital image from such infinitely large
2

---

Under review as a conference paper at ICLR 2023
Optrokhhatodd
Abarttasii
Inpui
Optimal puddded Margie
301
23
Cerrection
Repllrate paddeed feature
PPP wt Margin Cerrection
Principal Point Shifting
(c) Visualization Principal Shifting
by Strided
Margin Correction
(d) Impart PPP
Figure Principal point shift. (a) The stride-2 Conv2d only pads on one side, causing the principal point shift (red squares) in earlier layers. (b) Such shift requires careful margin correction while aligning algorithmically-paddee and optimally-paddes features (we describe the details point shift in Appendix A). (c) The shift is visible in the feature space (marked with red and yellow boxes). (d) is crucial to correct the principal point shift while measuring PPP. The PPP calculation involves pixel-wise distance functions, which are not robust to spatial shifts (Zhang et al., 2018).
and continuous 2D signals (Liu al., 2019; Ravi al., 2020). Let S { be a collection of such infinitely large and continuous 2D signals, and the collection of 2D images captured by cameras
spatial size (hn, wn.) be 5' {s n=1- padding scheme can be used to generate set of adgorithmically-pen padded images S = {$n IN by a padding function p:
S J = [
<f0<i<> hn and 0 < Wn
8n [
(1)
(((5 i,
otherwise,
where and are indexes of a pixel in the spatial dimension. We define a theoretical optimally-paddee collection 5 (ss 1 with aptimal-padding function p² by:
if 10<i< hn and < Wn (x,,,) = = otherwise.
S (i.
[injj]
Mb.
In practice, without curated data, the optimal-padding scheme described in Eq. is difficult achieve. We describe how we relax this constraint in Section 2.3
2.2 POSITIONAL-INFORMAATII
PATTERN FROM PADDING
Despite the previous literature discovered the existence of positional information caused by the model paddings, there is still no clear definition for such information, and lacks effective metrics detect quantify it. Ideally, two properties. First, is a spatial pattern, it contributes distinctive information to different spatial locations. Its shape enables the network to develop and exploit the absolute positional information of each pixel, eventually leading to the unattended and undesirable effects in certain tasks (Lin et al., 2022; Alsallakh et al., 2021a: Xu al., 2021; Ntavelis al., 2022; Choi al., 2021; Ge al., 2022; Alguacil et al., 2021). Second, as it represents the positional information purely contributed by the padding, it is constant pattern irrelevant to the image contents. We accordingly name the Positional-informatiioo Pattern from Padding (PPP).
effective metric for such positional information should have
Unfortunately, such pattern shares space with image features, where the image features typically have very diverse appearances and high dimensionality. When these two signals interfere with each other, the appearance PPP becomes extremely obscure and imperceptible in most cases (except zeros padding). Figure shows if we visualize features sample-by-sample there are obvious differences between optimally-padee features (colored surface). To address the issue, show that, by assuming the interferences between PPP and image features to random, its expectation over a large set of images will saturate to a constant bias and no longer hinder us from capturing PPP.
features (gray-scale surface) and algorithmically paddedd

---

Under review as a conference paper at ICLR 2023
Based on these observations and assumptions, we define PPP as the constant component independent of model imputs, and its presence is completely contributed by the existence of a padding scheme p. Given $ and a model F(&; p). which # is the model parameters and p is a padding scheme applied to F. Let the model feature extracted at k-th layer be fn, (a 0, p), where Fy. is the model from the first layer to the k-th layer. The PPP at k-th layer (PPP)) can be formulated by:
PPP E [ d ( (F 0, Pa(Sn; 0, p) ],
(3)
where d(.,,) can be any distance function. We use & distance in this work, and accordingly name the metric PPP-MAE.
important to note that, some CNN components can cause serious feature misalignment while computing PPP and leads to erroneous results. A typical example principal point shift, where the uneven padding in stride-2 convolution causes the center of features slightly drifted, as shown in Figure 2. Since the measurement of PPP requires perfect alignment, such drift should be carefully considered while integrating PPP into new architectures. We discuss the issue along with other pitfalls in Appendix A and provide three detailed examples of correcting the principal point shifting.
Pitfalls: feature misalignment.
2.3 SIMULATED OPTIMAL PADDING
In practice, it is impossible to gain access to S for calculating the optimal padding St described in Eq. 2. But fortunately, given our goal in Eq. is to analyze the model features within the (h w.) region, S* is an overshoot of the data we actually required. Given a vision model F(;; 0, p) trained at a field-of-view (hm; w,) pixels, the receptive field of such vision model is (hm Wm) ) pixels (we show the computation in Appendix A), where him >> h, and Wm >> Wn- Let an alternative image collection 5 { I==1 at (/mm Wm. pixels, the definition of receptive field implies Fi (sT; 0. equals to (s 0, for all k.
In other words, terms of computing Eq. 3, S is equivalent to S within the finite (h,, wn) region for given model architecture. Therefore, we can simulate the procedure described in Eq. I and Eq. 2 using 5 instead of St, as long as Vs E 8 the spatial size of s is strictly larger than (fmm Wm)-
2.4 RANDN PADDING
Most of the existing padding schemes (e.g., zeros, reflect, replicate, circular) exhibit certain consistent patterns that can be easily detected by some designed convolutional kernels. One may argue that the nature of easy detectability can be root cause of encouraging the models to learn to rely on these obvious patterns. This motivates us to design an additional sampling-bant padding scheme without any consistent patterns, namely randn (i.c., random normal) padding, which produces dynamical values from a normal distribution while following the local statistics. We first determine the maximal the
and minimal values of a sliding window (which can be easily achieved with max-pooling), average of them as a proxy mean (Ap, and use the difference between the mean and the maximal value as a proxy standard deviation op. For each padding location, we sample the padding value according normal distribution N (Mpp of) from the nearest sliding window. We include more implementation details in Appendix A.
Aside from creating pattern-less padding scheme with sampling, the design of randn padding is based on several factors. The sampled padding pixels are allowed to occasionally exceed the min/max bound of the sliding window. Without breaking the min/max bound can introduce detectable patterns in certain extreme cases, such as a gradient-like feature that has its maximal intensity at the top-left corner and minimal intensity at the bottom-right corner. We also design the padding scheme to follow the local distribution. The padding exhibits high entropy when the local variation is high, while degenerates value repetition with imperceptible perturbations while padding a flat area. As such, not only do the padding pixels exhibit less pattern, but it also prevents the padding pixels from breaking the features in the border region. We later show that a model still deliberately and incredibly built up PPP over time even with such a sophisticated padding scheme.
REVISITING PRIOR WORK
In this section, we first reproduce two experiments from the prior art, which aim to assess positional information from paddings. We show several critical design issues in these experiments and discuss how these problems affect the drawn conclusions. Finally, we propose two additional experiments to quantify the amount of positional information embedded in the paddings.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Background color as critical confounding variable in BHV test. We show that using a grey background similar to Figure 3 leads to discrepant results. The standard deviations are reported among 10 individual trials. We mark the best performance in green, and the worst two in red.
Pasking
Cone?
Black Background
Grey Backgrouud
Similar
Dissimilar (%)
(R)
Inconiistency (%)
Similar
Dissimilar (8)
Difl (9)
Incomsistency
Zeros
89. 24.40.98
89.83-20.00
89.244
3.21 6.35
0.98
-8.668
0.00
95.81 +: 2.07
18. 18.024 8.08
100.00 -00.00:: 0.00
100 000 0.00
4.96 5.93
4.774 6.52
-95.04
-95.23
96. 79-4 7.13
97.85 4.35
Cirvalor
80.31++a.2 23
80.31+
3.23
0.00
24.254
8.82
72.75+ 0.
72.75+
0.90
20.30. 6.55
39. 20 20 23
93.144
2.88
6.06
18.484 3.55
98 28.26:
92.40 4.23
-5.87
28.67 6.35
Reflect
N
Y
100.0040.00
100. 1000 00
11.70+16.88
12..77 72
-84.33
-8.30
91.18
97.384
0.16
19
100. 100.00 0.00
100. a 0.00
19.96113.54
17.10412.10
-82.84
-0.04
90.38 .55
98, 98.13 0.44
Replicate
N
Y
100. 00:00.00
98. 32.333
93:655
43 13.39:11.42
1:36
- -56.61
-4.67
32.601
75.32
4.87
100 00
97.17 0.48
33. 33.164 6.42
94.09
1/20
-66.83
-2.18
32. 32.15g 5.11
84, 84.09
Kanda
N
100. 100.00.0.00
100.00.40.00
11.31
10.80
-6.770
94.88
92.54 5.37
5.55
89.97 0.13
77.28_16.13
16. 13
15.47
66.70411.54
-64.50
-10.50
83.59 8.49
45.70
No-pad
100.00.0.00
3.21 8:35
-96.79
95.81 2.07
100.00 0.00
30.07 4.08
-69.93
81.30 2.44
3.1 POSENET
Islam et al. show zeros-padding provides CNN models positional information cues, and propose PosENet (Islam* et al., 2020) to quantify the amount of positional information encoded within CNN features. A PosENet experiment involves several components: a pretrained CNN model F, a shallow CNN Epem (i.e.. position encoding module), an image dataset X {z; } to examine, and a constant target pattern y (e.g.. 2D Gaussian pattern). PosENet first extracts intermediate features at k-th layer with ((.,)) = Fg (ara) using the pretrained CNN, and then optimizes Epem to minimize E, Epe (ft.e)) ) yi [a] Finally, the amount of positional information is quamtified by the average Spearman's 's correlation (SPC) and Mean Absolute Error (MAE) overall (pm(((( (f(,,) toward y.
A critical issue with PosENet is the use of optimization bassd metric. It is sensitive to hyperparam- eters with large variation. As shown in Table 2, for all the PosENet results, the standard deviation over five trials significantly dominates the differences between different types of paddings, and thus no definitive conclusions can be drawn. We also observed that PosENet can report NaN results in certain setups. Furthermore, PosENet quantifies the amount of positional information by the faithfulness of the final reconstruction. However, a better reconstruction does not have clear relationship to measuring the strength and significance of positional information. For instance, PosENet sometimes shows responses to no-padding models, demonstrating it is a metric with an indefinite bias pending on the memorization ability of Epem- Moreover, optimizing for pattern reconstruction is highly dependent on the underlying data distribution, simply changing the evaluation data distribution without changing the model weights can drastically change the PosENet numerical magnitudes and the conclusions of which model embeds the strongest positional information.
Another issue is that the no- no-padding scheme used in the Epem module in PosENet is known to have the foveal effect (Alsallakh al., 2021b; Luo et al., 2016), where model pays less attention the information on the edge of inputs. Using such a padding scheme for detecting positional information from paddings, which mostly concentrated on the edge of the feature maps, is less effective. This is an inevitable dilemma as PosENet aims to identify positional information from the padding of the pretrained F, while applying any padding scheme to Epem introduces intractable effects between the paddings of the two models.
3.2 F-CONV
Kayhan et al. propose a full- aull-padding scheme (F-Conv) (Kayhan & Gemert, 2020) and demonstrate is more translational invariant than the alternatives. One of the critical results is on "border handling variants" (Exp 2 of (Kayhan & Gemert, 2020)), which we call BHV test. The BHV test creates a toy dataset, where each image has black background with a green square and a red square in the foreground. The task is to predict if the red square is on the left of the green square (class 1), or vice versa (class 2). In addition, Kayhan et al. intentionally adds location biay such that both squares are located in the upper half of the image for class 1, and located in the lower half of the image for class 2. During testing, "similar test" inherits the same bias, while "dissimilar test" exchanges the bias (i.e., both squares are in the lower half of the image for class 1). As a truly translation-invarti CNN model should not be affected by the location bias, it should focus on the relation between
5

---

Under review as a conference paper at ICLR 2023
Pretrained
VGG19
ResNet50
Zeros
Circular
Reflect
Replicate
Randu
Zeros
Circular
Reflect
Replicate
Randa
X
ImageNet
Figure 4: Visualization of Position-Informatii Information Pattern from Padding (PPP). The visualiza- tions are calculated based on Eq. 3 480 GMap samples extracted at the 3rd layer-of-interest (Appendix A). The results show that the pretrained model significantly reinforces PPP compared to randomly initialized networks. Note that each image is normalized [0, separately, therefore the colors between images are not comparable. More visualizations are presented in Appendix B.
the red and green squares and perform similarly on both tests. Since the experimental results show that F-Conv performs best on the dissimilar test, it is concluded that F-Conv is less sensitive the location bias. The authors also conclude the circular padding performs worse due to the behavior of wrapping the pixels to the other side of the image, which leads to confusion between two classes.
However, as shown in Figure 3, we find the experimental design does not consider crucial confounding variable: the black back- ground has a zero intensity, making zeros padding the optimal padding that perfectly follows the background distribution. Ta- ble 1, we show that the dissimilar test is no longer in favor of F-Conv zeros after changing the background color grey. We also show that F-Conv replicate and F-Conv circular perform best on the dissimilar test, which different from the original observation.
Zuro Pukknn
Zero Pakkin
Figure 3: The BHV test trains binary classifier to predict the relative position of the two col- ored squares. hypothesizes if the padding provides no po- sitional information, the classi- fier will only focus the rela- tive position the two squares. (Lefft The black background is confounding variable. (Right) Zeros padding no- no-longer pads optimum values after changing the background color.
Finally, we report additional inconsistency rate to show that the CNN architecture used the BHV test actually has access the absolute position of the squares. Given random sample in class we create a trajectory two squares to the bottom of the canvas and recording the CNN- model prediction in all intermediate states. We label a trajectory to be inconsistent if the prediction of the CNN-model switches classes at any step of the trajectory. A CNN model with no access the absolute- -oositiin information should have all trajectories maintaining consistent predictions, with 0% inconsistency. Table I shows the inconsistent ratio 228 uniformly sampled trajecto- ries, where all models maintain high inconsistency rates, even with no-padding architecture. These results show that the CNN model used in the BHV test is not translation invariant. This can be attributed to that a CNN model has large receptive field covering the whole experiment canvas, therefore capable of gradually constructing absolute coordinates for each input pixel. Note that we only show the design of the BHV test is not suitable for quantifying the amount of positional information exhibited in a CNN model. Such a conclusion does not imply that F-Conv cannot potentially improve the translation-invart property of CNNs.
samples by simultaneously moving the
4 EXPERIMENTS AND ANALYSIS
Datasets Since most vision models are trained on tasks for recognizing objects, an image collection containing a diverse object appearance is more suitable for the task. As mentioned in Section 2.3, evaluating PPP requires images at a large field-of-view, in practice, we collect images at 2,048 x 2,048 pixels, which is larger than the receptive field of all the models large field-of-view, we compute PPP on three datasets (all at 2,048 x 2,048 pixcls): (a) 480 satellite images crawled from Google Map. (b) 1. 1,024 images synthesized by InfinityGAN (Lin al., 2022) trained with Flickr-Lann Lanrkk-LLndccppee dataset, and (c) 1,024 images synthesized by InfinityGAN trained with LSUN-Tower (Yu al., 2015) dataset. We crop the images depending on the requested input image sizes and principal point shifts from each model (see Appendix A for details). We will release the script for collecting and composing these large images.
tested. Due to the constraint of

---

Under review as a conference paper at ICLR 2023
Table 2: Comparing PosENet and our proposed PPP metrics. Most of the PosENet results are not distinguishable due to the high standard deviations. The standard deviation is computed by five different pretrained models for each test. The performance shows the accuracy (for classification) or weighted F-measure score (for saliency object detection). We use 2D Gaussian as PosENet reconstruction pattern, and PPP-MAE measured at the 4th layer of interest. Here, (f) indicates a higher value corresponds to stronger positional information or better performance on the task (vice versa for (4)). For each group of pretrained models, we label the strongest positional information response with red, and the experiments within its standard deviation range with orange.
PosENet
Model
Padding
Eval Dataset
PPP MAB(ours) (t)
Performance (1)
SPC (t)
MAE (L)
GMap
0.107
0. 0.196g0.000
0.0176
InlimityCAA--lick
0.368
0.183 00T
0.0163 20.000
74.0972200.0m0
InfiniyGAAN Lower
0.4922. 106
0.173gg 010
0.0179 000001
GMap
0.098
0.197 007
0.0158 000000
Circular
InfinityGAN flickr
0.323g..114
0.185.g
0.0137
74.471 0863
InfiiiyGAN:towe
0.460+0.102
0.176 000
4.0184=000005
GMap
0.109
0.196 007
0,0158 200000
Reflect
InfinityGAN
0.343+0.158
0.185=0 008
0.0146.0. 400000000008
74.0516 10051662..022
InlinitGGAN-wwerr
0.460+a.113
0.177+40.00
0.0168-0.00m
VGG-19
GMap
0.08110.187
0.197
0.0111
Replicate
InfinityGAN-liccs
0.356g..111
0,184go 007
0.0128 20.00012
73.996420.10
InfinityAAN Lower
0.498
0.173 010
0.0156.0.00
GMap
0.12540.164
0.195 000
0,0182 =0.0012
Randn
InlimityGAN flick
0.374
0.374+a.15 +0.187
0.185 007
0.0167 0.0167-00.0008 -0.0008
73.7716 0000055
InfiiityANN.owe
0.421 40.161
0.181
0.01 20.0012
GMap
0.001+0.250
0.204+0.013
0.0000-000000
NoPad
InfinityGAN.no flickr
0.303:4.192
0.187
0.0000
62.0396gg
InfniiyGAN-cower
0.51614.139
0.172g0.014
0.0000-0.0000
GMap
0.191
0.193
0.0162
Zems
InfinityGAN,fickr
0.682+0.100
0.152=a0.0z 019
0.0137=0.0004
75.6856+0.022
InfnityGAN-cower
0.721+0.077
0.1444:0..01
0.0153.00.0013
GMap
0.398 0.398g0.130
0.197=0.00f
0.0188 10.0016
Circular
InfinityGN--lickr
0. 0.62840..88.
0.159+0.013
0.0178
20178_200000 20.0005
76.1432+o.1020
lafnityGAN-ower
0 0.585+0.10s
0.165ga.014
0.0189 20.0022
GMap
0.1
0.192=0.000
0.0150=0.0000
ResNet50
Reflect
InfinityGAN-fickk
0.59440.000
0.169#0.012
0.0134
75.5068
IniniyGAN-tower
0.667.20.087
0.153go.0o0
0.0157 20.002
GMap
0.249+0.192
0.189
000
0.0000
Replicate
GAN-tick
0,700 4.. 09G
0.1/17
g0 018
0.0114=0.o0m
75.6122gp.0011
JalniiyGAN-tower
0.726 0.000
0.142 =0.00000000 016
0.01 00000007
GMap
0.210+a.192
0.191 000
=0.10700.0007
Randn
InN-nityGA-llikk
0.56610.100
0.171 10 011
0.0122:0000m
75.307620.10r
InfimityAAN-uuee
0.714g. oos
0.142+
0.0153g0.00m
GMap
0.156+a.212
0.201 + 017
0.004920.0000
Zeros
InfinityGAN-iicke
0.36510.140
0.184
0.0036
0.6269
InfinitGAAN-cowc
0.44940.120
0.179g0.000
0.0032:0.0001
GMap
0.011 200
0.20720.014
0.0062
Circular
InfinityGAN-fickr
0.329 133
0.187 012
0.0008 20-000
0.6260 0009
InfinityGAN tower
0.398
0.182
0.0050 20.002
GMap
0.062g0.210
0.205
0.005820.0001
Reflect
InfimityGAN -lickrick
0.322 133
0.188 013
0.0030
0.6243 0022
lafnityGA-cowwr
0.39640.126
0.183#0.018
0.0039
(PICANet)
GMap
0.07110.215
0.204
0.004320.000
Replicate
InfinityGAN-llick
0.335g..159
0.186gg..012
0.0023-20.000
0.6255g0.00s
InlmityGAN--uwwe
0.409
0.182
0.0032_0.00m
GMap
0.002
0.202 009
0.0001=0.0m00
Randn
InfinityGAN GAN-Biccr
0.228
0. 0.197
0.0001 0000120.0000
0.2370
InfinityGAN
0.21240.149
0.200
0.0001 0.0001=0.00m =00000
GMap
0.000 1.000....00 a6s
0.211 019
0.0000_0.oooo
NoPad
InfinityGAN flickr
0.454 40.144
0.178 (178+a0.022
0.0000
0.4759 + 2.1033
InfnityGAN
0.520 2..167
0.172 2..220
0.0000
7

---

Under review as a conference paper at ICLR 2023
Table 3:
Significant PPP gain from model training. We measure PPP-MAE on GMap with randomly initialized and fully trained models. The results show a consistent and significant increment of PPP is developed through the model training.
Padding
Model
Pretrained
Circular
Reflect
Replicate
Randn
VGG-19
ImageNet
x
0.0132 Do0G 0.017610 0000
0.01 1o 0006
1.000044.0000
0.01 58 0002
0.0000 0oco
0.0000 2o. po80
0.014440 0008
0.0000 0,0000
0.0182 200002
ResNet50
ImageNet
x
0.0162+0.0012
0. 0.00.2+0.0000
0.0188
0.0032 +0, 0004
0.001 8= 00000
0,0150 .0004
0.00 15 0001
0.0150 0000
0.01 =000007
0.0020 00.0002
4.1
VISUALIZING POSITION-INFORMATTON PATTERN FROM PADDING (PPP)
We start with visualizing PPP in Figure 4. All the visualizations are conducted at the 3rd layer of interest as detailed in Appendix A. We compute PPP using Eq. 3 and l norm as the distance metric, then average the resulting PPP in the channel dimension to generate gray-scale image. Since the quantities are small and difficult to perceive, we normalize the gray-scale image to [0, 1] range, and thus the colors between images not directly comparable.
In all scenarios, PPP noticeably spreads out after being pretrained on ImageNet. In Table 3, the PPP- MAE of the VGG19 and ResNet50 also reflects that the response of PPP is significantly strengthened after model training. That is, the model training has substantial effects on the construction of PPP. Although the formation of padding pattern is suggested to be mainly caused by the distributional difference between features and paddings (Alsallakh et al., 2021a), our results show that it only increases the response slightly, compared the considerable PPP-MAE gain through training.
Another intriguing observation is that, despite some variations in the detailed patterns, the overall structure of PPP remains similar. Regardless of padding minimum values with zero aroo-padding (consider the features are processed with ReLU activation), randn-padding padding that can sometimes produce large quantities by chance, or the unbalanced initial state of ResNet50 caused by strided convolution (the first row of ResNet50 in Figure 4), all models tend to have the maximal PPP response in the corner of the features after fully trained. While the underlying mechanism causing such consistent preferences remains unknown, such preferences may be an important factor to consider in future model design.
4.2 QUANTIFYING PPP AND COMPARING WITH POSENET
Table 2 shows the measurements of PPP and PosENet on various architectures and padding schemes. We train five models for each setup and measure the standard deviation of these models. Our PPP-MAE has significantly lower standard deviations compared to PosENet, where the standard deviation of PosENet dominates the differences between padding variants, and thus the quantities from PosENet cannot provide sufficient information for any analysis. Evaluating the true mean of PosENet requires an even larger number of pretrained models, each requiring full training on the target dataset (e.g., ImageNet), which is impractical in reality. The main reason that PosENet has such a large variation is due to its optimization-hant based formulation, and thus the final quantities highly depend on the convergence of the PosENet training. In fact, we also observe similar level of standard deviation even when the PosENet is measured on the same model for multiple trials. On the other hand, PPP based on a closed-form formulation, and thus the variations are only introduced by the differences among the parameters of the pretrained models. Furthermore, PosENet often reports positive SPC responses from no no--addiing models, as shown in its large standard deviation. In contrast, PPP has zero response to no-padding models by definition, and therefore is less biased for measuring the positional information from padding.
Although certain paddings seem to have slightly lower PPP-MAE than other paddings, in Table 3, we find the differences are not significant when comparing the extremely low PPP-MAE from most of the randomly initialized networks. In most cases, the network can effectively construct its PPP, even with the highly stochastic randn padding. The only exception seems to be the case of randn padding in the salient object detection (SOD) task, where the network fails to achieve compatible performance with other paddings The results show that the model training plays an important role
We use the same setting PosENet to evaluates PiCANet (Liu et al.. 2018) on the SOD task. PiCANet is initialized by model pretrained on ImageNet (with zero padding). The discrepancy in the padding scheme can he the major cause of failure while training the network on SOD task with randn padding.
8

---

Under review as a conference paper at ICLR 2023
Lsverol moum 2 pMt)
230
1015
VGG19
5508
280
600m
4.00
2802
600m
50030
100M
130
5506
ResNet50
400K
6039
5C19
197
600
Figure 5: Chronological PPP. We quantify PPP every 10 epochs and plot its development in four different layer of depth (the rightmost layer the one closest model output). All curves consistently show sudden surge the early stage, and all the later layers are slowly but steadily gaining stronger PPP until the end of training. The shadow region represents standard deviations among 5 individual training episodes. The colors represent zeros, circular, reflect, replicate, and randn paddings.
in the formation of PPP, and perhaps its contribution is much larger than which underlying padding scheme is being used. This motivates us to further analyze the PPP formulation during model training.
4.3 CHRONOLOGICAL PPP
To understand the formulation of PPP through time, we snapshot checkpoints every 10 epochs for all training episodes. By measuring the PPP-MAE at all the checkpoints, we plot chronological curve and monitor the progress of PPP. We train 5 individual models for each pair of model pandel-padiing setting and report the standard deviations, which demonstrates the significance of the trend.
Figure 5 shows all models achieve significant gain of PPP within the first 10 epochs in all inter- mediate layers. Most models continuously increase their PPP as training proceeds, especially in the fourth layer of interest, which is the last output from the convolutional layers before the final linear projection. Another interesting observation is that our randn padding, which is designed be less easily detectable with built-in stochasticity, indeed shows less PPP built-up at the intermediate stages in certain layers. However, the network still adjusts the behavior and ends up forming complete PPPs at the fourth layer of interest in all scenarios. All these evidences show that the network builds PPP purposely as a favorable representation to assist its learning.
5 CONCLUSION AND LIMITATIONS
In this paper, develop reliable method for measuring PPP and conduct series of analyses toward understanding the formation and properties PPP. Through a large-scale study, we demonstrate that part of its learning process, and its
PPP is representation that the network favorably develops formation has weak connections to the underlying padding algorithm. We show that reliable PPP metrics are important steps for understanding the effects of PPPs in different tasks, and useful for measuring the effectiveness of future methods in debiasing PPP.
However. an unfortunate and inevitable limitation of the PPP metries is that their measure is biased by the model architecture and parameters. Since the PPP metrics are based on the distributional differences between the paired model outputs (i.e., optimal padding to algorithmic padding), different architecture and layers of depth exhibit different and intractable biases due to different interactions between PPP and model parameters. Such a bias makes PPP metrics less comparable while dissecting models with different architectures or parameter distributions (e.g., weight decay and weight normal- ization), which important for studying the effect of architectural changes. However, this limitation is inevitable for any (and all existing) metric that attempts measure PPP using the outputs of model. We note future studies measuring PPP without model inferences will be an important step toward tackling and understanding the property of PPP under different architectural choices.
A related analogy of the contradictory prohlem can be found in neural architecture search literature (Mellor al., 2021), which aims assess the final performance of architecture without training the model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Antonio Alguacil, Wagner Gonçalves Pinto, Michael Bauerheim, Mare C Jacob, and Stéphanc Morcau. Effects of boundary conditions fully convolutional networks for learning ppatio-temporal dynamics. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2021. 1, 3,
Bilal Alsallakh. Narine Kokhlikyan, Vivek Miglani, Jun Yuan, and Orion Reblitz-Richardson Mind the pad- (enn)s can develop blind spots. In International Conference Learning Representations, 2021a. 1, 2, 3,
Bilal Alsallakh, Vivek Miglani. Narine Kokhlikyan, David Adkins, and Orion Seblitz-Richaarssnn Are convolu- tional networks inherently foveated? In SVRHM 2021 Workshop at NeurlPS, 2021b. 1,5
Jooyoung Choi, Jungheom Lee, Yonghyun Jeong, and Sungroh Yoon. Toward spatially unbiased generative models. In IEEE International Conference on Computer Vision, 2021. 1,3
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh, Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.036.2 2022. 1, 2,
Carlo Innamorati. Tobias Ritschel, Tim Weyrich, and Niloy Mitra. Learning on edge: Investigating boundary filters in cnns. International Journal of Computer Vision, 2020.
Md Amirul Islam", Sen Jia", and Neil D. B. Bruce How much position information do convolutional neural networks encode? In International Conference Learning Representations, 2020. 1,5
Amirul Islam, Matthew Kowal, Sen Jia, Konstantinos G. Derpanis, and Neil Bruce. Boundary effects in [enn]s: Feature or bug? https: ht////ppeeeviieeeeeeeeee net/ forum? iinnnidd--aa d-M4 (XdW44 2021a.
Md Amirul Islam, Matthew Kowal, Sen Jia, Konstantinos G Derpanis, and Neil DB Bruce. Position, padding and predictions: A deeper look position information in cnns. arXiv preprint arXiv:2101.12322 202
Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In IEEE Conference on Computer Vision and Pattern Recognition, 2020. 1,5
Kuangliu. pytorch-cifar. https: / github. com http://aagggilppppooohheeheeeeee
2017. 12
Chich Hubert Lin, Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, and Ming-Hsuan Yang. InfinityGAN: Towards infinite-pixt pixel image synthesis. In International Conference on Learning Representations, 2022. 3,
Nian Liu, Junwei Han, and Ming-Hsuan Yang. Picanet: Learning pixcl-wise contextual attention for saliency detection, In IEEE Conference on Computer Vision and Puttern Recognition, 2018. 8
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li, Soft rasterizer: A differentiahle renderer for image-hased 3d reasoning. In IEEE International Conference on Computer Vision, 2019. 3
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Neural Information Processing Systems, 2016. 1,
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning 2021. 9
Evangelos Ntavelis, Mohamad Shahbazi, Jason Kastanis, Radu Timofte, Martin Danelljan, and Lue Van Gool. Arbitrary-scale image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2022. 1,3
Oskyhn. Cnns-withouttbordees httpps///ggthu....o///s///////// Bordttt--er.....t..ddroe... 2019. 12
Pytorch. vision. tttpon/...r////thh...ooo/rrooorr
2016. 12
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Yen Lo, Justin Johnson, and Georgia Ckioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501 2020. 3
Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy. Positional encoding as spatial inductive
bias in gans. In IEEE Conference on Computer Vision and Pattern Recognition, 2021. 2,
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 106.03335 2015. 6
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features a perceptual metrie. In IEEE Conference Computer Vision and Pattern Recognition, 2018. 3
10

---

Under review as a conference paper at ICLR 2023
Supplementary Material
APPENDIX A IMPLEMENTATION DETAILS
A.1
ARCHITECTURE AND FEATURE ALIGNMENTS
Regular Padding
ptimal Padding
Feature Aligmment
VGG19
Testarn (P1) Principel
Sba Eaa Princded Paint (P1)
Trpd alt Mangle
Ret Algeneee
Crop
1111%
2x Con/2dlleeB, sml)
Soole
MasRoolateeee 9H10
28% 50.5
2x Com:2t/lew gmi)
12:1122
025 53
HILS 11
35
F11 10850 36-300
20c8
(014 259
12202
Com/2t(les3, s-))
inlernt PCEN-OL Ne
ModRto2atl=-2 =-2)
4xx Cam22t/1=44 5-4)
(Big 13.g
BR 2008- B:4)
1805 105
Cam/itlle4, s=1)
db Leve of interol
14r14
Regular Padding
Optimal P Padding
Feature A lignment
ResNet50
PTP
613663
Input Crop
Com/2d (k=4 s=2)
1121112
145 48
306.00
(15 150
MasPooiidilc-2, s-2)
58636
(24.0))
(6.30
ResEllocklls4, s-1)
2x Reselloddlee s=1)
Resélloddte4, s=2)
06 30
3x Restllockteerd s=1)
2el Layer of latrot
A6.38
15
66 36
18
RCPh ls 26
Reséllockke44,
14a38
Tasti
5x Deeelllleeeeeeeee s-1)
Resölockd--. s-2)
03.3
2x Res sml)
tn Layer hnveet BL-FO
13.30
13.30
Figure 6: The architecture for VGG19 and ResNet50 used in the paper. We mark the calculation of optimal padding in orange arrows and principal point in blue arrows. We label the layers of interest that are used in the paper. The red indicates where a principal point shift is identified.
11

---

Under review as a conference paper at ICLR 2023
A.2 PPP FEATURE MISALIGNMENT
There are several pitfalls in visualizing and quantifying PPP. We identify two critical pitfalls from the architectures we implemented. However, these may not be sufficient to cover all potential issues while integrated into other architectures. Therefore one must be alerted to any unusual behavior (e.g., Figure 2(d) in the main paper) throughout their implementation.
Principal point shifting. Conv2d has hidden behavior that few people are aware of, the operation one-pixel skewed while applying stride-two Conv2d on even-shaped features. To understand how the one-pixe mixel shift happens, we first define the principal point of feature map. We first define the principal point of the last feature map as the center pixel (note that we define as the middle-po -oint between the center-two pixels in case the last feature size is even). Then, we recursively define the principal point of the (N - 1- layer as the pixel that positions at the center of the Conv2d receptive field that mainly forms the principal point of the N-th layer. In the case of rrtimally-padddee features, the principal points in every layer are the center of the feature map. But, as shown in Figure ((a), the principal point of algorithmically-padeee convolution is applied to even-shaped features, which can be further amplified as more layers stack up. Such skew causes the principal points of algorithmically padded features shift several pixels away from the principal points of optimally-paddes -aadddd features. As PPP metrics use pixel-wise subtraction to distinguish the image content from PPP, the misalignment becomes a critical issue, since the image contents are no longer aligned and subtractable.
features will have one- one-pixel shift when stride-2
In Figure 6, we show the procedure of calculating the principal point in blue arrows and marking the values impacted by principal point shift with red f. For the ResNet50 architecture, the principal point shift accumulates to 16(= 224/2 96) pixels in the early layers.
Fortunately, such a displacement can be fixed by adding corrections to how we calculate the feature margins. As shown in Figure 2(b), the concept of the margin correction is to make the two principal points overlapping each other after adding the margin. In the example, the left-right margins are corrected (209, 180) (instead of the more intuitive choice of (195, 194) or (194.5, 194.6)).
We also show how the principal point shift visually looking like in Figure 2(c), notice the patterns have right-botton shifted 16 pixels. As shown in Figure 2(d), failing identify the principal point shift will result in checkerboard artifacts while calculating PPP, and adding correction eliminates the artifacts.
Maxpooling misalignment. This is hypothetical condition that may potentially happen but has not been observed in the three architectures we tested. Consider a case of a Maxpooling layer of window size 2 and stride 2, the sliding windows of each pooling operation have overlap, therefore the initial index of the first sliding window solely determines the spatial location of all sliding windows. Accordingly, there is chance that the initial condition of the aptimally-paddee features causes all of its sliding windows to be one-pixel misaligned to the soorithmicalll-paadee features. Fortunately, the condition can be easily determined by calculating the top and left margins of the feature alignment (similar to the aforementioned principal point shift calculation). For the case of a Maxpooling layer of window size and stride 2, the misalignment will not happen if the top and left margins are even numbers, and that is exactly the case for VGG19 and ResNer50, as shown in Figure 6.
A.3 RANDN PADDING
critical implementation detail is that such a padding scheme must be applied before activation func- tions. Since the paddings are based on the distribution within sliding windows, activation functions such as ReI.U, which clamps all negative values, can discard a significant amount of information be- wwwpoooowiiioooooonniaaaiiiioiii
we modify forehand. Instead of the traditional use of activation. Note that such change of order does not the order to convolution-normaliaaattt wow.anozaaionn satpiiittttoonddnnnn--iee affect the behavior or results of other padding schemes.
A.4 ACKNOWLEDGING
CONTRIBUTORS
Our implementation reuses codes from several open-source codebases, which greatly supports development. The repositories used in the paper are F-Cony (Oskyhn, 2019), torchvision (Pytorch, 2016) and Pytorch-cifar (Kuangliu, 2017).
12

---

Under review as a conference paper at ICLR 2023
APPENDIX B
MORE PPP VISUALIZATIONS
VGG19
Layer of Interest
Pretrained
Zeros
Circular
Reflect
Replicate
Randn
1
ImageNet
2
ImageNet
3
ImageNet
1
ImageNet
Pattern from Padding (PPP). The visualizations calculated based on Eq. over 480 GMap samples. The results show that the pretrained model significantly reinforces PPP compared to randomly initialized networks. Note that each image is normalized to (0, 1] separately, therefore the colors between images are not comparable.
Figure 7: Visualization of
13

---

Under review as a conference paper at ICLR 2023
ResNet50
Layer of Interest
Pretrained
Zeros
Circular
Reflect
Replicate
Randn
ImageNet
TmageNet
3
ImageNet
ImageNet
Figure 8: Visualization of Iesition-Informatiion Pattern from Padding (PPP). The visualizations are calculated based on Eq. over 480 GMap samples. The results show that the pretrained model significantly reinforces PPP compared randomly initialized networks. Note that each image is normalized to (0, 1] separately, therefore the colors between images are not comparable.
14

---

Under review as a conference paper at ICLR 2023
SOD (PiCANet)
Layer Interest
Pretrained
Zeros
Circular
Reflect
Replicate
Randn
ImageNet
ImageNet
ImageNet
ImageNet
ImageNet
Figure Visualization of Informmaiino Pattern from Padding (PPP). The visualizations are calculated based on Eq. over 480 GMap samples. The results show that the pretrained model significantly reinforces PPP compared to randomly initialized networks. Note that each image normalized to (0, 1] separately, therefore the colors between images are not comparable.
15


Reviews:
[{"review_id": "vayLd3O7ipv", "reviewer": "Reviewer_rkz4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the Position-information Pattern from Padding (PPP) encoded in neural networks. PPP is defined as difference between algorithmically and optimally padded activations, then SNR and MAE are computed on the difference to quantify how much PPP exists. The result shows a pre-trained CNN model contains a lot of position information. Comparing with the two previous studies, this paper is claimed to be more robust and reliable for PPP evaluation.\n", "summary_of_the_review": "Considering all of the strengths and weaknesses, I would not recommend this paper to be published at this moment.", "strength_and_weaknesses": "Strength: This study investigates a fundamental question that affects all convolutional neural networks used in computer vision. The proposed method seems to have lower variability than the other methods.\n\nWeaknesses:\nThis study did not discuss how the PPP affects CNNs in practice. The proposed padding did not compare against other commonly used paddings in vision application to show its efficacy, which is not convincing. The padding scheme designed in Section 2.4 is to weaken PPP detected by eq3, which is based on an optimal padding solution as shown in eq2. It is better to discuss why cropping from a bigger picture can be considered a theoretical optimal padding. Having more content(pixels) could (slightly)deliver more semantic information, but I do not think the information can be associated with PPP, and the proposed padding is also algorithmical solution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tBnx-CfTJNq", "reviewer": "Reviewer_D3Ci", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues based on prior work that the encoding of positional information from padding, abbreviated PPP by convolutional neural networks is an undesired property that can have unwanted side-effects. The degree to which PPP is learned as a consequence of different padding techniques is investigated and a new way to quantify it is proposed. In addition a new padding technique is explored, which could help to lower PPP.", "summary_of_the_review": "While I like the paper, I feel it needs more work in terms of clarity and presentation to make it acceptable.\n", "strength_and_weaknesses": "Strengths\n---------\n- Investigating PPP makes sense and the method proposed to do this is relatively simple.\n\n- The experiments are thorough and relatively large in scale showing clear results.\n\n- An ability to reliably quantify PPP is a requirement for further understanding of its consequences and possibly ways of mitigating its effects.\n\nWeaknesses\n----------\n- Some table and figure captions are lacking details and/or are confusing.\n  - Figure 1 is placed and referenced so early in the manuscript that it is hard to understand what it contains. E.g. what is the sampling process, how are the two signals generated, how is PP quantified? What is the right column in relation to the other four?\n  - I suggest you explain what the numbers indicate in table 1. Also this is a very large table with a lot of information that is only referenced once in text and in a single short sentence. Is all of this information really necessary?\n  - Table 2 is also relatively large, and I find it hard to deduce from the text what numbers should be compared and how this relates to the title in bold. Consider helping the reader a bit more.\n  - Table 3, here the caption seems a bit misleading. \"The results show a consistent and significant increment of PPP is developed through the model training\". Somehow this seems more fitting as a description of Figure 5, as table 3 seems to be about pretraining and it only compares pretraining with no pretraining.\n\n- As far as I understand Table 2 it shows the different possitional encoding measurement metrics (SPC, MAE, and PPP-MAE) for the different padding schemes and models. In section 4.2 it is concluded that because the standard deviations are larger than the differences between the padding variants, the PosENet based metrics cannot be used to say much about the benefits of these padding methods. While this may be true, as far as I can deduce, this does not necessarily indicate a problem with the PosENet based metrics, it may also just be the case that PPP varies a lot over training runs.\n\n- An intuitive explanation for the equations given (1, 2, and 3) would help understanding.\n\n Other smaller annoyances\n  - Consider avoiding the abbreviation of method names in the abstract or write what they are.\n  - \"Padding, one of the most fundamental components in neural network architectures, has received much less attention than other modules in the literature\", what do you mean by this? What is a module in this context?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TU_0YM-Yz6", "reviewer": "Reviewer_Mohz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new metric to detect \"Position-information Pattern from Padding\" which is an artifact resulting from different types of padding applied throughout a convolutional neural network.", "summary_of_the_review": "The paper presents a new metric but fails to demonstrate the value beyond an assumed importance based on existing published work. The paper is potentially impactful but it is not clear to me what the negative issues are with the model knowing where the border is. I think this work should directly have experiments with that generalization goal in order to demonstrate relevance.", "strength_and_weaknesses": "Strengths: The paper takes a systematic approach to the task. \n\nWeakness: I don't believe the authors present a convincing argument for why this problem is important and their experiments don't demonstrate the value of solving this problem in terms of model generalization.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SPvumMn3F4", "reviewer": "Reviewer_5amb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the effect of convolutional neural networks encoding position information due to padding. A new metric for measuring and visualizing the encoded positional information named Position-information Pattern from Padding (PPP) was proposed. The paper then discusses some of the disadvantages of the existing metrics based on PosENet and tests in F-Conv and show through experiments PPP has the potential to fix these disadvantages.", "summary_of_the_review": "The paper investigates an interesting topic of position information related to padding and includes a lot of well-thought-out discussions. The experiments seem to indicate some practical value of the proposed PPP metric although the experimental setup might be a little simplified. ", "strength_and_weaknesses": "# Strength\n\n1. Defining a novel metric for quantifying position information due to padding.\n\n2. Systematically discussed the issues with existing metrics\n\n# Weakness\n\n1. PPP metrics are potentially biased by the model architecture and parameters.\n\n2. Only some basically networks were used in analysis so generalizability to more SOTA network architectures still needs exploration.\n\n3. I wonder if the authors can illustrate some of undesired effects of padding on certain real world examples in addition to just showing the edge view of those blue-yellow blocks. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject