Title: PAD-NETS: LEARNING DYNAMIC RECEPTIVE FIELDS VIA PIXEL-WISE ADAPTIVE DILATION

Abstract: Dilated convolution kernels are constrained by their shared dilation, keeping them from being aware of diverse spatial contents at different locations. We address such limitations by formulating the dilation as trainable weights respect to individual positions. We introduce Pixel-wise Adaptive Dilation (PAD), a light-weighted extension that allows convolution kernels to flexibly adjust receptive fields based on different contents at pixel level. By inferring dilation via modeling inter-layer patterns, PAD-Nets also provide a possible way to partially understand the hierarchical representations of CNNs. Our evaluation results indicate PAD-Nets can consistently outperform their conventional counterparts on various visual tasks.

1. INTRODUCTION:

The power of prestigious Convolutional Neural Nets (CNN) (Simonyan & Zisserman, 2014; He et al., 2016; Huang et al., 2017) relies on the ability of hierarchically representing spatial features across input regions called Receptive Fields (RFs) (Luo et al., 2016) . Common practices usually prefer large RFs in order to achieve superior performances, making Dilated Convolution Kernels (DCKs) (Yu & Koltun, 2015; Yu et al., 2017) a favorable choice due to the ability of exponentially enlarging RFs while keeping kernels small. To further improve the dilated kernels, two obvious problems, which universally reside in most of existing dilated CNN structures, need to be properly tackled. First, all the weights share a single dilation value across all pixels. This could be very counter-intuitive as resultant monosized RFs are less capable of encoding huge spatial variances from inputs. Second, dilation selection is data-independent, requiring strong knowledge on input contexts to pick the proper value, which may only be suitable to a certain set of experimental settings.

In this paper, we answer the above challenges by combining the dilation selection with conventional CNN modules and incorporating them into a unified data-driven framework. We propose Pixelwise Adaptive Dilated Nets (PAD-Nets), a simple yet powerful extension for general DCKs, which treats dilation values as learnable weights and can be jointly optimized with other CNN weights in an end-to-end fashion. As shown in Figure 1 , in the newly formulated PAD kernels, dilation is learned to change at different input positions to reflect input spatial diversity, resulting in dynamic RFs with irregular shapes in a single convolution layer. In practice, there are two major difficulties to overcome.

How to decide the dilation value online? We handle this by regarding the dilation as a function of input at individual pixels. More specifically, the function samples dilation values through certain probability distributions that are conditioned by pixel-wise input features. To solve indifferentiable nature of general sampling process, we approximate it by employing Gumbol-Softmax (Jang et al., 2016) as a differentiable estimation in order to keep PAD end-to-end trainable.

What are proper dilation values for inputs? Since there is no clear explanation on how network layers work, we believe that it still remains an open question and can only be answered with valid hypotheses. For PAD kernels, we make the assumption that dilation values are related to inter-layer patterns between convolution layers due to their hierarchical nature. In such cases, RF size at each location is adjusted based on information flows between corresponding inter-layer pixels during forward propagation. The rest of this paper is organized as following. We review relevant literature in Section 2, then PAD-Nets are elaborated in Section 3. Sections 4 and 5 demonstrate experimental results, and Section 6 concludes the paper.

2. RELATED WORK:

Content-adaptive networks This research direction is focused on building dynamic internal structures via data-driven approaches to better leverage larger spatial variations from inputs. A set of related techniques tend to develop differentiable approximations for traditional image-adaptive filters and integrate them as end-to-end trainable layers for CNN models. For example, Jampani et al. (2016) includes bilateral filters (Aurich & Weule, 1995; Tomasi & Manduchi, 1998) in CNN models as a layer for character recognition; Wang et al. (2018) and Wu et al. (2018a) introduce their trainable version of non-local means filters (Buades et al., 2005) and guided filters (He et al., 2012) , respectively. These approaches conduct content-adaptive enhancements in separate layers without interacting with convolution kernels. Another set of techniques propose the idea of directly generating kernel weights based on layer inputs (Xue et al., 2016; Jia et al., 2016; Su et al., 2019) , and extend it with attention mechanism (Wu et al., 2018b) as well as other task-specific improvements. However, most of them rely on additional modules with large kernel sizes, being incapable of scaling up to more general network structures.

Dynamic receptive fields. Comparing to the above approaches to build content-adaptive nets, there is much less work aiming at enabling the content-aware ability via adjusting receptive fields (RFs). Majority of RF-related researches focus on how to effectively enlarge RFs in order to achieve better performance. Among them, dilated convolution kernels (Yu & Koltun, 2015) become a popular choice as it can exponentially increase RF sizes while maintaining small kernel sizes. However, this could also lead to negative impacts, such as sparsity and "gridding" effect (Yu et al., 2017) . Unlike static RFs produced by dilation, recent works such as Dai et al. (2017) and Zhu et al. (2019) argue that RFs should be more diverse in order to capture rich spatial variations. They propose deformable CNNs that learn to adjust the positions for convolving, resulting in free-form RFs that are totally data-dependent. Besides, Shelhamer et al. (2019) attempts to create diverse yet controllable RFs by composing the structured Gaussian kernels and unstructured ordinary convolution kernels.

3. PIXEL-WISE ADAPTIVE DILATED CONVOLUTION:

Now we elaborate the proposed approach for extending conventional dilated convolution kernels into PAD kernels. Without loss of generality, we assume all the convolutions in the rest of this paper are 2D operations. Suppose K W,b;d is a dilated convolutional kernel with dilation value d, and X ∈ R w×h×c is input. The output of convolution between K and X is

Y i,j = K m=0 K n=0 (W m,n x i+dm,j+dn + b m,n ) (1)

where K is the kernel size and i, j are coordinates for dimensions w and h, respectively. Apparently, d is a constant variable independent to i and j. Our goal is to convert d into a function D i,j such that the output of D i,j could be aware of location-specific contents. More specifically, we treat D i,j as an inference process that generates dilation values by sampling from position-dependent hidden distributions. Figure 2 sketches the basic idea of a PAD kernel.

3 (Jang et al., 2016; Maddison et al., 2016) as D i,j to approximate the inference of discrete dilation values. Suppose that there are D valid options for dilation value, and

d i,j ∈ [0, 1] D

is the estimation of one-hot vector that corresponds to the dilation value at position (i, j), then sampling d i,j ∼ GS(h i,j ) can be achieved by

d i,j = D i,j (h) = exp((h i,j + g i,j )/τ ) exp((h i,j + g i,j )/τ ) (2)

where means summation of all tensor elements here; h, h i,j are content-related hidden priors and their subtensors at each positions, respectively; g i,j ∈ R D are i.i.d. samples drawn from the Gumbel(0, 1) distribution and τ controls how much the GS is close to a true categorical distribution.

3.2. HIDDEN PRIOR GENERATION:

As mentioned in Section 1, we believe dilation adaptation should be governed by feature hierarchy and build up our dilation inference mechanism upon inter-layer pattern modeling to capture dependencies between abstraction levels. Inspired by Lin et al. (2017) ; He et al. (2017) and Kirillov et al. (2019) , we consider aggregation as a feasible way and will generate hidden priors h through sequentially aggregating multiple Y from hierarchical layers. Suppose l is the newly added layer index, then there are several aggregation options for inter-layer patterns modeling.

Recurrent Aggregation. A straightforward way for sequential aggregation can be written as

EQUATION

where W l h and U l h are 1×1 kernels weights with output channel of D; f (•) is a non-linear activation function. In this case, h l i,j continuously accumulates information from each layers as l goes deeper, implying layers are highly dependent with each other to mutually decide proper RF sizes.

Gated Aggregation. To model inter-layer pattern smarter, we introduce a gate variable a l h to modulate information from each layer in a data-driven manner. We use a similar way to Hochreiter & Schmidhuber (1997) and Chung et al. (2014) for computing a l h , with which the entire aggregation can be formulated as following

EQUATION

Table 1 : mIoU for feature level study. σ 2 (d i,j ) is variance of pixel dilation sampling.

CONV3 CONV4 CONV5:

σ 2 (di,j) MIOU 1.96 × 10 -4 63.9 1.84 × 10 -4  64.7  4.01 × 10 -6  66.5  2.45 × 10 -4  65.4  1.24 × 10 -4  66.1  1.93 × 10 -4 65.9 

EQUATION

where σ(•) is the sigmoid activation and • means element-wise multiplication. In this way, layers are not strictly dependent following their hierarchical order and will impact dilation sampling in a more complicated way.

Markov Aggregation. An important extreme case of Recurrent Aggregation, Markov Aggregation sets the kernel weights W l h from equation (3) to 0. Similar to the Markov model (Gagniuc, 2017), this means RF sizes are dominated by the last layer. No other inter-layer patterns need to be aggregated for multiple hierarchical layers.

3.3. DISCUSSION:

Basically, PAD is a light-weighted extension for general CNN structures, which means it introduces little extra weights for dilation inference and is independent to feature channel size. More specifically, all the weights brought with PAD have an output channel of D, while other RF adaptation works such as Dai et al. (2017) often rely on additional modules that need to match the same output channel size of features. Since D is usually much smaller than feature channel size, PAD kernels are easier to be deployed at higher level convolution layers.

Moreover, the proposed dilation inference grants a manageable way to partially understand the feature hierarchy. Through learning RF adaptations based on inter-layer pattern aggregations, we can better explore the principles behind inter-layer interactions via analyzing dilation changes. It is also expected that inferring proper dilation values according to inter-layer patterns may help comprehend the differences between low and high level convolution layers. Such information could be invaluable knowledge for designing interpretable architectures. Furthermore, the flexibility of inter-layer pattern modeling allow PAD-Nets to be attuned to various tasks and applications.

4. PAD-NETS FOR SEMANTIC SEGMENTATION:

Since the proposed PAD module is highly related to RF adaptation, dense prediction tasks could be ideal to test its effectiveness. Thus, we first evaluate PAD-Nets through semantic segmentation to explore their properties from various aspects. We will show that PAD-Nets is designed for general purpose and can be applied to solve more problems in later sections.

4.1. GENERAL EXPERIMENTAL CONFIGURATIONS:

We implement PAD-Nets with various backbone architectures via PyTorch library (Paszke et al., 2017) . Unless otherwise specified, we will employ VGG-16 (Simonyan & Zisserman, 2014) as backbone net and follow the same training protocol as FCN-8s (Long et al., 2015) for evaluation. The default dataset is Pascal VOC 2012 (Everingham et al., 2010) and we report mean Intersection over Union (mIoU) on its validation set as evaluation results. All the models will be optimized via Adam optimizer (Kingma & Ba, 2014) . All architectural descriptions can be found in Appendix A.

4.2. FEATURE LEVEL STUDY:

In this section, we conduct several experiments to answer the question: which convolution level is suitable for PAD kernels? Although in static cases RF size for a single layer should keep the same regardless of lower level dilation, this might not be hold for PAD kernels since dilation values are subject to various level of sensitivities due to hierarchical feature representations. To confirm this, we individually extend each convolution block of a vanilla VGG-16 backbone net with PAD kernels and Markov Aggregation, and run experiments following default settings described in Section 4.1. Table 1 summarizes the mIoU for different cases. When only one block is modified, mIoU increases when the feature level for PAD changes from low to high. This matches our expectation that PAD kernels for higher level features perform better than PAD kernels in lower level, as low-level PAD kernels are more sensitive to local variances and tend to focus on capturing information in a smaller region; while high-level kernels are usually related to complicated and abstract concepts, leading them to be more responsive for larger input regions. To further support such a claim, we visualize both RFs and Effective RFs (ERFs) (Luo et al., 2016) for a randomly picked image and put them along with their segmentation results in Figure 3 . As we can see, both RFs and ERFs continuously expand their sizes as feature level for PAD goes higher; meanwhile, visually better segmentation results can be achieved with larger RFs and ERFs. This provides us a supportive example that encourages PAD extension for higher feature level in practice.

Besides, we also test several cases of combining multiple extended blocks into more complicated PAD-Net architectures (the last three lines of Table 1 ). To our surprise, stacking additional PADblocks to single PAD-block may result in inferior performances. We further investigate possible explanations by calculating the variances of dilation sampling for each cases. We find performances always decrease when conv5 is combined with more PAD-blocks, along with notable variance increments. Such increments brought by additional sampling might be the reason for performance downgrading as they make the entire structure more unstable.

4.3. PATTERN AGGREGATION STUDY:

Now we focus on studying the impacts brought by each pattern aggregation strategies described in Section 3.2. As suggested from Section 4.2, we only extend conv5 block of a VGG-16 backbone into PAD kernels to avoid too much dilation sampling. All three (conv5-1, conv5-2 and conv5-3) sub-layers are upgraded with PAD kernels and connected as each aggregation asks. We also include ResNet-101 (He et al., 2016) combined with DeeplabV3+ (Chen et al., 2018) as an additional backbone to see if skip connections may result in different impacts.

The results are concluded in Table 2 . Basically, all three strategies have better results than backbone. However, for both cases Markov Aggregation always yields a better result than other two options, implying too much redundant inter-layer patterns might be accumulated. To further explore the root behind such phenomenon, in Figure 4 , we calculate and visualize the mathematical expectations at each pixel for all three sub-convolution layers. We can see that during the streaming from conv5-1 to conv5-3, PAD-Net with Markov Aggregation is more likely to choose larger dilation everywhere without carrying spatial patterns of input; while both Gated and Recurrent Aggregation are more willing to adjust RF sizes according to spatial structures from input and reserve some spatial clues for dilation sampling. In such cases, information aggregated by lower level features could be too local-sensitive, forcing next layer to put its RF in a smaller region in order to capture such local variations. Thus, Our results for semantic segmentation indicate Markov Aggregation is the best option among the three without overly aggregating inter-layer patterns.

4.4. DILATION BOUNDARY DETERMINATION:

In this section, we aim to figure out whether more dilation options for a PAD kernel can always lead to better performance or not. We setup experiments for comparing mIoUs of a VGG-16 backbone with one, two and three available dilation options for their conv5 blocks, respectively. Based on the discussion in Section 4.3, we only consider the cases with Markov Aggregation to get rid of impacts from multiple inter-layer patterns. Other settings remain default.

Our results are shown in Table 3 , where we gradually increase the available dilation options based on their values from top to bottom and compare the changes of mIoU. Note that the case with the single dilation value 1 is identical to a vanilla backbone net. Apparently, there is a significant performance boost as the number of dilation options is increased from one to two. However, the third dilation option only brings a minor improvement. This suggests that major performance gain is brought by the second one with value 2. We also visualize the output of PAD blocks with a randomly picked input for each case in Figure 5 . When options increased to two and three, There are more neurons being activated than single dilation with similar spatial distributions. This means more dilation options may not further improve the performance, as PAD-Net can intelligently decide the best boundary for dilation values without worrying about overlarge candidates.

4.5. PERFORMANCE BOOSTING FOR BACKBONE ARCHITECTURES:

Finally, we verify PAD-Nets can be easily combined various popular base architectures to further improve their performance. In addition to VGG-16, we also employ another four popular architectures, ResNet-101 (He et al., 2016) , Dilated Reside Nets (DRN) (Yu et al., 2017) , Xception (Chollet, 2017) and MobileNet-v2 (Sandler et al., 2018) , as additional backbone nets. We combined these base structures with FCN (Long et al., 2015) and Deeplabv3+ (Chen et al., 2018) framework and evaluate them on Cityscapes (Cordts et al., 2016) , a more challenging dataset. Detailed architectures and configurations be found in Appendix A (Table 8 ).

We report mIoUs for each backbone net and corresponding PAD-Net in Tables 4 and 5 , respectively, along with other state-of-the-art results for comparison. From these two tables we can see PAD-Nets could always yield better results for every backbone structure on both datasets, exhibiting strong robustness and versatility. We also visualize part of segmentation results in Figure 6 , which coincides with mIoU that PAD-Nets have more correctly labeled pixels and more details preserved. More results on class IoU and segmentation can be found in Appendix B.

5. PAD-NETS FOR IMAGE CLASSIFICATION:

In this section, we demonstrate that the proposed PAD-Nets are not only suitable for dense prediction tasks such as semantic segmentation, but also available for more general applications. More specifically, two fundamental tasks, large-scale and fine-grained image classification will be performed to evaluate the performance of PAD-Nets with several backbone architectures. We show that PAD-Nets can constantly yield better results than their regular counterparts with little extra costs.

5.1. LARGE-SCALE IMAGE CLASSIFICATION:

As an important yet challenging work, large-scale image classification usually requires a CNN model with more layers in order to achieve better performances. Unfortunately, it also makes the model significantly increase its model size. We believe PAD-Nets could properly address such limitations as light-weighted extensions for their base nets, with better performance and similar training efficiency. To prove this, we select four popular CNN architectures, VGG-16, ResNet-50, DRN-C-26 and MobileNet-v2, as backbone nets and run experiments on ILSVRC-2012 dataset (Russakovsky et al., 2015) . Similar to segmentation experiments, we only consider Markov Aggregation in following experiments since our pilot studies indicate it always yields better results. Other configuration details can be found in Appendix A (Table 9 ).

We report both top-1 and top-5 classification accuracies for every pair of vanilla and PAD-Net in Table 6 , along with the comparison of their model complexity changes. Considering millions of parameters that backbone models contain, several thousand extra weights introduced by PAD kernels are trivial burdens regarding to total model complexity. Meanwhile, we can observe around 1% improvement of top-1 accuracies for each PAD-Net and slight top-5 accuracy improvements for most cases, suggesting new modules with less than 0.1% size overhead bring 10 times of performance boosting. This provides us a strong evidence to demonstrate the efficiency of PAD-Nets for largescale classification problem. In addition, training curve comparison can be found in Appendix C.

Under review as a conference paper ICLR 2020 

5.2. FINE-GRAINED IMAGE CLASSIFICATION:

Unlike general classification problem, fine-grained task puts a special emphasis on mining subtle discriminative information in order to recognize objects from different sub-categories. In this section we empirically demonstrate the proposed PAD-Nets could properly handle such challenges via their dynamically dilated kernels. We use all backbones from Section 5.1 except for VGG-16 due to its extremely huge size and initialize corresponding PAD-Nets with their pretrained weights. Experiments are conducted on Stanford Cars (Krause et al., 2013) and FGVC-Aircraft (Maji et al., 2013) datasets following their default protocol with two input sizes, 224 and 448. Similarly, we put other configuration details in Appendix A (Table 10 ).

All of our experimental results are summarized in Table 7 , where we compare the top-1 accuracy for each pair of PAD-Net and its vanilla equivalent. We can only observe trivial improvements with input size 224 for both datasets. However, performance gain increases to over 2% for nearly every type of PAD-Net, indicating PAD-Nets could be a much better option for high-resolution images to get higher performance without more significant resource burdens. Moreover, by comparing the activation maps of DRN and PAD-DRN, we further discover that PAD-Nets can better capture parts information and preserve more details than baselines with fixed, predefined dilation values. We provide such evidence with more visualized activation maps for DRN-C-26 and its PAD-extension in Appendix B.

6. CONCLUSION:

In this paper we formulate the dilation as a learnable weight for convolution kernels such that its value can be dynamically decided during the running time. This leads to PAD-Nets, a light-weighted, end-to-end trainable framework that allows their kernels to adjust pixel-wise RFs in a data-driven manner. To infer proper dilation values based on feature hierarchy, we model inter-layer patterns via several sequential aggregation strategies. Our studies on semantic segmentation explore various properties of PAD-Nets and indicate better performance can be achieved when PAD kernels are with higher feature levels and Markov Aggregation. We also demonstrate PAD-Nets can consistently boost performances over several popular backbone architectures, and be a valuable option for more general visual tasks such as large-scale and fine-grained image classifications.

A ARCHITECTURE AND CONFIGURATIONS:

With respect to segmentation tasks, five deep neural networks, as backbones, are modified by incorporating PAD layers. Specific architecture and configuration information is listed in Table 8 . For classification tasks, four deep models, , are modified to PAD-Nets and their architectures and configurations on corresponding PAD-Nets are listed by Table 9 . Concerning fine-grained tasks, three deep models, including ResNet-50, DRN-C-26, and MobileNet-V2, are modified to PAD-Nets, and specific architecture and configuration information is listed in Table 10 . All PAD-Nets are trained by following Markov Aggregation. The same as the above.

MobileNetV2 The Layer-2 with t=6, c=32, n=3, and s=2 is modified with PAD units.The dilation variables are learnt based on 1, 2, and 3.

The same as the above. 

C TRAINING CURVE FOR LARGE-SCALE IMAGE CLASSIFICATION:

The training curve for large-scale image classification is shown in Figure 10 . We can see PAD-Nets have similar and even better convergence rates comparing to their conventional counterparts. 

:



Figures and Tables:

Figure 1: Comparison of regular and pixel-wise adaptive dilation. Colors stands for distinct dilation.

Figure 3: The top row indicates the input image and its visualized RFs and ERFs for PAD-VGG16 with different conv blocks. Patches means RFs and red dots inside are ERFs. The bottom row shows the ground truth and corresponding segemtation results. GT stands for groundtruth.

Figure 4: Mathematical expectation of dilation sampling at each pixel for individual sub-layers (from left to right: conv5-1 to conv5-3). Brighter color means higher dilation and vise versa. The input is the same as the one in Figure 1.

Figure 5: Activation maps for PAD-Nets with different number of dilation options.

Figure 6: Semantic segmentation results on Cityscapes dataset.

Figure 7: Semantic segmentation results on Pascal VOC 2012.

Figure 8: Semantic segmentation results from ResNet-101 and PAD-ResNet-101 on Cityscapes.

Figure 9: Semantic segmentation results from Xception and PAD-Xception on Cityscapes.

Figure 10: The training curve of large-scale image classification using PAD-Nets based on the VGG-16 and ResNet-50 .

mIoU for pattern aggregation study. VGG-16 backbone is combined with FCN-8s and ResNet-101 is with Deeplab v3+.

Table content: <table><tr><td colspan="3">AGGREGATION VGG-16 RESNET-101</td></tr><tr><td>MARKOV</td><td>66.5</td><td>77.2</td></tr><tr><td>GATED</td><td>65.5</td><td>76.7</td></tr><tr><td>RECURRENT</td><td>65.3</td><td>75.6</td></tr><tr><td>BACKBONE</td><td>64.7</td><td>75.1</td></tr></table>

mIoU of cases with available dilation options change.

Table content: <table/>

mIoU for VOC 2012 validation set.

Table content: <table><tr><td>METHOD</td><td>MIOU REGULAR</td><td>PAD</td></tr><tr><td>DPN (LIU ET AL., 2015)</td><td>67.8</td><td>-</td></tr><tr><td>CRF+RNN (ZHENG ET AL., 2015)</td><td>69.6</td><td>-</td></tr><tr><td>DCNN (DAI ET AL., 2017)</td><td>75.1</td><td>-</td></tr><tr><td>VGG-16+FCN-32S</td><td>62.8</td><td>65.1</td></tr><tr><td>VGG-16+FCN-8S</td><td>64.7</td><td>66.5</td></tr><tr><td>RESNET-101+DEEPLABV3+</td><td>75.1</td><td>77.2</td></tr><tr><td>XCEPTION+DEEPLABV3+</td><td>73.5</td><td>74.4</td></tr><tr><td>DRN-D-54+DEEPLABV3+</td><td>75.4</td><td>77.2</td></tr></table>

mIoU for CityScape validation set.

Table content: <table><tr><td>METHOD</td><td>MIOU REGULAR</td><td>PAD</td></tr><tr><td>DANET-101 (FU ET AL., 2019)</td><td>77.6</td><td>-</td></tr><tr><td>PSP-NET (ZHAO ET AL., 2017)</td><td>78.8</td><td>-</td></tr><tr><td>IN-ABN (ROTA BUL Ò ET AL., 2018) +RESNEXT-101 (XIE ET AL., 2017)</td><td>79.2</td><td>-</td></tr><tr><td>PSP-NET + GFF (LI ET AL., 2019)</td><td>80.4</td><td>-</td></tr><tr><td>GSCNN (TAKIKAWA ET AL., 2019)</td><td>80.8</td><td>-</td></tr><tr><td>MOBILENETV2+DEEPLABV3+</td><td>70.3</td><td>71.5</td></tr><tr><td>XCEPTION+DEEPLABV3+</td><td>77.5</td><td>79.0</td></tr><tr><td>RESNET-101+DEEPLABV3+</td><td>80.1</td><td>80.7</td></tr></table>

Accuracies for lage-scale image classification on ILSVRC 2012 and corresponding model complexities. p# means model size and ∆p# is the number of weights introduced by PAD-Nets; (∆p#)/(p#) is the percentage of model size that PAD-Nets have increased.

Table content: <table><tr><td>METRIC</td><td/><td colspan="2">ACCURACY (%)</td><td/><td colspan="3">MODEL COMPLEXITY</td></tr><tr><td>BACKBONE</td><td colspan="4">TOP@1 REGULAR PAD REGULAR PAD TOP@5</td><td>P#</td><td>∆P#</td><td>(∆P#)/(P#) (%)</td></tr><tr><td>VGG-16</td><td>73.0</td><td>74.5</td><td>91.2</td><td>92.0</td><td>138M</td><td>21K</td><td>0.016</td></tr><tr><td>RESNET-50</td><td>76.0</td><td>76.9</td><td>93.0</td><td colspan="2">93.4 25.5M</td><td>1K</td><td>0.004</td></tr><tr><td>DRN-C-26</td><td>75.1</td><td>75.9</td><td>92.4</td><td colspan="2">92.6 21.1M</td><td>4K</td><td>0.019</td></tr><tr><td>MOBILENETV2</td><td>71.8</td><td>72.6</td><td>91.0</td><td>90.8</td><td>3.5M</td><td>2.7K</td><td>0.078</td></tr></table>

Top-1 Accuracy for Fine-Grained Visual Classification on different databases.

Table content: <table><tr><td>TASK</td><td/><td/><td colspan="4">FINE-GRAINED CLASSIFICATION</td><td/><td/></tr><tr><td>DATASET</td><td/><td colspan="2">STANFORD CARS</td><td/><td/><td colspan="2">FGVC-AIRCRAFTS</td><td/></tr><tr><td>CROP SIZE</td><td colspan="8">224 REGULAR PAD REGULAR PAD REGULAR PAD REGULAR PAD 448 224 448</td></tr><tr><td>RESNET-50</td><td>91.2</td><td>91.6</td><td>92.3</td><td>93.5</td><td>86.1</td><td>86.6</td><td>87.9</td><td>90.1</td></tr><tr><td>DRN-C-26</td><td>91.0</td><td>91.4</td><td>90.3</td><td>92.4</td><td>86.3</td><td>86.5</td><td>86.8</td><td>89.6</td></tr><tr><td>MOBILENETV2</td><td>88.7</td><td>88.8</td><td>80.6</td><td>82.7</td><td>83.2</td><td>84.1</td><td>80.5</td><td>87.6</td></tr></table>

PAD-Nets on semantic segmentation tasks. (Section 4.5)

Table content: <table><tr><td>PAD-Nets</td><td>Architectures Description</td><td>Configurations</td></tr><tr><td>FCN8s/FCN32s + VGG-16</td><td>Conv-5 is modified to PAD</td><td>Follow FCN8s/FCN32s</td></tr><tr><td/><td>layers. Dilation variables are</td><td>training scheme with no</td></tr><tr><td/><td>learnt with 1, 2, and 4.</td><td>augmentation or image</td></tr><tr><td/><td/><td>pre-processing or</td></tr><tr><td/><td/><td>post-processing. FCN8s is</td></tr><tr><td/><td/><td>trained in at once mode.</td></tr><tr><td>ResNet-101 + DeeplabV3+</td><td>The 3x3 convolution layers in</td><td>Follow deeplab VOC-2012</td></tr><tr><td/><td>Layer-4 block are modified to</td><td>training scheme with batch</td></tr><tr><td/><td>PAD layers. Dilation</td><td>size of 8 and outstride of 16.</td></tr><tr><td/><td>variables are learnt with 1, 2,</td><td/></tr><tr><td/><td>and 4.</td><td/></tr><tr><td>DRN-D-54 + DeeplabV3+</td><td>Layer-7 and Layer-8 are</td><td>Follow deeplab VOC-2012</td></tr><tr><td/><td>modified to PAD layers.</td><td>training scheme with batch</td></tr><tr><td/><td>Dilation variables are learnt</td><td>size of 8 and outstride of 16.</td></tr><tr><td/><td>with 1, 2, and 4.</td><td/></tr><tr><td>Xception + DeeplabV3+</td><td>The 3x3 convolution layers</td><td>Follow deeplab VOC-2012</td></tr><tr><td/><td>are modified in Middle-flow</td><td>training scheme with batch</td></tr><tr><td/><td>and Exit-flow where strides</td><td>size of 8, outstride of 16, no</td></tr><tr><td/><td>are 1.</td><td>BN fine-tuning, and</td></tr><tr><td/><td/><td>multi-scale testing.</td></tr><tr><td>MobileNetV2 + DeeplabV3+</td><td>The 3x3 convolution layers</td><td>Follow deeplab VOC-2012</td></tr><tr><td/><td>with stride 1 in depth-wise</td><td>training scheme with batch</td></tr><tr><td/><td>modules with output channels</td><td>size of 8, outstride of 16, no</td></tr><tr><td/><td>of 160 and 320 are modified</td><td>BN fine-tuning, and</td></tr><tr><td/><td>to PAD layers.</td><td>multi-scale testing.</td></tr></table>

PAD-Nets on large-scale image classification tasks. (Section 5.1)

Table content: <table><tr><td>PAD-Nets</td><td>Architectures Description</td><td>Configurations</td></tr><tr><td>VGG-16</td><td>Only Conv-5 is incorporated with</td><td>Follow Pytorch ImageNet training</td></tr><tr><td/><td>PAD units. The dilation variables are</td><td>default settings with 128 batch size</td></tr><tr><td/><td>learnt based on 1, 2, and 4.</td><td>and 120 epochs.</td></tr><tr><td>ResNet-50</td><td>The last three convolution layers in</td><td>Follow DRN ImageNet training</td></tr><tr><td/><td>Layer-2 block are modified with PAD</td><td>scheme with 192 batch size and 120</td></tr><tr><td/><td>units. The dilation variable are learnt</td><td>epochs.</td></tr><tr><td/><td>based on 1, 2, and 4.</td><td/></tr><tr><td>DRN-C-26</td><td>Layer-8 is modified with PAD units.</td><td>Follow original DRN reported</td></tr><tr><td/><td>The dilation variables are learnt based</td><td>training scheme with 192 batch size</td></tr><tr><td/><td>on 1, 2, and 4.</td><td>and 120 epochs.</td></tr><tr><td colspan="2">MobileNetV2 The Layer-2 with t=6, c=32, n=3, and</td><td>Follow the setting of 1.0-224 reported</td></tr><tr><td/><td>s=2 is modified with PAD units. The</td><td>in Sandler et al. (2018) with 256</td></tr><tr><td/><td>dilation variables are learnt based on</td><td>batch size and 300 epochs.</td></tr><tr><td/><td>1, 2, and 4.</td><td/></tr></table>

PAD-Nets on fine-grained image classification tasks. (Section 5.2)

Table content: <table><tr><td>PAD-Nets</td><td>Architectures Description</td><td>Configurations</td></tr><tr><td>ResNet-50</td><td>The last three convolution layers in</td><td>The setting follows that of DLA</td></tr><tr><td/><td>Layer-2 block are modified with PAD</td><td>reported in Yu et al. (2018) with two</td></tr><tr><td/><td>units. The dilation variable are learnt</td><td>crop sizes, i.e., 224 and 448.</td></tr><tr><td/><td>based on 1, 2, and 3.</td><td/></tr><tr><td>DRN-C-26</td><td>Layer-7 and Layer-8 are modified</td><td/></tr><tr><td/><td>with PAD units. The dilation</td><td/></tr><tr><td/><td>variables are learnt based on 1, 2, and</td><td/></tr><tr><td/><td>3.</td><td/></tr></table>

PAD-FCN8s IoUs on VOC-2012 across all classesMore quality results on semantic segmentation tasks are shown in Figure7, Figure8, and Figure9. More class IoUs are included in Table11, Table 12, Table 13, and Table 14. For fine-grained classification tasks, detailed feature maps for fine-grained image classification are shown in Figure11.

Table content: <table><tr><td/><td colspan="3">background aeroplane bicycle</td><td>bird</td><td>boat</td><td>bottle</td><td>bus</td><td>car</td><td>cat</td><td>chair</td><td>cow</td></tr><tr><td>PAD-FCN8s</td><td>0.914</td><td>0.833</td><td>0.388</td><td>0.751</td><td>0.627</td><td>0.740</td><td colspan="3">0.802 0.744 0.805</td><td>0.252</td><td>0.805</td></tr><tr><td>FCN8s</td><td>0.908</td><td>0.798</td><td>0.363</td><td>0.776</td><td>0.581</td><td>0.742</td><td colspan="3">0.775 0.749 0.799</td><td>0.292</td><td>0.712</td></tr><tr><td/><td>dining table</td><td>dog</td><td>horse</td><td colspan="5">motorbike person potted plant sheep sofa</td><td colspan="2">train tv/monitor mIoU</td></tr><tr><td>PAD-FCN8s</td><td>0.474</td><td>0.724</td><td>0.729</td><td>0.783</td><td>0.791</td><td>0.510</td><td colspan="3">0.729 0.370 0.773</td><td>0.600</td><td>0.665</td></tr><tr><td>FCN8s</td><td>0.375</td><td>0.684</td><td>0.673</td><td>0.765</td><td>0.780</td><td>0.490</td><td colspan="3">0.760 0.344 0.789</td><td>0.572</td><td>0.647</td></tr><tr><td colspan="11">B MORE RESULTS ON SEMANTIC SEGMENTATION AND FINE-GRAINED</td></tr><tr><td colspan="4">IMAGE CLASSIFICATION</td><td/><td/><td/><td/><td/><td/></tr></table>

PAD-ResNet-101 IoUs on VOC-2012 across all classes

Table content: <table><tr><td/><td colspan="3">background aeroplane bicycle</td><td>bird</td><td>boat</td><td>bottle</td><td>bus</td><td>car</td><td>cat</td><td>chair</td><td>cow</td></tr><tr><td>PAD-ResNet-101</td><td>0.932</td><td>0.838</td><td>0.393</td><td>0.848</td><td>0.622</td><td>0.756</td><td colspan="3">0.908 0.848 0.918</td><td>0.373</td><td>0.874</td></tr><tr><td>ResNet-101</td><td>0.922</td><td>0.770</td><td>0.388</td><td>0.853</td><td>0.626</td><td>0.698</td><td colspan="3">0.913 0.836 0.886</td><td>0.225</td><td>0.835</td></tr><tr><td/><td>dining table</td><td>dog</td><td>horse</td><td colspan="5">motorbike person potted plant sheep sofa</td><td colspan="2">train tv/monitor mIoU</td></tr><tr><td>PAD-ResNet-101</td><td>0.584</td><td>0.879</td><td>0.851</td><td>0.805</td><td>0.833</td><td>0.554</td><td colspan="3">0.852 0.534 0.835</td><td>0.648</td><td>0.772</td></tr><tr><td>ResNet-101</td><td>0.568</td><td>0.862</td><td>0.791</td><td>0.810</td><td>0.815</td><td>0.452</td><td colspan="3">0.764 0.461 0.824</td><td>0.691</td><td>0.751</td></tr></table>

PAD-DRN-54 IoUs on VOC-2012 across all classes

Table content: <table><tr><td/><td colspan="3">background aeroplane bicycle</td><td>bird</td><td>boat</td><td>bottle</td><td>bus</td><td>car</td><td>cat</td><td>chair</td><td>cow</td></tr><tr><td>PAD-DRN-54-D</td><td>0.927</td><td>0.823</td><td>0.384</td><td>0.845</td><td>0.668</td><td>0.729</td><td colspan="3">0.915 0.838 0.852</td><td>0.294</td><td>0.876</td></tr><tr><td>DRN-54-D</td><td>0.921</td><td>0.799</td><td>0.345</td><td>0.846</td><td>0.660</td><td>0.723</td><td colspan="3">0.868 0.848 0.884</td><td>0.313</td><td>0.820</td></tr><tr><td/><td>dining table</td><td>dog</td><td>horse</td><td colspan="5">motorbike person potted plant sheep sofa</td><td colspan="2">train tv/monitor mIoU</td></tr><tr><td>PAD-DRN-54-D</td><td>0.568</td><td>0.839</td><td>0.836</td><td>0.814</td><td>0.813</td><td>0.491</td><td colspan="3">0.805 0.434 0.781</td><td>0.693</td><td>0.772</td></tr><tr><td>DRN-54-D</td><td>0.528</td><td>0.840</td><td>0.801</td><td>0.805</td><td>0.800</td><td>0.475</td><td colspan="3">0.739 0.492 0.750</td><td>0.675</td><td>0.754</td></tr></table>

PAD-ResNet-101 IoUs on Cityscapes across all classes

Table content: <table><tr><td/><td colspan="4">road sidewalk building wall</td><td>fence</td><td>pole</td><td>light</td><td>sign</td><td>vegetation terrain</td></tr><tr><td colspan="2">PAD-ResNet-101 0.984</td><td>0.867</td><td>0.934</td><td colspan="4">0.610 0.654 0.668 0.737</td><td>0.817</td><td>0.930</td><td>0.653</td></tr><tr><td>ResNet-101</td><td>0.983</td><td>0.860</td><td>0.931</td><td colspan="4">0.625 0.638 0.648 0.726</td><td>0.801</td><td>0.929</td><td>0.659</td></tr><tr><td/><td>sky</td><td>person</td><td>rider</td><td>car</td><td>truck</td><td>bus</td><td colspan="2">train motorcycle</td><td>bicycle</td><td>mIoU</td></tr><tr><td colspan="2">PAD-ResNet-101 0.954</td><td>0.840</td><td>0.674</td><td colspan="4">0.956 0.810 0.919 0.808</td><td>0.722</td><td>0.796</td><td>0.807</td></tr><tr><td>ResNet-101</td><td>0.953</td><td>0.833</td><td>0.658</td><td colspan="4">0.953 0.797 0.912 0.815</td><td>0.720</td><td>0.787</td><td>0.801</td></tr></table>

Peer review (iclr2020):
This paper extends work on dilated convolutional nets to Pixelwise Adaptive Dilated Nets (PAD-Nets) by allowing for spatially varying, pixel and feature dependent dilations. Dilation as a function of input at individual pixels is achieved through sampling probability distributions that are conditioned by pixel-wise input features (Gumbel-Softmax is used to approximate the sampling process in order to ensure end-to end differentiability). The paper makes the assumption that dilation values are related to inter-layer patterns between convolution layers due to their hierarchical nature and the RF size at each location is adjusted based on information flows between corresponding inter-layer pixels during forward propagation. PROS: Addresses adaptation of local context and feature dependent modification of receptive field size, Experiments are shown to improve semantic segmentation performance in a number of datasets. Cons: Incremental evolution of existing ideas, Too much claim (especially on discussion section) on how the design leads to more interpretability. The claims are still largely empirical without theoretical grounding. Only a few images are shown illustrating performance. The paper should also show examples where there are fundamental limitations. Thus, one does not see what challenges (if any) remain. If the authors address these above concerns in the rebuttal, I will be inclined to reexamine the rating.