Shape-Tailored Deep Neural Networks Using PDEs for Segmentation
Authors: Naeemullah Khan
Angira Sharma
Philip Torr
Ganesh Sundaramoorthi
Venue:  (2021)
Keywords: robustness
covariance
invariance
convolutional neural nets
PDEs
segmentation

Abstract: We present Shape-Tailored Deep Neural Networks (ST-DNN). ST-DNN extend convolutional networks, which aggregate data from fixed shape (square) neighbor-hoods to compute descriptors, to be defined on arbitrarily shaped regions. This is useful for segmentation applications, where it is desired to have descriptors that aggregate data only within regions of segmentation to avoid mixing data from different regions, otherwise, the descriptors are difficult to group to a unique region.  We formulate these descriptors through partial differential equations (PDE) that naturally generalize convolution to arbitrary regions, and derive the methodology to jointly estimate the segmentation and ST-DNN descriptor. We also show that ST-DNN inherit covariance to translations and rotations from the PDE, a natural property of a segmentation method, which existing CNN based methods lack. ST-DNN are 3-4 order of magnitude smaller than typical CNN. We empirically show that they exceed segmentation performance compared to state-of-the-art CNN-based descriptors using 2-3 orders smaller training sets on the texture segmentation problem.

Under review as a conference paper at ICLR 2021
SHAPE-TAILORED DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We present Shape-Tailores Deep Neural Networks (ST-DNN). ST-DNN extend convolutional networks (CNN), which aggregate data from fixed shape (square) neighborhoods, to compute descriptors defined on arbitrarily shaped regions. This is natural for segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape. We formulate these descriptors through the Poisson partial differential equation (PDE), which can be used to generalize convolution to arbitrary regions, We stack multiple PDE layers to generalize a deep CNN arbitrary regions, and apply it to segmentation. We show that ST-DNN are covariant to translations and rotations and robust to domain deformations, natural for segmentation, which existing CNN based methods lack. ST-DNN are 3-4 orders of magnitude smaller then CNNs used for segmentation. We show that they exceed segmentation performance compared to state-of-the-arr CNN -ased descriptors using 2-3 orders smaller training sets on the texture segmentation problem.
INTRODUCTION
Convolutional neural networks (CNNs) have been used extensively for segmentation problems in computer vision He et al. (017)) : He et al. (2016); Chen et al.] 2017]; Xie & Tu (2015). CNNs provide framework for learning descriptors that able discriminate different textured or semantic regions within images. Much progress has been made in segmentation with CNNs but results are still far from human performance. Also, significant engineering must be performed to adapt CNNs to segmentation problems. A basic component in the architecture for segmentation problems involves labeling or grouping dense descriptors returned by a backbone CNN. A difficulty in grouping these descriptors arises, especially near the boundaries of segmentation regions, as CNN descriptors aggregate data from fixed shape (square neighborhoods) each pixel and may thus aggregate data from different regions. This makes grouping these descriptors into a unique region difficult, which often results in errors in the grouping.
In segmentation problems (e.g., semantic segmentation), current methods attempt to mitigate these errors by adding post-processing layers that aim to group simultaneously the (coarse-scale) descriptors from the CNN backbone and the fine-level pixel data. However, the errors introduced might not always be fixed. A more natural approach to avoid this problem is to consider the coarse and fine structure together, avoiding aggregation across boundaries, prevent errors at the outset.
To avoid such errors, one could design descriptors that aggregate data only within boundaries. To this end, Khan al. (2015) introduced "shape-tailored" descriptors that aggregate data within region of interest, and used these descriptors for segmentation. However, these descriptors are hand-crafted and do not perform on-par with learned approaches. Khan & Sundaramoorthi (2018) introduced learned shape-tailored descriptors by learning a neural network operating on the input channel dimension of input hand-crafted shape-tailored descriptors for segmentation. However, these networks, though deep in the channel dimension, did not filter data spatially within layers. Since an advantage of CNNs comes from exploiting spatial filtering at each depth of the network, this work, we design shape-tailored networks that are deep and perform shape-tailored filtering in space at each layer using solutions of the Poisson PDE. This results in shape-tailored networks that provide more discriminative descriptors than a single shape-tailored kernel. This extension requires development of techniques to back- bock-propagate through PDEs, which we derive in this work.
Our contributions are specifically:
1. We construct and show how to train ST-DNN, deep networks that perform shape-tailored spatial filtering via the Poisson PDE at each depth so as to generalize a CNN to arbitrarily shaped regions.
2. We show analytically and empirically that ST-DNNs are covariant to translations and rotations as they inherit this property from the Poisson PDE. In segmentation, covariance ka.k.a., equivariance)

---

Under review as a conference paper at ICLR 2021
to translation and rotation is desired property: if segment in an image is found, then the corresponding segment should be found in the translated rotated image (or object). This property is not generally present with existing CNN-based segmentation methods even when trained with augmented translated and rotated images Azulay & Weiss 2019), and requires special consideration.
3. We show analytically and empirically that ST-DNNs are robust to domain deformations. These result from viewpoint change or object articulation, and so they should not affect the descriptor.
4. To demonstrate ST-DNN and the properties above, validate them the task of segmentation, an important problem in low-level vision Malik & Perona [990); Arbelaez et al. (2011b).
Because of properties of the PDE, ST-DNN also have desirable generalization properties. This is because: The robustness and covariance properties are built into our descriptors and do not need to be learned from data, b) The PDE solutions, generalizations Gabor-like filters Olshausen & Field (1996); Zador 2019). have natural image structure inherent in their solutions and so this does not need to be learned from data, and c) Our networks have fewer parameters compared to existing networks in segmentation. This is because the PDE solutions form basis and only lincar combinations of a few basis elements are needed to learn discriminative descriptors for segmentation. In contrast, CNNs spend lot of parameters to learn this structure.
1.1 RELATED WORK
Traditional approaches to segmentation rely on hand-crafted features, e.g., through a filter bank Haralick & Shapiro (1985). These features are ambiguous near the boundaries of objects. In Khan et al. (2015) hand-crafted descriptors that aggregate data within object boundaries are constructed to avoid this, but lack sufficient capacity to capture the diversity of textures or be invariant to nuisances. Deep-learning based approaches have showed state-of-the-art results in edge-based methods Xie & Tu (2017); He et al. 2019);; Deng et al. (2018). Watershed is applied on edge-maps to obtain the segmentation. The main drawback of these methods is it is often difficult to form segmentations due to extraneous or faint edges, particularly when "textons" in textures are large.
CNNs have been applied to compute descriptors for semantic segmentation, where pixels in an image classified into certain semantic object classes Li et al. (2019); |Huang et al. (2019); Du al. (011); Pang et al. (2019); Zhu et al. (2019); Liu et al. (2019). Usually these classes are limited to few object classes and do not tackle general textures, where the number of classes may be far greater, and thus such approaches are not directly applicable to texture segmentation. But semantic segmentation approaches may eventually benefit from our methodology as descriptors aggregating data only within objects or regions are also relevant to these problems. A learned shape-tailored descriptor Khan Sundaramoorthi (2018) is constructed with a Siamese network on hand-crafted shape-tailored descriptors. However, Khan & Sundaramoorthi (2018) only does shape-tailored filtering in pre-processing as layering these requires new methods to train. We further examine covariance and robustness, not examined in Khan & Sundaramoorthi (2018).
Covariance to rotation CNNs has been examined in recent works, e.g., Weiler et al. (018);; Yin et al. (2019); Anderson et al. (2019). They, however, not shape-tailored so do not aggregate data only within shaped regions. Lack of robustness to deformation (and translation) in CNNs is
examined in Azulay & Weiss (2019) and theoretically in Bietti & Mairal (2017). [Sifre & Mallat (2013) constructs deformation robust descriptors inspired by CNNs, but are hand-crafted.
2 CONSTRUCTION OF SHAPE-TAILORED DNN AND PROPERTIES
In this section, we design a deep neural network that outputs descriptors at each pixel within an arbitrary shaped region of interest and aggregates data only from within the region. We want the descriptors to be discriminative of different texture, yet robust to nuisances within the region (e.g.. local photometric and geometric variability) to be useful for segmentation. Our construction uses Poisson PDE, which naturally smooths data only within a region of interest. Smoothing naturally yields robustness to geometric nuisances (domain deformations). By taking linear combinations of derivatives of the output of the PDE, we can approximate the effect of general convolutional kemnels but avoid mixing data across the boundary of region of interest. ST-DNN is also covariant to
2

---

Under review as a conference paper at ICLR 2021
translations and rotations, inheriting it from the Poisson equation, which leads to the segmentation algorithm being covariant to such transformations.
2.1 -AILLRED DNN DESCRIPTORS THROUGH POISSON PDE
Shape-tailored Smoothing via Poisson PDE: To construct a shape-tailored deep network, we first smooth the input to a layer using the Poisson PDE so as to aggregate data only within the region of interest, similar what done in Khan et al. 2015) for just the first layer. Let R C Q C R be the region of interest, where Q is the domain of the mppt image I S2 - I and k is the number of input channels to the layer. Let u R + TR (M is the number of output channels) be the result of the smoothing; the components u of u solve the PDE within Rt
((x) (r) = I(x) Vu(x) N 0
x E R E àR
(1)
where / is a channel of I. 0R is the boundary of R. N is normal to OR, a is the scale of smoothing and A/V the Laplacian and the gradient respectively. It can be shown that the smoothing be written in the form u(x)) -J (xx, (y) dy where K(...) is the Green's function of the PDE, a smoothing kernel, which further shows that the PDE aggregates data only within R.
We can now generalize the operation of convolution tailored to the region of interest by taking linear combinations of partial derivatives of the output of the PDE equation This is motivated by the fact that in Rt = IR2, linear combinations of derivatives of Gaussians can approximate any kernel arbitrarily well. Gaussian filters are the solution of the heat equation, and the PDE equation [ relates to the heat equation, i.e., equation i the steady state solution of a heat equation. Thus, linear combinations of derivatives of equation I generalize convolution to an arbitrary shape Rt; in experiments, few first order directional derivatives are sufficient for our segmentation tasks (see Section for details). A layer of the ST-DNN takes such linear combinations and rectifies it as follows:
Shape-tailored Deep Network:
[a))) = o L o (T),
where / R + IR* is the input to the layer, T' is an operator that outputs derivatives of the solution of the Poisson PDE equation [ Lyyy) way + b, is point-wise linear function (i.e., a x 1 convolution applied to combine different channels), r is the rectified linear function, and indexes the layer of the network. Notice that since r and L, are pointwise operations, they preserve the property of T that aggregates data only within the region R. We now compose layers to construct ST-DNN as follows:
(z)]()) 8 Jm fi--1 fm-2 0 fo (xx)
(3)
where F[]((x) is the output of the ST-DNN, fo. fm input image, and represents the soft-max operation (to bound the output values).
fm are the m + layers of the network, / is the
ST-DNN does not have a pooling layer because the PDE already aggregates data from a neighborhood by smoothing; further, the lack of reduction in spatial dimension allows for more accurate shape estimation in our subsequent segmentation, and avoids the need for up-sampling layers. We will show that we can retain efficiency in training and inference.
2.2 COVARIANCE AND ROBUSTNESS OF ST-DNN
In addition to ST-DNN generalizing CNNs to arbitrary shaped regions, the ST-DNN is also covariant in-plane translation and rotation, and robustness to domain deformations due to properties of the Poisson PDF. This means covariance also extends to our segmentation method, which is important as any object segmented in an image will also be segmented if the camera undergoes these trans- formations. Robustness to deformations is important as this means that small geometric variability (e.g., shape variations in textons, small viewpoint change, object deformation) will not affect the descriptors and the segmentation much. We make these properties more precise, and give intuition for proofs, leaving details to Appendix B
A covariant operator commutes with set of transformations:
Definition An operator S I + I (from the set of images I to itself) is covariant to a class W of transformations if S w|] [ST] o for every I E I and w E W.
3

---

Under review as a conference paper at ICLR 2021
ST-DNN is covariant W, the set of in-plane rotations and translations:
Theorem The ST-DNN equation 3 covariant to the set of translations and rotations, i.e., & + Rx + T where R
2 rolation matrix and T E IR2.
This follows from the covariance of the Laplacian and point-wise operations (rectification, 1 convolution), and lack of sub-sampling.
We now make precise the robustness of the ST-DNN to domain deformations:
Theorem The ST-DNN equation 3 is insensitive to deformations, i.e.,
[[ o w| F[]] Cl|w id]l
where w Ω2 + Ω is a domain deformation, id is the identity map, H' is the Sobolev norm (measures both the amount and smoothness of the deformation), and C is constant independent of w.
Intuitively, this follows from the fact that the Poisson PDE locally averages input data, and local averages are robust to translation and hence deformations, which are locally translations.
TRAINING OF THE NETWORK AND BACK-PROPAGATION
In this section, we describe the training of ST-DNN by introducing a loss function, how the weights can be learned, and the implementation. As the network layers solutions to PDEs, one needs to differentiate through such layers, which we describe.
3.1 Loss FUNCTION FOR TRAINING
Given the ST-DNN of Section 2.1 the loss function train such descriptors from ground truth segmentation masks (motivated by consistency to the segmentation algorithm in Section 4 that is based on classical energies Chan Vese (2001); Yezzi Jr al. (1999) from computer vision) is defined as:
L(W)= Σ
R [w(())
ai ai1/dr
ΣΣ jti
la: aail
where i,j E {1, 2, N} are the indices for the regions the ground truth segmentation, F Fw (a) the output of the ST-DNN, W are the weights of the network (i.e., weights on derivatives of the Poisson PDE solution), R& is the area of region i, and a; is the average descriptor within the 7tth region, i.e., a [パ.. Jn. Fw (2)dd.. The loss function is comprised of two terms. The first component of the loss is minimized when the learned descriptor is constant within regions R; so that each region consists of parts of the image with uniform descriptor. The second term forces the learned descriptor of different regions to be different to discriminate different textures.
3.2 COMPUTING GRADIENTS OF THE LOSS AND TRAINING
Computing gradients of the loss function for training requires consideration as it involves differen- tiating through PDEs. The most straightforward way to do this involves discretizing the PDE, so the solution linear matrix system packages to perform back
do below. This allows the of existing deep learning back-propagatie propagation by storing the matrix in memory. However, this can lead
large memory consumption as the matrix can be large and is only feasible for small images. Fortunately, our PDEs involve scale parameter a we can train using down-sampled images, facilitating the use of existing packages back-prop, and then infer on native resolution images by simply scaling a by the down-sampling factor, which we do in experiments.
The more accurate method, though more difficult to implement, is to avoid storing the matrix and instead compute the PDE solution by iterative numerical PDE method that does not require storage of matrices. This requires formulating variant of the back bockkpropagation algorithm, that is similar but involves forward propagation of layer derivatives with respect to the weights through the PDE solutions. This is unfortunately not available in standard deep learning packages. For completeness, provide the mathematical formulation of this approach in Appendix C which
4

---

Under review as a conference paper at ICLR 2021
Inticaliationn
Upiaator
Laver
Figure 1: ST-DNN computation and joint segmentation. The input the network is the image and an initial segmentation mask. ST-DNN dense descriptors are computed for each region of the mask using equation 3 The segmentation updates by taking few steps in the gradient direction of equation g The process is iterated with the updated segmentation until the regions converge.
we have performed experiments for few layer cases. This did not give an appreciable performance increase given the complexity of implementation, but could be useful in other applications.
Implementation: Using the first method above, we discretize equation
as:
u(i,)) B.EA((())RR
E
(u(k,1) I(i.)) for (i,))
(6)
where ji, represents the 4-pixel neighborhood of pixel (i.j), which represents the 2hh row and jth column, and intersection means that only neighbors in the region considered as implied by the Neumann boundary condition in the PDE, avoiding aggregation outside R. The discretization approximately preserves the rotation covariance, and any errors vanish with increasing resolution. Note that more accurate discretizations exist, but our experiments demonstrate the sufficiency of this scheme. We can vectorize and I and write equation 6 as:
ARu = II and AR 'I
The above linear transformation from I u. The size of A (mn X mn), where m and are the number of rows and columns in T. With the PDE layers defined through this matrix multiplication, we can use the usual back-pro pookkpropagation method to compute derivatives with respect to weights (sce Appendix for more details). In experiments, we downsampled images to 32 32 for training.
APPLICATION TO SEGMENTATION
In this section, we describe the procedure for segmentation using the trained ST-DNN. During inference time, the regions of segmentation are estimated iteratively together with updates of the ST-DNN for each of the regions as they evolve. The evolution of the regions to determine segmentation is obtained by optimizing the following energy (based on the classical energy Chan & Vese (200)):
N Σ
E(R)
(2; W) - ai l2d2 +
ds,
(8)
R.
JRR;
where R = ( R1, R2,..., Rx) are regions in segmentation, F FR, is the shape-tailored descriptor from the DNN given the learned weights W and within R, and 3 0 is the arclength regularization parameter. Note that this energy differs from the loss function used for training in two ways. First, the second term equation 5 is omitted as is used in training to avoid the descriptor from learning to be uniform across different textured regions; during inference, the network is already trained to be different across different regions. Second, we add regularization to keep the region boundaries smooth; is not needed in training since do not solve for the regions as ground truth is available.
5

---

Under review as a conference paper at ICLR 2021
To minimize the (non-convex) energy with respect to the region, we gradient decent. The gradient with respect to the region R, is approximately given by [R, (z; W) - arll2 ||E (z; W) - aj a3113 + _Ks,]N; where N₂ is the unit outward normal to the region, and K; is its curvature. The curve (boundary of regions) evolution to determine the regions is implemented with a method analogous to level set methods methods[Osher & Sethian (1988) by evolving smooth indicator functions of regions for convenient implementation, details of the algorithm and implementation are shown in Algorithm 2 in Appendix DJ The method involves joint updates of the regions and the shape-tailos tailored descriptors within the evolving regions (see Figure [T. We used box tessellation initialise the regions, typical of level set methods for segmentation. Our method typically takes few iterations (approx. 20) to converge in our experiments.
5 EXPERIMENTS
Network Architecture: We use a layer ST-DNN, which is optimal for the datasets used: fewer layers lead to less accuracy and more layers lead to overfitting (see Appendix [E). The layer fo outputs a 40 dimensional descriptor, with 3 color channels, a gray scale channel and oriented gradients at angles {0, π /4, 2/2, 3T/ /4] over 5 shape-tailored smoothing levels (scales) = (5, 10, 15, 20, 25). The four fully-connecter
layers have 100, 40, 20, 5 hidden units respectively. The smoothing parameter a for all subsequent layers is set to 5. The training on the datasets (below) takes less than 2 hours on Nvidia Quadro RTX 6000 GPU and Intel Xeon 2.60GHz CPU. The inference time (joint segmentation) is 2 seconds images of size 256 x 256.
Datasets: We apply ST-DNN to texture segmentation (since covariance and robustness properties important for texture descriptors Julesz (1981); /Sirre & Mallat (2013). We evaluate two challenging texture segmentation datasets Khan et al. (2015) the Real-World Texture Segmentation dataset (RWTSD)- 256 complex real-world images (128 training and testing images) and the Synthetic Texture Dataset consists of 200 test images and 300 training images generated from the Brodatz dataset. We have also tested on multi-region segmentation BSDS500 and Synthetic Texture datasets, details are provided below.
We compare our method against popular deep learning architectures in computer vi- sion - DeepLab-v3 Chen et al. (2017), and FCN- ReN-Res He et al. (2016). In our notation resnet/deeplabbv3x> 3-yy, resnet and deeplabv3 represents FCN-ResnetI01 and DeepLab-v3 respectively, x denotes the data used in training (subsequently fine-tuned on the texture segmentation dataset). 'x' can be 'm' for MSRA dataset, 'd' for DUTS dataset, 'all' for a combination of datasets (see Appendix [ for more details), and 'TD' for RWTSD (augmented with 8 rotations and 5 scales). 'y' denotes the loss function, 'ce' represents cross-entropy and 'ours' represents the loss introduced in this paper. Segmentation for all methods is done by minimizing equation 8 We also compare our methods against the state-of- ohe-the-rrt methods for texture segmentation, which contain both the classical Arbelaez et al. [b;; Arbeláez et al. (2014); [sola al. (2014) 4): Kokkinos
Methods:
(2015): Khan et al. 2015) and deep-learning methods Khan & Sundaramoorthi (2018) 8). ST-DNN is trained only on texture datasets.
Evaluation Metrics:
We compare on evaluation metrics from Arbelaez et al. 2011a). Ground truth covering (GT-Cov), Random Index (RI) and Variation of Information (VOI) measures region accuracy (higher GT-Cov, RI and lower VOI are more accurate), and F-measure (higher is better) measures boundary accuracy.
Ablation Studies:
We have performed ablations studies that are summarized in Appendix E Table They show that more PDE filtering layers give higher accuracy (up to point of overfitting at layers), and that shape-tailored descriptors updated as the region updates outperforms non-shape tailored descriptor (ST-DNN computed on the whole image and not updated as the region evolves).
4
To demonstrate the covariance of ST-DNN to translation and rotation, we performed an experiment on Real-world Texture Segmentation dataset. Each image in the test set was randomly rotated with 0 E (T/6, 2tT/ 6,, 3 3 /2, 4 positions (to simulate translation) in the rotated image; we ensure the rectangle only contains data from the original image. We segment the original and the transformed image, denoted S[/] and S[I a w). respectively, where w is the transformation used to produce the translated/rotated image. We then measure the difference between S[T] w and S(T o w) through GT-covering; both should
Testing Covariance:
4 5n /6, T and cropped to rectangle at random
6

---

Under review as a conference paper at ICLR 2021
be equal if the descriptor is covariant. Results summarized in Figure 2 ST-DNN outperforms resnet10l-all--e by margin of almost 25%. Note ST-DNN uses no data augmentation, whereas the competing networks augmented with translated and rotated images from RWTSD. Perfect covariance may not be achieved as translation/rotaaion also necessarily include occlusion.
-ealeti0l-all-ce
ST-DNN
image
90 retation
image
90" rotation
ST-DNN
resnet101-all-ov
GT Covering
Rand Inlex
GT Covering
Rand Index
0.87
0.89
0.69
0.70
Figure 2: Comparison of covariance rotation and translation of ST-DNN, and sota CNN descriptor. Left: A sample result with outputs for original and transformed images. Right: Quantitative result on Real-World Texture Dataset: Higher scores indicate better covariance.
Testing Deformation Robustness: To demonstrate robustness to deformation, we apply the trained networks above to randomly deformed versions of the RWTSD test set. We generate random smooth deformations using truncated Fourier series: v(x) Σk=-N Os exp(i2xk xx) where E [0, 1] k (k1, k2), wxx) x + v(x) is the deformation, a is randomly generated, and N = 10 (appropriate for the resolution). The Soboev norm is ||u||2 = |ao! Σk=-N |k|
[|
For each image in the dataset we generate 8 random deformations of varying norm ||/|| from 10 to 80 in steps of 10. We examine the robustness of descriptors to deformations of increasing norm by comparing the segmentation of the original and deformed images similar to the previous experiment. Results and qualitative samples are Figure 3 which show that ST-DNN more robust by large margins than competing descriptors, and the robustness over competing methods increases with increasing norm. Note a descriptor could be robust, but not accurate, but this is not the case for ST-DNN (next experiments).
original image deformed images (increasing deformation
Ground Covering
Sobolav Norm
20
40
ST-DNN
DeepLab v3
0.85
0.76
0.66
FCN ResNet101
0.81
0.75
ST.DNN
0.85
DeepLab-v3
Rand Index
Scholev Norm
20
40
DeepLab-v3
0.86
0.77
FCN ResNet101
0.82
0.77
0.68
FCN-ResNet01
ST-DNN
0.89
0.86
0.82
Figure 3: Comparison of robustness to deformations of ST-DNN with sota CNN descriptors. Sample results segmentation of original and deformed images (left), and quantitative results (right): higher values indicates more robustness.
Comparison of ST-DNN Standard DNNs: We segment descriptors (ST-DNN and common deep network backbones) by minimizing equation g Quantitative results are in Table and qualitative samples are shown in Figure 5 ST-DNN outperforms all other descriptors. ST-DNN has 8900 parameters and is trained with only 128 training images of real world texture dataset. ST-DNN is around orders of magnitude smaller than standard deep networks and takes around orders of magnitude less training data (e.g.. FCN-ResNetll -1l-allouurs uses 50,000 images plus augmented data and has 45 million parameters), but still outperforms these networks (see Figure
Comparison to Texture Segmentation Methods on Real-World & Brodatz Synthetic Texture Datasets: We compare to state-of-the-a texture segmentation methods: edge-based (gPb, MCG. Kok, CB) et al. (201 1b): Arbeláez et al. (2014); Isola et al. (2014); Kokkinos (2015) and region based (STLD Khan et al. (015) and Siamese Khan Sundaramoorth (2018)) STLD is shape-tailored (but hand crafted) approach using PDEs, and non-STLD uses the same PDE as STLD but on the whole image so is not shape-tailore
Siamese uses a neural network on the channel dimension returned by STLD, but does not layer PDE solutions for both texture datasets are in Table Table[land and a few qualitative samples the results are in Figure [ Our method out nut-performs all others by significant margins all regions metrics and achieves close
approach. Quantitative results
7

---

Under review as a conference paper at ICLR 2021
Trainig Data Size vs Accuracy
Number of Farameters vs Accuracy
0.95
GT-ONN
* ST-DMM
A manei100 101-a
[ despleso
Acc.00]
00000
0.82
0./0
Totol
Training images
Network Purammttrrs
Figure 4: ST-DNN is smaller in size and uses fewer training images compared with SOTA DNNs.
images
ground truhh
STLD
remect001--cce
resnet reseet00-alllourr
desplabv3-all-ours
ST-DNN
Figure 5: Sample representative results on Texture Dataset. We compare the ST- DNNs (ours), STLD, and deep learning based methods.
to the best result on the contour metric RWTSD, while on the Synthetic Brodatz, method out performs all methods on all metrics. More visual results and comparison to more methods is in Appendix H
Contour
Repion metrics
Table 1: Results on Real-World Seg- mentation Datasets of Deep Net- works. Algorithms are evaluated on contour/ region metrics. Higher F- measure for the contour metric, ground truth covering (GT-cov), and rand index indicate better fit, and lower variation of information (Var. Info) indicates better fit to ground truth.
F-meas.
GT-cow.
Rand. Index
Var. Info.
(ouns)
101--ce
0.39
0.84
0.75
0.70
resact-d-ours
0.83
0.74
resnet]0l-all-ce
0,04
0.79
0.55
1.39
resmet101 oull-ours
0.48
0.83
0.87
0.50
Conet100-TDD-omm
0.17
0.83
0.66
0.94
remee101-TD.cc
0.11
0.80
0.63
1.35
deeplahy3-all.com
dcoplaby3-all-on
0.42
0.85
0.76
0.67
HEl 2015
0.04
0.53
0.60
1.69
Comparison on Multi-Region Segmentation: We have also tested our method on two multi-region segmentation datasets, BSDS500 2) Synthetic Texture dataset. For BSDS500 dataset we train our methods and state of the art deep networks on the training + validation set (200+100 images augmented with with 8 rotations and scales) and on the test-set on BSDS dataset (200 images). Results are summarized in [ad few quantitative samples are show in figure Iin Appendix E BSDS500 not the best dataset for training on region segmentation as the regions are not marked based appearance and instead watershed is used edge maps to generate segments. Hence region with similar looking appearance are often marked as separate regions in images. Despite this drawback ST-DNN performs on par or better than state of the art Deep Neural Networks. During training all images are resized to 256 x 256 and normalised to have zero mean and unit variance. In post-processing, the outputs clustered into 20 regions and then regions with less than 2% pixels
8

---

Under review as a conference paper at ICLR 2021
Real-World Texture Datasel
Synthetic Brodatz Texture Datasel
Contour
Region meirics
Contour
Region meirics
F-meas.
G/-cov.
Rand, Index
Var.
GT-cov.
Rand. Index
Into.
ST-DNN (OurS)
0.64
0.94
0.94
0.35
ST-DNN (ours)
01.49
0.92
0.92
0.44
Stamese
0.65
0.92
0.92
0.43
Siamese
0.90
0.89
0.46
SILD
0.58
0.86
0.88
0.63
STLD
0.41
0.87
0.86
0.53
mog
0.54
0.82
0.85
0.66
gPb
0.40
0.81
0.82
0.75
0.79
1.09
DOn STLD
0.20
0.83
0.84
0.79
non STL.
0.18
0.84
0.84
0.65
Table 2: Results on Real-World Segmentation Dataset (left) and Synthetic Dataset (right). Com-
parisons are performed against CB Isola et al. (2014), gPb Arbelaez et al. Tb)) Siamese Khan &
Sundaramoorthi (2018), Kok Kokkinos (2015). meg Arbeláez al. (2014) See Table caption for
details on the measures. Additional results in Appendix
BSDS
Synthetic Multi- Regilnn Texture Dataset
Region metries
Region Metric
GT-cov
Rand. Index
Var. Info.
Avg. Accuracy
Deeplab.
Deeplah.s2 FCN-resnet101
FCN-Resnet
0.38
0.71
1.98
0.43
ST-DNNNours
0.39
0.73
1.80
ST-DNN (ours)
0.45
Table 3: Results on BSDS500 (left) and Synthetic multi-region Dataset (right). Comparisons performed against ttate-of-the-aae deep learning based methods. See Table [caption for details on the measures
of the entire image smoothed out using conditional random fields, pydensecrf Mibierrrrrr Following which we run 20 iterations of [2] For quantitative comparisons we have provided analyses on region metrics BSDS500 benchmark.
We have also tested on a large scale muuti-regioon synthetic texture segmentation dataset This extension of Khan et al. (2015), we have maximum of 4 regions per image comprising different appearance textures. The dataset consists of 42000 training images, 3000 validation images and 5000 test images. The images are generated from 1084 textures collected from Brodatz dataset. Our methods is trained on only 200 images from the dataset, other methods are trained on the entire training set. Our method outperforms state of the art deep learning based methods by healthy margin despite being significantly smaller in number of parameters and training data required for training. Similar to BSDS500 dataset we resize images to 256 x 256 and normalise them to have zero mean and unit variance. In post-processing we simply cluster the image into 4 regions and run iterations of 2. For quantitative comparison we have provided analyses on region accuracy metric provided with the dataset, motivated from PascalVOC class accuracy metric.
dataset is designed
CONCLUSION
We have introduced ST-DNNs, which generalize CNNs to arbitrary shaped regions by stacking layers that solve PDEs. These are relevant segmentation since they avoid aggregation of data across segmentation regions. They are also covariant to translations and rotations, and robust to deformations. Experiments showed that ST-DNNs achieve state-of-the-art results on texture segmentation benchmarks: they outperform state-of-the-art deep networks by 20% edge metrics and 10% on region metrics while being 3 orders of magnitude smaller and using 2 orders of magnitude less training data. Experiments also validated covariance and robustness of ST-DNNs. To show the strength of the formulation we have tested it on four datasets, which include binary and multi-region segmentation datasets.
http:://gtttaaaccccom////fffffdd dataset

---

Under review as a conference paper at ICLR 2021
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gameut (eds.), Advances in Neural Information Processing Systems 32, 14537-14546. Curran Associates, Inc., 2019.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 898-699 May 201 la. ISSN 0162-8828. doi: 10.110001. pRL
doi. ozg/ /1////1...00 1109/111AAATT 2010.161
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(5):898-9) 9966, 201 lb.
Pablo Arbeláez. Jordi Pont-Tuset, Jonathan Barron, Ferran Marques, and Jitendra Malik. Multiscale combina- torial grouping. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 328 335, 2014.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transforaatons? Journal of Machine Learning Research, 10018)))--,, 2019.
Alberto Bietti and Julien Mairal. Invariance and stability of deep convolutional representations. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems 30, pp. 6210-6220. Curran Associates, Inc., 2017.
Tony Chan and Luminita A Vese. Active contours without edges. Image processing, IEEE transactions on, 10 (2):266-22) 277, 2001.
Liang-Chial Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. jun 2017. [RL.tttp://////// wttpttttt//tt /ttp:////aiivv/a/ http://vwwg/a///////o/a/// 000.006667777777
Ming Ming Cheng, Niloy Mitra, Xiaolei Huang, and Shi Min Hu. SalientShape: Group saliency in image collections. Visual Computer, 30(4):433 453, 2014. ISSN 01782789, doi: 00.10077/00031-01330066
httt::/t:://// //mpp//g//mcc.gg./m//m net. ///
M.M. Cheng, G.X. Zhang, N.J. Mitra, Xiaolei Huang. and S.M. Hu. Global contrast based salient re- gion detection. In IEEE International Conference on Computer Vision and Puttern Recognition, pp. 409-416, 2011. URL httt:://c..cs.c. www.hussttinnnnnoeeo.oec.umuaeet edu, en wttppp/ppppe/eeeeeeeeee //ieeexplore Leee, org. wwtp/o/ggppppppaee
cmm/ saw.scy//teeeppppee alll. 18t.aaammmmmrrrr55344454
Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu. Learning to predict crisp boundaries.
CoRR, abs/1807. 10077 2018. URL http://araa http::///xxww..r///p//..... 807. 10097
Liang Du, Jingang Tan, Hongye Yang, Jianfeng Feng, Xiangyang Xue, Qibao Zheng, Xiaoqing Yc, and Xiaolin Zhang. Ssf-dan: Separated semantic feature based domain adaptation network for semantic segmentation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Lawrence Evans. Partial differential equations. 1998.
Robert M. Haralick and Linda G. Shapiro. Image segmentation techniques. Computer Vision, Graphics, and URL
Image Processing, 29(1):100-133 jan 1985. ISSN 0734189X. doi: 10. https: //1 http::://iinniinninec e1 sevi er com retr eeee/
/507341111111 0 898859018377
Jianzhong He, Shiliang Zhang, Ming Yang, Yanhu Shan, and Tiejun Huang. Bi-directional cascade network for perceptual edge detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-can. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980-2909 2988, Oct 2017. doi: 10.1 10.1109/CC222017..32..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, volume 2016-Decem, pp. 770-778, 2016. ISBN 9781467388504, doi: 10.11099CVP222016.90. URL
http: http://mmmgeeeee..ce // Lttpteess/LVV///113/////
Byung-War Woo Hong, Stefano Soatto, Kangyu Ni, and Tony Chan. The scale of a texture and its application to segmentation. In Computer Vision and Pottern Recognition, 2008. CVPR 2008, IEFE Conference on, pp. 1-8.
IEFF, 2008.
10

---

Under review as a conference paper at ICLR 2021
Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Conet: Criss-cross attention for semantic segmentation. The IEEE International Conference on Computer Vision (ICCV), October 2019.
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Crisp boundary detection using pointwise mutual information. In Computer Vision-ECCCV 2014, pp. 799-719 814. Springer, 2014.
Bela Julesz. Textons, the elements of texture pereeption, and their interactions. Nature, 290(5802):91-97 1981.
Nacemullah Khan and Ganesh Sundaramoorthi. Learned Tailored Descriptors for Segmentation. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018. ISBN 9781538664209. doi: 10.1 109/CVPR. 201.P...018.007777
Naeemullah Khan, Marei Algarni, Authony Yezzi, and Ganesh Sundaramoorthi. Shape-tailored local deseriptors and their application to segmentation and tracking. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pp. 3890-3899, 2015.
Iasonas Kokkinos. Surpassing humans in boundary detection using deep learning. CoRR, abs/1511.073366 2015. 173366
Shawn Lankton and Allen Tannenbaum. Localizing region-based active contours. Image Processing, IEFE Transactions on, (7(11):2029-203) 2008.
Guanbin Li and Yizhou Yu. Visual saliency detection based on multiscale deep CNN features. Technical Report 11, 2016. Uttps:::: www.seitttsoogooloo.oooooooogooe com/ simmm/siiii wwtsssstsssee httpe///ii/g//mmm/aaaaaaaeeeeeee
Xia Li. Zhisheng Zhong, Jianlong Wu, Yibo Yang. Zhouchen Lin, and Hong Liu. Expecaatin-mmaxmiiaato maximizaaion attention networks for semantic segmentation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille. The secrets of salient object segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 280 287, 2014. ISBN 9781479951178. doi: 10.1109/CVPRR.20144 2014.43.
Chenxi Liu, Liang- Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan I.. Yuille, and Li Fei-Fei. Auto- deeplah: Hierarchical neural architecture search for semantic image segmentation. In The IEFE Conference on Computer Vision and Puttern Recognition (CVPR). June 2019.
Jitendra Malik and Pictro Perona. Preattentive texture discrimination with early vision mechanisms. JOSA A, (5):923 932, 1990.
Oleg Michailovich, Yogesh Rathi, and Allen Tannenbaum. Image segmentation using active contours driven by bhattacharyya gradient flow. Image Processing, IEEE Transactions on, 16(11):2787- 2801, 2007.
Kangyu Ni, Xavier Bresson, Tony Chan, and Selim Esedoglu. Local histogram based segmentation using wasserstein distance. International Journal of Computer Vision, 84(1):9--11,,2003 2009.
Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6583):607-609, 1996. doi: 10.1038/381607aa) URL
htttss://dci.oo org. /00 1038/38160788
Stanley Osher and James Sethian. Fronts propagating with curvature-dependent speed: algorithms based on hamilton-aacobi formulations. Journal of computational physics, 12-1), 1988.
Yanwci Pang, Yazhao Li, Jianbing Shen, and Ling Shao. Towards bridging semantic gap to improve semantic segmentation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Laurent Sifre and Stephane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimina- tion. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pp. 1233-12322211 1240. IEEE, 2013.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Disa-Biinchh, and R. Gamett (eds.). Advances in Neural Information Processing Systems 31, pp. 10381- 10392. Curran Associates, Inc., 2018.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection, 2015.
11

---

Under review as a conference paper at ICLR 2021
Saining Xie and Zhuowen Tu. Holistically-NNste Edge Detection. International Journal of Computer Vision. 125(1-3):3 18, apr 2017. ISSN 15731405, doi: 10. 10.10077/11263-017-2200
WwRLhttp://arxi..cm
Abs/1504 06375
Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. Hierarchical saliency detection. In Proceedings of the IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition, pp. 1155-1162, 2013. doi: 10.1 CVPR.2013.153 URLhttp:: //wtt: cse. cuhk wec.hkeedd.eee wwt.://eooiiaaaaaaa htt.b/...dd...iieeeeaaaaro/opop
Anthony Yezzi Jr, Andy Tsai, and Alan Willsky. A statistical approach to snakes for bimodal and trimodal imagery. In Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, volume 2. pp. 898-903, IEEE, 1999.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness computer vision. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alché-Buc. Fox. and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 1127611286. Curran Associates, Inc., 2019.
Anthony M. Zador. critique of pure learning and what artificial neural networks learn from animal brains. Nature Communications, 10(1):3770, August 2019. ISSN 2041-1723. doi: 00.1038/s41467-00 1176-6.. Number: Publisher: Nature Publishing Group.
Yu Zeng, Huchuan Lu, Lihe Zhang, Mengyang Feng, and Ali Borji. Learning to Promote Saliency Detectors, In CVPR, pp. 1-7, 2018.
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In The IEEE International Conference Computer Vision (ICCV), October 2019.
A APPENDIX: OUTLINE
The Appendices material is divided into the following sections.
Analytical Proofs of Covariance and Robustness B
Alternative Training Method for ST-DNN With Lower Memory Requirements
Implementation Details for Training and Joint Segmentation
Additional Experimental Analysis and Results
B ANALYTICAL PROOFS FOR COVARIANCE AND ROBUSTNESS
The proof of Theorem | (covariance of ST-DNN to rotation and translation) follows from basic properties of the Laplace equation Evans (1998); we state these properties terms of our language of covariant operators, and show the proof for the convenience of the reader. We will show Theorem (robustness of ST-DNN to domain deformations) as a consequence of properties of linear PDE theory.
We repeat the definition of covariant operator:
Definition An operator T + is covariant to a class W of transformations if
T(T o = [TT] w.
for every I E I and w E W.
We show covariance of the Laplace operator, and as a consequence, the covariance of the Poisson PDE and the ST-DNN.
Theorem The Laplacian operator A Es rotation matrix.
8 dzy
is covariant rotations, - Ra where R is
Proof Let u E I, and let R be a rotation. Consider
T
(Rx)
u( Rz)
Ru()))
(9)
dr,
Or.
o, R Vu (xx (vu(Rx))= Rey
(10)
12

---

Under review as a conference paper at ICLR 2021
Also.
0
du
4 Ou T dz.
(Rx)
Rea
(II)
dr:
dry
so,
0
[u)( = Hu( ) Re;
(12)
0z,
s0,
0 Fr.
[V((Rx))* = (Re:)" (oa))"Ho Hu(Re).
(13)
Thus,
8 dn?
u (Px)
[u((Rr)) Re: (Re,)* Hu((H)()))
(14)
Or,
Then,
0²
Auuo R)(x) = Σ
u (Rx)
(15)
dxg
Σ " Hu(Re))ee = tr[R² Hu(Rx)
(16)
-r((Hu(Rx))
(17)
Au(Ra))
(18)
where the second last equality is due to the invariance of the trace to similarity transformations.
Theorem The Laplacian operator is covariant to translations, I - x + t, where is vector.
Proof 2 We have,
i Ox;
Ou Oz;
ju(a f)]
(r t).
(19)
Similarly,
8²
Ou
[u(x - t)] =
(x + t)
(20)
dr
0z,
82
(x + 1)
(21)
dx?
Thus, Auuo (1)) = (Au) o T where T(x) = x. +
Corollary The solution of the Poisson equation, i.e.,
u(x) auu(() 1(x),
(22)
2 = T[]] is covariant to translations and rotations.
Proof 3 This follows from the covariance of the Laplacian, and the identity map.
We can now show covariance of the ST-DNN to translations and rotations.
Theorem The ST-DNN equation 3 is covariant to translations and rotations.
Proof 4 Since ST-DNNs are composition of solutions of Poisson Equations with fully connected layer across channels and non-linearity in multiple layers. A linear combination of channels and
point-wise non-lineat operation preserves the covariance of rotation and translations, hence the ST-DNNs covariant translations and rotations.
13

---

Under review as a conference paper at ICLR 2021
We show robustness of the ST-DNN to domain deformations with respect to the Sobolev norm; that is, we show that the output of a ST-DNN layer does not change much if deformed by a transformation with small Sobolev norm (a smooth transformation that has small displacement). The Sobolev measures the smoothness of the deformation, i.e., the norm of the deformation and the gradient of the deformation. We have the following theorem:
Theorem 6 The solution of the ST-DNN is robust deformations, i.e.,
|T[I w| T[ S Cl [oll-id]mm id]
(23)
where is the mapping from the image to the solution of the SI-DNN, w : Ω -+ is a domain deformation, id is the identity map, and 11 indicates the Sobolev norm.
Proof This is a consequence of Lemma equation ! below, which shows that each layer of the ST-DNN is robust. Stacking such layers preserves the robustness by applying Lemmal Caanma/suucsaaa successively.
We now prove robustness of layers of ST-DNN. For convenience in the proof, we assume the input operates on the domain = R², which avoids having to consider the boundary and more complicated formulas that do not affect the essence of the argument. Let
/[ = - D K)
(24)
be a layer of ST-DNN. Here r is the rectified linear unit, K' is the kernel representing, the Green's 's function of the Poisson equation (we assume for this proof that the domain the image is all of R&²), D is the derivative operator representing oriented gradients (or parial derivatives of an finite order), and L is weight matrix of fully connected layer across channels.
We state the robustmess of a layer of ST-DNN:
Lemma A layer, f[]]= = r[ o Do o K * 1I, of a ST-DNN is Lipschitz continuous with respect to diffeomorphisms in the Sobolev norm, ie.,
[s[ w| f[]] I C'[lw-i|]
-d]
(25)
where id(x) = x is the identity map, and C is a constant (independent of w and only of function of the class of images), and [|l|] ? = (g + [w(x)]²)dx, Note that w id is the displacement.
Proof 6 Let w be a smooth diffeomorphism. Then by Lipschitz continuity of the ReLu,
If[l wx)x) f(]]()) S |M w)(() M I(x))
(26)
where M = Do K. Note that by a change of variables.
M * [(x) Σ M(x MM(x-w(y) ((V)) det Vw(y).
(27)
Note that the determinent of the Jacobian appears if we weight the sum by the measure, which approximates the integral. Therefore,
M () M 1(x) IIM(= y) -Mxx w(y)) det Vw(y))((w(y)) (28)
We let w(y) y + v(y). This gives us det Vw(y) = 11 dir(u())) + det Vu(y).
We may bound the second term as
(div(u(y)) + det V
by basic inequalities. Therefore,
[M y) - (xx w(y)) det Vw(y)] 5
IM( y) - -M(x w(v))l + MMM( vo(r)) (29)
14

---

Under review as a conference paper at ICLR 2021
By Lipschitz continuity of the Poisson kernel and derivatives, we have
[M - y) (((x-ww())) & Call (u(y)]
(30)
Note that the Poisson kernel has singularity at the origin, the statement is not precise; however, as common in PDE analysis, as we will below compute integrals of the left hand quantity, we can break the integral into two terms and the other that integrates the right hand side, that we analyze below. The former will disappear to zero as the ball goes to zero. We omit the details avoid hiding the main argument.
that integrates the singularity in a small ball (which is finite)
We also have that
IM w(y))l S C2|1L|1oo
(31)
Therefore,
f[]ow( - -11[(=) c1 (1v()) [Vv(y))
(32)
= -c idl
(33)
For multi layer multiple layers network with N layers will have:
T o m| T(1) s II Col 1/ // - iddl = Cl(w id idl
(34)
C ALTERNATIVE TRAINING METHOD FOR ST-DNN
In this section, we provide an alternative algorithm for training ST-DNN that has a smaller memory footprint than the method present the main paper to avoid having to train on small images. Unfortunately, this is not supported in current deep learning packages, so it requires a custom implementation, and it may be more computationally costly depending on the size of the image. The basic idea is instead of back propagating (through large matrices to solve the PDE, and storing matrix multiplies) to determine derivatives of the loss with respect to the weights, we instead forward propagate the derivatives of each layer through variations of the rest of the layers (see Figure [6). The advantage of this method is that the variations of the layers with PDEs can he computed by solving the PDE itself with input being the layer weight derivative. This can be solved with an iterative technique. This does not require accumulating matrix multiplies consuming large memory as standard back propagation, forward propagating through the PDE is analogous to matrix-vector multiply, which as solved through an iterative technique and does not need to store the matrix explicitly.
I
f1
f2
f3
f4
L
8f1
8f2
ffz
8f4
fa T
L
R122
0f1
0fz Own
0 Dwg
0f4 Dwa
Ow
Figure 6: Example loss gradient with respect to weights calculation for the ST-DNN using the "forward propagation" method. The example red path in arrows represents the calculation of the gradient of the loss with respect to the weights the second layer. This procedure avoids having to accumulate large matrix multiplies (to solve PDEs) in pec-proppagtiion
Below, we present the mathematics to show that this forward propagation technique is valid for the case of PDES layers, and give details of the technique.
15

---

Under review as a conference paper at ICLR 2021
C.I VARIATION OF DAILORDD DESCRIPTORS
The shape-tailored descriptors are given by Poisson equation with Neumann boundary conditions.
Tu(z) - aLu(z) = (x) Vu(x)) N =
I E
(35)
I E OR
To take the variation of the PDEs we take the variation of each partial differential equation. To find the variation of the first equation we first calculate the value of u(x) when it's input is perturbed by 81
(z) - Auurrerr (x) [(x) + (x)
(36)
The variation the direction of 57 is defined as
4 u1+661 (x) (x()
lim c+0
(37)
Next, we replace this value in the equation for the variation and get
4ss (a) - (n)ur(()) (x)).
(38)
Similarly for the second term, we have
Vusi (z) N = 0.
(39)
So the variation of the shape-tailored descriptor denoted by ux(x)) = (.. in the direction of perturbation (xx of the input image (xx) given by:
(x) (2) Vun (z) 0
uun
a (1x)) R
(40)
x 0R
C.2 DERIVATIVE OF ENERGY WITH RESPECT TO WEIGHTS
First, we find the variation of layer of the chape-tailored network. A layer shape-tailores network in composed of r o L o T. The expression for variation of a layer of the network is given below:
8f()).81 (r (1)
(41)
(rroL)(T())T(1
where,
(r = ("( L
(42)
Next, we calculate the variation through the entire network.
8F(1).01 =s'8fm fm- o fi..81
=m'mfm (m)) Fm =(soro o Lm) y' (T[ oT. (T. Lm Lm-)) o 8T. Im 2
(43)
... TT. (T)((T) (T(ØT)
where, P; is the output of the 7th layer of the network.
Now, we derive the expression for derivative of the energy w.r. to the weights of the network. These derivative would be used to update the weights of the network during the application of the stochastic gradient decent to the energy function.
To start with, we derive the derivative of the descriptor w.r.t. the weights of the network (w; are weights b; are biases of the fully connected layer). The derivative w.r.t to the weights of the last layer of the network are:
(F(x)
=s' 0fm
Own
Fwm
(44)
=s' (r o Lm o F (x)))) o of (Fm(())) Fm-1(2)
16

---

Under review as a conference paper at ICLR 2021
0F(x)
05m
=8
Obm
Hbm
(45)
=s' Lm [[Fm=1(=)]]) or (F
where F; represent the output of the network at the 7th layer. Next, we calculate the derivative of F w.r.t we and b;, where and by the weights and biases of the fully connected layers, and i represents intermediate layer of the network.
0f(x) =s' mm 585m1 88 o 551+1 Qf:
Ow;
dw,
(46)
=(soro I.m) Y ('T)((((()) [ (m)) 07. 0T..(r m
(r Im
... ...o (elrro/41lll (TH)) (2)1(1)
similarly,
(F(2)
=s o dm o o 8fmm1 mm. o ofit1 0f:
0b;
0b.
(47)
= (soro Lm) [T[[mm-l) (T Lm-1)(((Fmm)))
[-
8T. Lm 2) (T) m3)) ... .r. Lif (((1) (f)((())
Finally,
Ou, E(T) Σ <
1
2(F(x) - (())),
(F(x)
> 2 dx
R
R:
(F
Σ
2(F(x) - a(x)),
>12 dydr
R.
RR
Ow,
R.
Ra
(48)
(x
ΣΣ
2(aa aj),
>L dx
R
Owi
RR
FF(x)
ΣΣ< 37
2 2(a)
ay),
JR.
P1 dx
R
Owi
where < > represent the inner produce two vectors. Now, that we have the derivatives the energy w.r.t. the weights of the network, can train the shape-tailored networks for applications like segmentation (see Figure for a visualization of the forward pass for gradients calculation).
D IMPLEMENTATION DETAILS
In this section we outline the important implementation details for our experiments. Figure shows a schematic of the ST-DNN in experiments.
D.1 ST-DNN SETUP
We setup two versions of ST-DNN, one with MatConvNet where we use a C++ wrapper for the solution of Poisson equation of Eq 35 and a faster version Pytorch. the MatConvNet version the Poisson equation with Neumann boundary condition is solved with conjugate gradient algorithm, and in Pytorch version, we explicitly invert the right hand side of Equation 35 find the solution. We noticed that training by sol solving the PDEs with (conjugate gradient algorithm on full size images) MatConvNet was slow compared to the faster Py PoTch version, and didn't 't have support available in Py Porch and TensorFlow and the results for the faster Pytorch solution (trained on downsampled images) were almost on par with the MatConvNet version (at least for the datasets we have tested on), however, learning ST-DNNS from full size images might be helpful in other applications and
17

---

Under review as a conference paper at ICLR 2021
Image
L2
L3
LO
Mask
We have four layer ST-DNN with additional pre- processing layer. The pre-processing layer, LO the schematic, extracts color channels a gray scale channel and 4 oriented gradients at 5 scales of the network have a smoothing layer a = (5). a fully connected layer, and a non-linearity. The number of hidden units for each layer 100, 40, 20, 5 respectively.
Figure 7: Schematic of the ST-DNN:
{5, 10, 15, 20, 25) The subsequent layers
future work. The two methods are different in how they handle the solution of the PDEs. Below we provide the details of both methods.
As first step, we provide the discretisation of the Poisson Equation 35
u(i,j) a
Σ R.EEN()))
(u(k, u(i.))) (i,,)), for (1,))
(49)
where (i, j) represents the 4-pixel neighborhood of pixel (i.j), which represents the 2th row and jth column, and intersection means that only neighbors in the region are considered as implied by the Neumann boundary condition in the PDE, avoiding aggregation outside R.
For small images, we can linearise the image and write the above equations as:
ARu=1
(50)
where, and are linearised image and descriptor respectively. We can then find the solution by simply inverting the system of equations
u= AR II
(51)
Notice that size of An (m n. n.), where and number of and columns in the image respectively and Ar depends of the mask of region R. For small images (32 x 32) the size of AR (256 256) and we can store and invert it effectively, hence we use small images for training in our Pytorch implementation. Also, notice that if we implement the PDEs solution as in Equation 51 Pytorch (autograd) takes care of the back proppagation through the PDE layers and do not have to explicitly implement the back proppagation (see Algorithm
Algorithm Training of ST-DNNS
1: Input: Image I and ground truth mask R
2: compute: -1 defined in Equation equation 51
(Notice depends on only R)
3: initialise weights of the ST-DNN F(z) = Lm A oro Lm-1 A -1 ...A - Loo
4: repeat
5: Update Weights: using gradients from backpropagation
6: until validation error minimized
For full size images we had to implement the equations of Section C. where we want to solve the PDEs without explicitly storing the smoothing kernel for each pixel in the image (as it would have huge memory requirement). We can solve Equation 49 using conjugate gradient algorithm as A in Equation is symmetric positive definite (more detail in Khan et al.) (2015)). Notice however that, if we use this approach we can not use autograd algorithms standard packages like Py Torch and TensorFlow because Equation 50 is the relations from output to input (and not from input
18

---

Under review as a conference paper at ICLR 2021
output), have to implement the gradient decent equations by calculating equations of Section C explicitly. We have implemented this using MatConvNet, however, we see that the results (at least on the datasets have tested on) of training size image. Hence in the paper we have reported results from the Pytorch implementation. The full implementation, however, might be useful in other applications and future work.
small scale images are par with results of full
D.2 INFERENCE (JOINT SEGMENTATION) ALGORITHM
We present the details of the algorithm for segmentation and joint ST-DNN descriptor updates (Algorithm{2 2 We use an approach analogous to level set methods, but instead evolve functions 6 : Ω > that are smooth indicator functions of the regions R. The gradients of the energy with respect to the regions can be converted into an evolution of the indicator functions shown in the d.
algorithm. Note that K; is the curvature of the boundary of region R, and can be written in terms of
Algorithm 2 Texture Segmentation with ST-DNNs
1: Input: An initialization of Φ;
2: repeat
3: Set regions: Rg far Q i= argmax,0,
4:
Compute dilations, ((R)). of R.
5:
Compute FR, in (RR)). compute
1/| .nn (x)dr.
6:
Compute band pixels B; (RR,) n (
7:
Compute G; = FR, (z)) - a: 12 for x E B, evaluated from the neural network.
8:
Update pixels E (R))) (R))) follows:
7 (z) = (x) -
(c AT Bk. (o]
(52)
9: Update all other pixels as
)T+ = of (x) AT Br.Bo,/OO[())
Clip between and 1: max (0, min 0.)}. until regions have converged
11:
E ADDITIONAL EXPERIMENTS AND RESULTS
E.1 ABLATION STUDIES
In Table 4 we show ablation studies of ST-DNN with varying number of layers and compare it against Siamese Khan & Sundaramoorthi (2018) (that takes hand-crafted shape- tailored deseriptors input and then uses a neural network in the chamnel dimension) on the Real- World Texture Segmentation Dataset. We test the performance against the number of layers in two settings. First, for both ST-DNN and Siamese, we compute the descriptors, choosing only one region that contains the whole image, i... Ro Ω to initially compute the descriptors. We do update the descriptors the regions of segmentation evolve to minimize the energy. Secondly, we update the descriptors as the regions evolve. This is done to show layering more PDE layers increases performance independent of the joint segmentationn/desriipto approach. Results under both settings (first setting on the left and second setting on the right) are shown in Table They show that increasing the number of layers in ST-DNN under both settings increases performance up to 4 layers, and then performance decreases (probably due to overfitting with too many parameters after layers). Thus, the performance of ST-DNN is due to layering multiple (shape-tailored) spatial filtering layers, one of the contributions of this work. The fact that performance increases when updating the descriptors as the region evolves, which makes the descriptors thipe-ta to the segmentation, shows that the thape-tailored aspect of the descriptor is essential.
In comparison to Siamese (result reported on 4 layers processing channels of the shape-tailored pre-pre precpescessin) with four layers ST-DNN out- out-performs on all region metrics and performs on par on the contour metric. Note that this makes perfect sense, since stacking Chape-tailone spatial
19

---

Under review as a conference paper at ICLR 2021
filtering layers smooths the data more than Siamese (which uses only one spatial filtering layer), making it slightly more difficult to localize the boundary, but the large effective spatial receptive size from layering spatial filtering in ST-DNN allows capturing regional properties better, leading to better performance on the region metrics.
Table 4: Left: Descriptors computed on whole image (without iteratively updating descriptors in Equ.
8) with varying number of layers. Right: Descriptors computed on Region (with iteratively updating
descriptors in Eqn. (8) with varying number of layers layers. Compared against Siamese Khan &
Sundaramoorthi (2018).
wathout per- rerriteration descriptor update
with per per-iteration descriptor update
Contour
Region mettics
Contour
Region metries
methed (layers)
F-meas.
GT-cov.
Rand. Index
Var. Info
method (layers)
F-meas.
GT-eow.
Rand, Index
Var. Info.
STDNN (1)
0.22
0.79
0.8
0.74
ST-DNN (I)
0.23
0.8
0.8
0.72
ST.DNN (2)
0.72
(2)
ST-DNN (3)
0.47
0.86
0.86
0.59
ST-DNN (3)
0.51
0,89
0.9
0.5
SN (4)
0.52
0.9
0.9
0.45
ST- (4)
0.64
0.94
0,94
0,35
ST-DNN
0.51
0.89
0.89
0.56
ST-DNN (5)
0.63
0,94
0,94
0.37
Siamese (4)
0.53
0.89
0.89
0.47
Siamese (4)
0.65
0.92
0.92
0.43
E.2 DEFORMATION ROBUSTNESS
We provide more detailed results from the deformation robustness experiments the main paper. Figure X shows the plots of the performance of ST-DNN against SOTA deep learning methods, showing that ST-DNN outperforms SOTA DL methods terms of robustness, and the margin of out-performance inereases with increasing deformation amount (measured by the Sobolev norm). The robustness is measured through 3 metries (GT-Cov, Rand Index and Variation of Information). The first two metries higher values indicate more robustness, and the last lower values indicate more robustness. Figure 9 shows some visual results of the experiment.
Grundhruh Convaing vs Sotalee Nore
Fanden Inttx vo Nom
Inrtetionl of Intomation Sobeley Hor
0:23
Herttou If
706 Fariett)
359
3.4
0.50
0.50
22
Sobder fort
5d00 Nmm
Figure 8: Comparison of ST-DNN with SOTA deep networks for deformation robustness. ST- DNN is more robust as measured on all metrics.
E.3 ST-DNN PERFORMANCE AGAINST COMPETING METHODS
We show some additional qualitative samples of our results to better motivate our experiments. shows additional samples of results, where we compare our ST-DNN setup with state-of-the-art deep architectures on the Real-World Texture Segmentation Dataset (RWTSD). As can be seen from the samples our results are comfortably better than the SOTA deep architectures. Table shows quantitative resuls on RWTSD. In our notation resnet /edeeee////eeeppab->>// resnet 10/deeepa/))) represents the back-bone architecture used, x represents the dataset used in training, 'd' for DUTS dataset fine-tuned on real-world texture dataset, 'm' for MSRA dataset fine-tuned on real-world texture dataset and 'all' for combination of all datasets in Table 6 and RWTSD without any fine- tuning. With 'y' represent the loss function used to train the network, 'ce' represents cross-entropy loss and 'ours' represents the loss function we have presented in this paper. We have tested state-of- the-art deep learning architecture with the loss introduced in this paper to show that although our loss function improves the results compared to cross-entropy, the bulk of improvement of ST-DNN over state-of-fe -ftt-of-the-rrt networks comes from the construction of shape-tailored covariant descriptors.
Figures 10 12
20

---

Under review as a conference paper at ICLR 2021
images (increusing deformation )
DeepLab-v3
FCN-ResNetiol
images (increasing deformation
D
PCN RecNet101
Figure 9: Sample representative results on Real-World Texture Dataset with varying deforma- tions. We compare the ST-DNNs (ours), and deep learning based methods.
Figures 13 and 14 represent the visual samples (compared to other texture segmentation methods) on the World Texture Segmentation Dataset and Synthetic Brodatz Texture Segmentation Dataset, respectively. Again, our results are better. Additional quantitative results are provided in Tables (for RWTSD) and (for SBTSD), comparing against more methods than in the main paper.
Contour
Region metrics
F-mcas..
GT-coo.
Rand. Index
Var. Info.
ST-DNN (ours)
0.63
0.63
0.94
0.94
0.94
0.94
0.35
0.35
1e-ne1100-d-ce
0.39
0.84
0.75
0.70
0.70
resnet-d-ours
0.39
0.39
0.83
0.83
0.76
0.76
0.74
0.74
resnet101
resnet resnet101-maae 011-moours
0.35
0.35
0.89
0.89
0.76
0.76
0.78
0.78
resnet101 ce
0.04
0.04
0.79
0.79
0.55
0.55
1.39
1.39
resnet10l-alli -a--ll-ours
0.48
0.48
0.83
0.83
0.87
0.87
0.50
0.50
rcsnct111-DDae
0.17
0.17
0.83
0.83
0.66
0.66
0.94
0.94
deeplabv3-all-oo
deeplaby3 all ours
0.42
0.42
0.85
0.85
0.76
0.76
0.67
0.67
HRT 2015
0.60
Table 5: Results on Texture Segmentation Datasets of Deep Networks. Algorithms are evaluated on contour/ region metrics. Higher F-measure for the contour metric, ground truth covering (GT--co),, and rand index indicate better fit, and lower variation of information (Var. Info) indicates a better fit to ground truth.
21

---

Under review as a conference paper at ICLR 2021
grand trich
backbune: resmel101: pre training: MS COCO; training DUTS; line tuiing real world texture datasel; loss Function: cruss entrupy
backbone: resnetl01; pre- traininnn MS-COCO; training: DUTS; fine fine-tuning real- world texture dataset; loss function: ours
backbonc: resneti01: pre-training MS-COCOP training: MSRA, hin-tuning real-world texture dataset; loss function: cross-cntropy
backbone: resnet101; pre- training: MS COCO; training MSRA: line line-tuning real- texture dataset; loss function:
backhome: resnet (O1: pre training:MS. COCO, training: datasets; loss functon:
backbone: resnetlol; pre-training
MS- MS-COCO; training: datasets; function:
pre-traimin: COLO, traniigg
function:
deeplab v3: pre training:
COCO: traiing:
HED
DNN (ours)
Figure 10: Sample representative results Real-World Texture Dataset. Comparison with state-of-the-art deep learning methods.
22

---

Under review as a conference paper at ICLR 2021
images
grond
backbone: 1es: pre training: MS COCO; training DUTS; line tuning real world texiure dataset; loss Function: cruss entrpyy
backbone: resnetl01; ree-traningg MS-COCO; training: DUTS: fine-tuning real-world texture dataset; loss function: ours
backbonc: resnetiol; pre-training MS-COCO, training: MSRA, fic-tuning real-world texture dataset; loss function: cross-catropy
backbone: resnet101; pre- training: MS- COCCO training MSRA: line-tuning real-world texture dataset, loss function:
backhome: resnet (01: pre Imiiiing:SS........ COCO; training: datasets; loss function: enimpy
backbonc: resnetlol; MostrrininggM--OOOC MS-COCO; training: datascts; function:
pre-trainin:: MS-COCO, traningg real-world function:
deeplab pre training: COCO: training:
HED
DNNoou))
Figure 11: Sample representative results Real-World Texture Dataset. Comparison with state-of-the-art deep learning methods.
23

---

Under review as a conference paper at ICLR 2021
images
groun trich
backbune: resmell01: pre training: MS COCO; training: DLTS; line tuinn real world texture dataset; loss Function: cruss entrupy
backbone: resnet101; rre-trinnnng: MS-COCO; training: DUTS; fine-tuning real world texture dataset; loss function: ours
backbonc: resneti01: pre-training MS MS-COCO; training: MSRA, nnc-tuning weal-world texture dataset; loss function: cross-catropy
backbone: resnet101; training: MS COCO; training MSRA: line-tuning real- texture dataset: loss function:
backhone: resnet OT: triminngMSS COCO, training: dalasets: function: enimpy
backbonc: 1osn pre-training MS-COCO; training: datasets; function:
resnet101; pre-traiing: MS-COCO, traniig: real-world
function:
backhone: deeplab v: pre training:
COCO: traningg dalasels;
HED
ST DNN(ous))
Figure 12: Sample representative results Real-World Texture Dataset. Comparison with state-of-the-art deep learning methods.
24

---

Under review as a conference paper at ICLR 2021
ground truth
SILD
STDDNN
Images
ground
STLD
ST-DNNN
Images
ground Truth
STLD
ST-DNNN
Figure 13: Sample representative results on Real-World Texture Dataset. We compare the ST- DNNs (ours) and STLD.
25

---

Under review as a conference paper at ICLR 2021
images
ground
SILD
ST-DNN
images
ground truth
STLD
ST-DNN
T
Images
ground truth
ST-DNN
T
Figure 14: Sample representative results on Synthetic Texture Dataset. We compare the ST- DNNs (ours) and STLD.
26

---

Under review as a conference paper at ICLR 2021
Dataset
Images Dataset
MSRA 10K Cheng et al.
(2011)
10000
DUTS-TR Zeng al.
2018)
10553
DUTS-TE Zeng al.
2018
5019
ECCSD Yan el al. (2013)
1000
MSRA-B Cheng et al. (2011)
5000
PASCAL-S Li al. 2014)
850
HKU-IS Li & Yu (2016)
4447
THUR15K Cheng et al. (2014)
Yan al.
2013)
15000
Table 6: Datasets used in training
Contour F-meas.
Region metrics
GT-cov:
Rand. Index
Var. Info.
ODS
OIS
ODS
OIS
ODS
ODS
OIS
ST-DNN (OUIS)
0.64
0.64
0.94
0.94
0.94
0.94
0.35
0.35
Siamese Khan & Sundaramoorthi
2018
0.65
0.65
0.92
0.92
0.92
0.92
0.43
0.43
STLD
0.58
0.58
0.86
0.86
0.88
0.88
0.63
0.63
meg Arbeláez el al.
2014
0,51
0.54
0.74
0.82
0.77
0.85
0.80
0.66
gPh Athelaez
0.78
Kok
Kokkinos
201
0.64
0.66
0.56
0.56
0,56
0.57
0.92
0.92
CB 500 ct 0L
201
0.54
0.56
0.75
0.80
0.79
0.84
0.81
0.76
SIFT
0.13
0.13
0.54
0.54
0.58
1,50
1,50
Entropy Hong çt al
2008
0.19
0.19
0.74
0.74
0.76
0.76
1.00
1.00
Hist-S
Chan. Vesc hain & Vese 2001
0.19
0.19
0.73
0.73
0.76
0.76
1.07
1.07
LAC Lankion Tannenhaum 2008
0.58
1,51
Globàl Michalllloo et 2007
0.14
0.14
0.66
0.66
0.68
0.68
1.16
1.16
Table 7: Results on Texture Segmentation Datasets. See Table measures.
5
caption for details on the
ADDITIONAL COMMENTS
Choice of Poisson Equation: We have chosen Poisson equation in ST-DNN because it is a linear PDE and can be efficiently solved, even for large images since the matrix in Au is symmetric positive definite
conjugate gradient algorithm to solve for u efficiently. However, other PDEs like Heat equation can also be used with our formulation.
Pre-processing The zeroth layer (pre-processing layer of our ST-DNN extracts color, grayscale and oriented gradient channels at multiple scales. We have used this design choice because oriented gradient are shown to be important for textural appearance of segments Khan et al. (2015): [Sifre& & Mallat (2033).
Case for Shape-Tailored Tailored descriptors: In Khan et al. (2015) the authors show the effect of aggregation of statistics across region boundaries. They show a marked improvement the performance of the descriptors by simply tailoring to the region of interest.
27

---

Under review as a conference paper at ICLR 2021
Contour
Region metries
F-meas. ODS
GT-cov. ODS
Rand. Index
Into.
OIS
OIS
ODS
OIS
ODS
OIS
ST-DNN (ours)
0.49
0.49
0.92
0.92
0.92
0.92
0.44
0.44
Siamese Khan & Sundaramoorthi 2018
0.45
0.45
0.90
0.90
0.89
0.89
0.46
0.46
Don STLD
0.18
0.18
0.84
0.84
0.84
0.84
0.65
0.65
gPh Arbelaezz
10
0.73
CB bolh et at 201
0.30
0.29
0.75
0.77
0.76
0.79
1.09
1,08
SIF1
0.11
0.11
0.70
0.70
0.70
0.70
1.07
1.77
Entropy Hone of 2008
0.13
0.13
0.75
0.75
0.75
0.91
Hist-5 Ni cc al 2013)
0.32
0.32
0.67
0.67
0.68
0.68
1.10
1.10
2000
LACC [ Lankfon Tannenbium 2008
0.14
0.14
0.72
0.72
0.70
0.70
1.14
1.14
Michailovich et
2007
0.28
0.75
0.75
0.79
Table 8: Results on Synthetic Texture Segmentation Dataset. See Table the measures.
capption for details on
28

---

Under review as a conference paper at ICLR 2021
Images
Groundtruth
Deeplaab
FCN-ResNettO1
ST.DNN
Images
GroundTrath
Desplab-v3
FCN Resnet 101
ST-DNN
Table 9: Results on and Synthetic multi-region Dataset and BSDS500. Comparisons are per- formed against oate-ooftthe-at deep learning based methods.
29


Reviews:
[{"review_id": "wFnqYpQ2NrH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a new \"shape-tailored\" convolutional layer for improving the accuracy of semantic segmentation. The shape-tailored layer is inspired by the Poisson partial differential equation which aggregate features from neighboring pixels through the linear combinations of partial derivatives of the output of PDEs. The paper proves several properties of the proposed ST-DNN and demonstrated good results in comparisons with previous semantic segmentation algorithms.\n\nPositive:\nThe paper has done good work in proving the properties of the proposed shape-tailored network and have described how to implement and train the proposed ST-DNN.\n\nNegative:\nThe experimental comparisons are quite limited which it has only compared with DeepLab-v3 and FCN-ResNet101. Both methods were published on or before 2018. The more recent state-of-the-art image segmentation methods (published in 2019&2020) are not compared. The experiments on texture segmentation, especially for the experiments on increasing deformation,  are also not very impressive as these datasets are mostly synthetic.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwvjnc_fSYK", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose shape-tailored deep neural networks which performs filtering according to Poisson PDE using convolutions and output robust descriptors for the task of texture segmentation. \n\nThe strengths:\n1. The relationship to Poisson PDE and its formulation as a convolution. \n2. Property of covariance illustrated through proof and experimentation\n3. Exact formulation and numerical approximations to the approach for faster training\n4. Extensive experiments on texture segmentation.\n\nWeakness:\n1. Focus on only texture segmentation. It will be valuable to show experments on background-foreground segmentation which is a similar task. \n2. Inadequate validation of the decriptors derived from ST-DNN. If the descriptors indeed are more valuable, this should be demonstrated using simple tasks like using the 'average' or 'aggregated' descriptor for linear classification tasks. This would be valuable to the representation learning community, but was not addressed in the experiemnts. \n3. ST-DNN is favourable with respect to data efficiency and parameter efficiency, but the sophisticated nature of the Poisson operator adds to the complexity. The cons in terms of flops for training and inference should be discussed. Also, the authors should clarify the limitations of the approach more clearly.\n\nPost rebuttal comment: Having read all the reviews and in light of the additional experiments, I am slightly raising my rating. The reason I am still not fully convinced is no experiments indicating the quality of descriptors learned by the method. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jibC4ZMaRbR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers presents shape-tailored deep neural networks (ST-DNN) and apply to the task of texture segmentation. ST-DNN are motivated by the prior work on shape-tailored descriptors (or smoothing) that aggregate image statistics within regions of the interest, and defined as a solutions to the Poisson PDE which balances image fidelity and smoothness. This paper applies this formulation to generalize convolutions from square-shared operations to arbitrary regions, and constructs a deep neural networks by stacking this smoothness operator with 1x1 convolutions and ReLU (ST-DNN layer), repeatedly. For texture segmentation task, ST-DNN takes the input data (input images channels, together with grayscale and oriented gradients computed at 5 scales) and initial segmentation mask, and applies several layers of ST-DNN layers to produce an updated segmentation mask. The network parameters are trained using a loss that is minimized when the descriptors within a segment are homogenous, and different from descriptors in other regions.\n\nStrengths\n+ Paper presents a novel deep network architecture using shape-tailored smoothing operations to aggregate data over arbitrary regions, thus, generalizing the convolution operation.\n+ Proposed formulation also enables the ST-DNN to be covariant to the in-plane rotation and translations as well as small deformations. Theoretical justifications as well as empirical validation are included (in the appendix).\n+ Proposed outperform other shape-tailored variants for texture segmentation as well as earlier works on texture segmentation.\n+ Proposed operation could potentially benefit other applications such as tracking or image segmentation and hence may be of relevance to broader vision community.\n\nConcerns\n- Input preprocessing for the segmentation task: The relevance of adding the grayscale, together with oriented gradients over numerous scales seems unclear. The compositional operation (simulated by stacking of multiple layers) should implicitly compute. An analysis of different feature space and how it impacts the performance seems to be missing. This would be important for the readers attempting to generalize the proposed method to other tasks.\n- Evaluation datasets: Similar to prior shape-tailored methods which were applied to benchmarks datasets of tracking and segmentation such as BSDS500, it would be helpful in the proposed method can be applied and evaluated on such benchmarks to make it easier for readers to assess the benefits of the proposed method.\n- Comparison with deep segmentation methods: From the texture segmentation, it seems a natural formulation would be use Siamese or triplet networks to facilitate homogeneity of within each region and discriminativeness across regions (as is done in [23], as well as other papers such as https://ieeexplore.ieee.org/document/8545348). How is the cross-entropy loss used to train the baselines? Is each texture region in the dataset a unique label? It would help better appreciate the poor segmentation performance of these baselines.\n- Model selection strategy: How are the best performing hyperparameters obtained? Table 3 shows that the performance changes significantly as the number of layers increase until 4 layers and over-fits as soon as 5th layer. Are the reported quantitative results on the testing dataset the most optimistic estimate, or were the parameters determined using a model-selection/validation dataset?\n- How well does the proposed method handle scales? With standard segmentation methods, the scales are handled by explicit down/up sampling layers. How does the consideration of oriented gradients at fewer scales impact the performance?\n\nOverall the paper presents an interesting formulation which could be useful to segmentation tasks, as well as tracking (as presented in [22]) but there are several concerns which need to be addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SYGZtgPuK8g", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\nThe paper suggests replacing convolutional layers with ST-DNNs. ST-DNNs, in contrast to conv. layers, can natively support non-rectangular neighborhoods (regions). Similarly to conv. layers, ST-DNNs can be stacked to increase expressivity. ST-DNNs themselves correspond to solutions to the Poisson PDE. The paper describes how these layers can be trained and used for prediction. Results are reported on a texture dataset.\n\nNovelty\nThe authors suggest ST-DNNs that natively support non-rectangular regions. ST-DNNs are natively co-variant to translations and rotations (as opposed to CNNs)\n\nImpact\nST-DNNs can outperform CNN-based networks in a texture segmentation setting when the data is scarce.\nClarity\nThe paper is exceptionally clear. The mathematical reasoning is sound and the notation helps to understand the relationship between the components. The appendix contains a plethora of additional information and is clearly structured.\n\nEvaluation\nThe authors performed experiments with regard to two texture datasets, (i) Real-World Texture Segmentation, and (ii) Synthetic Texture Dataset. Both datasets are comparatively small (<300 images)\n\nStrengths (Reasons to accept)\nMathematically speaking, the paper presents solid derivations that are easy to follow. The notations help to understand the meaning.\nIn general, the clarity of the plots, the diagrams and the algorithms are good \nThe description of the training and the prediction phase is extraordinarily clear and sensical.\n\n\nWeaknesses (Reasons to reject)\nThe choice of the Poisson PDE seems somewhat arbitrary. The argumentation why this choice was made seems to also apply in a much more general context to simpler approaches\nThe authors argue (Related work) that pre-existing segmentation approaches have difficulty in handling the larger number of classes typically found in texture segmentation tasks. This claim doesn’t seem warranted and a proper justification is missing.\nSection 3.2 mentions that images need inevitably to be downscaled to make the model fit into memory. This seems to be an achilles heel of the model that could have probably been addressed differently.\nThe evaluation of the paper is unnecessarily restricted to texture segmentation. It would have been interesting to test the system on classical segmentation problems that necessitate a more hierarchical segmentation approach (e.g. autonomous driving) as opposed to the very local nature of texture segmentation. If ST-DNNs underperform on mainstream segmentation tasks, this is valuable information to the community. \nMost of the proofs are straightforward (owed to the fact that some property is inherited from the Poisson PDE) and could have either been left out or abridged\nThe argument that neighborhoods ideally have to be non-rectangular is not entirely convincing. A practical toy example (simpler than texture segmentation) that showcases the (supposed) shortcomings of CNNs would be useful here.\nThe reliance on an initial mask is a limitation of ST-DNNs compared to conventional CNNs.\nAdmittedly, ST-DNNs seem to cope better with small dataset sizes than CNNs in the texture setting. The readers should be convinced that this property is important, perhaps through a high-impact example where data is scarce.\n\nFormatting\nThe authors did not follow the ICLR style for citations as described in Section 4 of the template, replacing named citations (e.g. Gou et al, 2017) with a numeric citation (12). This effectively gives the authors more space for content than competing submissions.\n\nQuestions and other comments to the authors\nWhich are the most compelling reasons to prefer Poisson PDEs over other PDEs or even simpler approaches?\n\n\n=============UPDATE==============\nI am generally satisfied with the answers provided by the reviewers and I have increased my score accordingly. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-W01Hp-lWBp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, I think this paper is well-written and easy to follow, although the English can be\nfurther improved. But the experiments part are weak for me.\n1. I see that the authors conducted two experiments on texture segmentation datasets where\nboth are very small datasets. Why does this happen? Small datasets usually can not tell us\nstatistic conclusions as random things always influence the results pretty much. Or is it because\nthat ST-DNN can only be applied or be useful on small datasets? If yes, this needs to be stated\nand clarified in the paper.\n2. The experiments are conducted on two texture segmentation datasets which are both very\nsmall with a few hundred images. In table I, the authors compare with more general baselines,\ni.e., Deeplab. However, Deeplab is proposed for large dataset which will experience overfiting on\nthe small datasets used by the authors. So I feel that Table 1 is not a very fair comparison and\nindicates less meaning to us in terms of the performance of ST-DNN compared to general\nsegmentation methods.\n3. If we fix the consideration inside texture segmentation, where the authors kindly compare\nwith a few state-of-the-art in Table 2. However, only two datasets are tested while I think &gt;4 are\nsufficient for a good quality publication.\nOverall, I think the motivation and novelty are good. However, the validation needs to be\nenhanced.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new kind of CNN that convolves on deformable regions and cooperates with the Poisson equation to determine the deformable regions. Experiments on texture segmentation look promising.

Pros:
1. The paper is well written and easy to follow.
2. The idea is interesting and the reviewers liked it. 
3. The experiments on texture segmentation are promising.

Cons:
1. Actually, convolution on non-rectangular region is not new, in contrast to the authors' claim and reviewers' belief, although the authors may argue that the mechanisms of determining the region for convolution are different and the CNNs are used for different tasks. See, e.g., 

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei: Deformable Convolutional Networks. ICCV 2017: 764-773

and other papers by the same first author. So the AC would discount the novelty of the paper.

2. As most of the reviewers commended, the experiments on texture segmentation were insufficient. Although extra experiments were added (thank the authors' effort on doing this), the reviewers actually still deemed that they were not convincing enough, e.g., should compare with more state-of-the-art methods.  Reviewers #2&#4 both confirmed this issue in confidential comments.

Although Reviewer #1 increased his/her score, the final average score is still below the threshold. So the AC decided to reject the paper.
Decision: Reject