Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose shape-tailored deep neural networks which performs filtering according to Poisson PDE using convolutions and output robust descriptors for the task of texture segmentation. 

The strengths:
1. The relationship to Poisson PDE and its formulation as a convolution. 
2. Property of covariance illustrated through proof and experimentation
3. Exact formulation and numerical approximations to the approach for faster training
4. Extensive experiments on texture segmentation.

Weakness:
1. Focus on only texture segmentation. It will be valuable to show experments on background-foreground segmentation which is a similar task. 
2. Inadequate validation of the decriptors derived from ST-DNN. If the descriptors indeed are more valuable, this should be demonstrated using simple tasks like using the 'average' or 'aggregated' descriptor for linear classification tasks. This would be valuable to the representation learning community, but was not addressed in the experiemnts. 
3. ST-DNN is favourable with respect to data efficiency and parameter efficiency, but the sophisticated nature of the Poisson operator adds to the complexity. The cons in terms of flops for training and inference should be discussed. Also, the authors should clarify the limitations of the approach more clearly.

Post rebuttal comment: Having read all the reviews and in light of the additional experiments, I am slightly raising my rating. The reason I am still not fully convinced is no experiments indicating the quality of descriptors learned by the method. 
review_title: Interesting method and good results on specific problem of texture segmentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This papers presents shape-tailored deep neural networks (ST-DNN) and apply to the task of texture segmentation. ST-DNN are motivated by the prior work on shape-tailored descriptors (or smoothing) that aggregate image statistics within regions of the interest, and defined as a solutions to the Poisson PDE which balances image fidelity and smoothness. This paper applies this formulation to generalize convolutions from square-shared operations to arbitrary regions, and constructs a deep neural networks by stacking this smoothness operator with 1x1 convolutions and ReLU (ST-DNN layer), repeatedly. For texture segmentation task, ST-DNN takes the input data (input images channels, together with grayscale and oriented gradients computed at 5 scales) and initial segmentation mask, and applies several layers of ST-DNN layers to produce an updated segmentation mask. The network parameters are trained using a loss that is minimized when the descriptors within a segment are homogenous, and different from descriptors in other regions.

Strengths
+ Paper presents a novel deep network architecture using shape-tailored smoothing operations to aggregate data over arbitrary regions, thus, generalizing the convolution operation.
+ Proposed formulation also enables the ST-DNN to be covariant to the in-plane rotation and translations as well as small deformations. Theoretical justifications as well as empirical validation are included (in the appendix).
+ Proposed outperform other shape-tailored variants for texture segmentation as well as earlier works on texture segmentation.
+ Proposed operation could potentially benefit other applications such as tracking or image segmentation and hence may be of relevance to broader vision community.

Concerns
- Input preprocessing for the segmentation task: The relevance of adding the grayscale, together with oriented gradients over numerous scales seems unclear. The compositional operation (simulated by stacking of multiple layers) should implicitly compute. An analysis of different feature space and how it impacts the performance seems to be missing. This would be important for the readers attempting to generalize the proposed method to other tasks.
- Evaluation datasets: Similar to prior shape-tailored methods which were applied to benchmarks datasets of tracking and segmentation such as BSDS500, it would be helpful in the proposed method can be applied and evaluated on such benchmarks to make it easier for readers to assess the benefits of the proposed method.
- Comparison with deep segmentation methods: From the texture segmentation, it seems a natural formulation would be use Siamese or triplet networks to facilitate homogeneity of within each region and discriminativeness across regions (as is done in [23], as well as other papers such as https://ieeexplore.ieee.org/document/8545348). How is the cross-entropy loss used to train the baselines? Is each texture region in the dataset a unique label? It would help better appreciate the poor segmentation performance of these baselines.
- Model selection strategy: How are the best performing hyperparameters obtained? Table 3 shows that the performance changes significantly as the number of layers increase until 4 layers and over-fits as soon as 5th layer. Are the reported quantitative results on the testing dataset the most optimistic estimate, or were the parameters determined using a model-selection/validation dataset?
- How well does the proposed method handle scales? With standard segmentation methods, the scales are handled by explicit down/up sampling layers. How does the consideration of oriented gradients at fewer scales impact the performance?

Overall the paper presents an interesting formulation which could be useful to segmentation tasks, as well as tracking (as presented in [22]) but there are several concerns which need to be addressed.
review_title: A novel architecture using aggregation operator over arbitrary regions but concerns on evaluation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: In general, I think this paper is well-written and easy to follow, although the English can be
further improved. But the experiments part are weak for me.
1. I see that the authors conducted two experiments on texture segmentation datasets where
both are very small datasets. Why does this happen? Small datasets usually can not tell us
statistic conclusions as random things always influence the results pretty much. Or is it because
that ST-DNN can only be applied or be useful on small datasets? If yes, this needs to be stated
and clarified in the paper.
2. The experiments are conducted on two texture segmentation datasets which are both very
small with a few hundred images. In table I, the authors compare with more general baselines,
i.e., Deeplab. However, Deeplab is proposed for large dataset which will experience overfiting on
the small datasets used by the authors. So I feel that Table 1 is not a very fair comparison and
indicates less meaning to us in terms of the performance of ST-DNN compared to general
segmentation methods.
3. If we fix the consideration inside texture segmentation, where the authors kindly compare
with a few state-of-the-art in Table 2. However, only two datasets are tested while I think &gt;4 are
sufficient for a good quality publication.
Overall, I think the motivation and novelty are good. However, the validation needs to be
enhanced.
review_title: The paper proposed a new DNN - ST-DNN for segmentation with PDE. Compared with traditional CNN where fixed shape regions are described, it can describe arbitrary shape regions. Experiments are conducted on texture segmentation problem, showing good results and robustness to translation, rotation and domain deformation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary
The paper suggests replacing convolutional layers with ST-DNNs. ST-DNNs, in contrast to conv. layers, can natively support non-rectangular neighborhoods (regions). Similarly to conv. layers, ST-DNNs can be stacked to increase expressivity. ST-DNNs themselves correspond to solutions to the Poisson PDE. The paper describes how these layers can be trained and used for prediction. Results are reported on a texture dataset.

Novelty
The authors suggest ST-DNNs that natively support non-rectangular regions. ST-DNNs are natively co-variant to translations and rotations (as opposed to CNNs)

Impact
ST-DNNs can outperform CNN-based networks in a texture segmentation setting when the data is scarce.
Clarity
The paper is exceptionally clear. The mathematical reasoning is sound and the notation helps to understand the relationship between the components. The appendix contains a plethora of additional information and is clearly structured.

Evaluation
The authors performed experiments with regard to two texture datasets, (i) Real-World Texture Segmentation, and (ii) Synthetic Texture Dataset. Both datasets are comparatively small (<300 images)

Strengths (Reasons to accept)
Mathematically speaking, the paper presents solid derivations that are easy to follow. The notations help to understand the meaning.
In general, the clarity of the plots, the diagrams and the algorithms are good 
The description of the training and the prediction phase is extraordinarily clear and sensical.


Weaknesses (Reasons to reject)
The choice of the Poisson PDE seems somewhat arbitrary. The argumentation why this choice was made seems to also apply in a much more general context to simpler approaches
The authors argue (Related work) that pre-existing segmentation approaches have difficulty in handling the larger number of classes typically found in texture segmentation tasks. This claim doesn’t seem warranted and a proper justification is missing.
Section 3.2 mentions that images need inevitably to be downscaled to make the model fit into memory. This seems to be an achilles heel of the model that could have probably been addressed differently.
The evaluation of the paper is unnecessarily restricted to texture segmentation. It would have been interesting to test the system on classical segmentation problems that necessitate a more hierarchical segmentation approach (e.g. autonomous driving) as opposed to the very local nature of texture segmentation. If ST-DNNs underperform on mainstream segmentation tasks, this is valuable information to the community. 
Most of the proofs are straightforward (owed to the fact that some property is inherited from the Poisson PDE) and could have either been left out or abridged
The argument that neighborhoods ideally have to be non-rectangular is not entirely convincing. A practical toy example (simpler than texture segmentation) that showcases the (supposed) shortcomings of CNNs would be useful here.
The reliance on an initial mask is a limitation of ST-DNNs compared to conventional CNNs.
Admittedly, ST-DNNs seem to cope better with small dataset sizes than CNNs in the texture setting. The readers should be convinced that this property is important, perhaps through a high-impact example where data is scarce.

Formatting
The authors did not follow the ICLR style for citations as described in Section 4 of the template, replacing named citations (e.g. Gou et al, 2017) with a numeric citation (12). This effectively gives the authors more space for content than competing submissions.

Questions and other comments to the authors
Which are the most compelling reasons to prefer Poisson PDEs over other PDEs or even simpler approaches?


=============UPDATE==============
I am generally satisfied with the answers provided by the reviewers and I have increased my score accordingly. 

review_title: A clearly presented framework for shape-tailored networks, but needs better motivation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a new "shape-tailored" convolutional layer for improving the accuracy of semantic segmentation. The shape-tailored layer is inspired by the Poisson partial differential equation which aggregate features from neighboring pixels through the linear combinations of partial derivatives of the output of PDEs. The paper proves several properties of the proposed ST-DNN and demonstrated good results in comparisons with previous semantic segmentation algorithms.

Positive:
The paper has done good work in proving the properties of the proposed shape-tailored network and have described how to implement and train the proposed ST-DNN.

Negative:
The experimental comparisons are quite limited which it has only compared with DeepLab-v3 and FCN-ResNet101. Both methods were published on or before 2018. The more recent state-of-the-art image segmentation methods (published in 2019&2020) are not compared. The experiments on texture segmentation, especially for the experiments on increasing deformation,  are also not very impressive as these datasets are mostly synthetic.
review_title: A paper with theoretical support and some good results

Meta-review: This paper proposes a new kind of CNN that convolves on deformable regions and cooperates with the Poisson equation to determine the deformable regions. Experiments on texture segmentation look promising.

Pros:
1. The paper is well written and easy to follow.
2. The idea is interesting and the reviewers liked it. 
3. The experiments on texture segmentation are promising.

Cons:
1. Actually, convolution on non-rectangular region is not new, in contrast to the authors' claim and reviewers' belief, although the authors may argue that the mechanisms of determining the region for convolution are different and the CNNs are used for different tasks. See, e.g., 

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei: Deformable Convolutional Networks. ICCV 2017: 764-773

and other papers by the same first author. So the AC would discount the novelty of the paper.

2. As most of the reviewers commended, the experiments on texture segmentation were insufficient. Although extra experiments were added (thank the authors' effort on doing this), the reviewers actually still deemed that they were not convincing enough, e.g., should compare with more state-of-the-art methods.  Reviewers #2&#4 both confirmed this issue in confidential comments.

Although Reviewer #1 increased his/her score, the final average score is still below the threshold. So the AC decided to reject the paper.
Decision: Reject