Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Semantic segmentation using deep learning techniques becomes challenging if the input images are of high resolution. The image resolution can be lowered and the network learned, but downsampling based on uniform grid pattern on the high resolution image can lead to network not being able to learn semantic information in high frequency regions, i.e. the loss of information is uniform over the image. In order to alleviate this problem, the downsampling can be done based on a non-uniform grid on the high resolution image which is content adaptive where more samples are selected at the high frequency/important regions while low frequency regions are sampled sparsely. Previous work has computed this non-uniform downsampling in a deep learning framework as an independent task. This paper proposes to jointly learn the downsampling grid estimation task and the particular task of semantic segmentation. They show improved IoU metric over semantic segmentation results on standard benchmarks.
main_review: + The paper is well motivated based on detailed description of the limitations of the closest recent work.
+ The results are detailed and well explained.

- "Consider a relative coordinate system such that I[u, v] is the pixel value of I where u, v∈[0,1]." is not clear that why the coordinates need to be normalized between [0,1]
- The term b[i,j] in Eq.2 is not explained. Although its an equation from another paper, it would help to have some definition of this variable.
- Eq 3 needs to be explained a bit better. Also what is the kernel k in Eq.3 and how is it defined in the paper implementation.
summary_of_the_review: The paper shows that jointly doing content driven subsampling and the high level task (semantic segmentation here) can lead to better results rather than doing them independently. This idea may be propagated to other high level vision tasks which are limited by image size of training data. Thus, the reviewer is inclined favorably towards this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve the performance of segmenting ultra high-resolution images by replacing the commonly used downsampling functions with the learned deformation sampling module. The biggest difference of the proposed method is, instead of sampling all areas uniformly, the sampling density is estimated based on the contents. The experiment results on the Cityscape, DeepGlobe and PCa-Histo datasets show some improvements on mIoU and cost-performance trade-offs compared with uniform downsampling.
main_review: Strength:
- This paper proposes a new deformed sampling module that explicitly downscale the image into target resolution. The idea of making the network to learn the sampling density is very intuitive, and it is also adaptable to classical sampling functions.
- Besides, the author noticed the importance of regularization and added the edge loss in training. 
- The proposed method is end-to-end trainable, and can be plugged into different backbone networks.

Weakness:

Still, I have complex feeling about this paper: the motivation is really good: following all the previous works that try to improve the segmentation performance by allocating the computations to most important regions, the proposed module does make a lot of sense, and it's plug-and-play. But when I read the experiment results, I find that it is not the module itself that brings the most difference, but the joint loss. From Fig.5, it can be seen that adding the module trained with single loss is not as good as the uniform one at the most time. This implies that the most important contribution of this paper is the joint loss.

Though, if this paper wants to illustrate the importance of joint loss that avoids the trivial solution, there are still some missing points to support this claim:
- Not enough ablation studies. If applying the joint loss on other methods, how the performance would be influenced? 
- More analysis of why the single segmentation loss would not work. After reading this paper, I still do not understand: why single loss would lead to a trivial solution?

I read the appendix and find the downsampling module consists of 3 layers of CNN with 3x3 kernel. Since the other works in 2.1 all aim to cover multiple scale information, does it matter that this module only has this limited spatial window? And, Fig. 12 shows that the sampling density suddenly increases on the edge of the picture, even if the content is still solid-color. Why would this happen? Is this learned module vulnerable to noises in image?

Besides, since this work adopts a learned CNN module to predict the sampling density, and then conduct rescaling by interpolation, I wonder why not directly makes the network to directly downsample/super-resolve the input? Such ideas are applied in "Learning to Resize Images for Computer Vision Tasks". How does the proposed method compare with this paper?


Minor errors:
- In the caption of Fig. 5, the last line should note "(c) a local prostate cancer histology dataset...".
summary_of_the_review: Overall, this paper has good motivation and proposes an interesting method. But after reading the whole paper, I felt that it is not the proposed method, but the joint loss that contributes most to the performance improvement. Besides, this paper lacks of in-depth explanation of the performance drop when applying the single loss, and comparisons of several similar methods. So I would give score 5 for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method for ultra-high resolution image segmentation. It follows the learn to zoom approach to non-uniform downsample the original high resolution image. Furthermore, it adds edge-based loss to guide the downsample process. The upsampling step is also included in the training stage. The proposed method is evaluated on several public datasets and achieves promising results. Effectiveness of components are verified in the experiments. 
main_review: The strengths of the papers are:
1. It extends the learn to zoom approach for classification to segmentation tasks. A deformable module is applied for downsampling the original image.
2. The upsampling processing is incorporated in the training phase.
3. The edge information is also used to guild the downsampling step which improves the segmentation performance. 

The overall presentation of the method is great. I have some questions about the paper. 
1. The loss function consists of segmentation loss and edge loss. A weight term is used to balance these two terms. Is this the same problem as mentioned in Marin et. al (2009) in the barrier of the SOTA section? 
2. It would be better to add more text to generate the downsampled image (ex. R^{52x100}) from low-res (ex. R^{13x25}).

Missing references:
Kirillov, Alexander et. al., Pointrend: Image segmentation as rendering, CVPR 2020
Huynh, Chuong et. al., Progressive Semantic Segmentation, CVPR 2021
summary_of_the_review: The method mainly extends the learn to zoom approach for classification to segmentation tasks. Importantly, the edge loss is applied to guild the learning based downsample. The authors present extensive experiments and analysis to assess the components of the method. This method brings new ideas or approaches to handle high resolution images, especially in segmentation tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper address the semantic segmentation problem on high-resolution images. Existing methods uniformly downsample the original image to a small version to meet the memory requirement, while the uniform downsampling is suboptimal. The authors propose a deformed downsampling method in this paper and mainly compare it with the previous edge-based downsampling, showing better performance. Although the proposed method is somewhat similar to one previous method, while directly applying which on segmentation task does not improve the performance. This demonstrates the importance of the modifications proposed by the authors in this paper.  
main_review: Strengths:

1. The paper is well-written. The problem, motivation, idea, and method are clearly stated. Sufficient visualization is provided to make the paper easier to read.

2. Comprehensive experiments and analyses are conducted. The results on three publicly available datasets are reported.



Weaknesses:

1. As the authors said, the proposed method is similar to [Recasens et al. 2018], so the novelty of the proposed method is somewhat limited. However, the authors show that directly applying [Recasens et al. 2018] to segmentation tasks is not necessarily effective. Hence, I agree with the authors that the proposed method is somewhat novel. 

summary_of_the_review: The paper is well-argued, and the proposed method is effective. Novelty is somewhat limited but can meet the ICLR bar.
review_title: null

Meta-review: Overall, this paper receives positive reviews. The reviewers find the technical novelty and contributions are significant enough for acceptance at this conference. The authors' rebuttal helps address some issues. The area chair agrees with the reviewers and recommend it be accepted at this conference.
Decision: Accept (Poster)