Learning to Downsample for Segmentation of Ultra-High Resolution Images
Authors: Chen Jin
Ryutaro Tanno
Thomy Mertzanidou
Eleftheria Panagiotaki
Daniel C. Alexander
Venue: ICLR 2022 Poster (2022)
Keywords: ultra-high resolution image segmentation
non-uniform dowmsampling
efficient segmentation
large volume image segmentation
medical image segmentation

Abstract: Many computer vision systems require low-cost segmentation algorithms based on deep learning, either because of the enormous size of input images or limited computational budget. Common solutions uniformly downsample the input images to meet memory constraints, assuming all pixels are equally informative. In this work, we demonstrate that this assumption can harm the segmentation performance
because the segmentation difficulty varies spatially (see Figure 1 “Uniform”). We combat this problem by introducing a learnable downsampling module, which can be optimised together with the given segmentation model in an end-to-end fashion. We formulate the problem of training such downsampling module as optimisation of sampling density distributions over the input images given their low-resolution views. To defend against degenerate solutions (e.g. over-sampling trivial regions like the backgrounds), we propose a regularisation term that encourages the sampling locations to concentrate around the object boundaries. We find the downsampling
module learns to sample more densely at difficult locations, thereby improving the segmentation performance (see Figure 1 "Ours"). Our experiments on benchmarks of high-resolution street view, aerial and medical images demonstrate substantial improvements in terms of efficiency-and-accuracy trade-off compared to both uniform downsampling and two recent advanced downsampling techniques.

Published as a conference paper at ICLR 2022
LEARNING TO DOWNSAMPLE FOR SEGMENTATION OF ULTRA-HIGH RESOLUTION IMAGES
Jin', Ryutaro Tanno?, Thomy Mertzanidou' Ellffthria
, Panagiotakil Daniel C. Alexander
Centre for Medical Image Computing. Department of Computer Science, University College London 2 Healthcare Intelligence, Microsoft Research Cambridge (chen. jin, t. mertzanidouu c. panagiotaki, d. alexander @ucclaae ac. uk tytanncemicrosoft. com
ABSTRACT
Many computer vision systems require low-cost segmentation algorithms based on deep learning, either because of the enormous size of input images or limited computational budget. Common solutions uniformly downsample the input images to meet memory constraints, assuming all pixels are equally informative. In this work, we demonstrate that this assumption can harm the segmentation performance because the segmentation difficultyy varies spatially (see Figure "Uniform"). We combat this problem by introducing a learnable downsampling module, which can optimised together with the given segmentation model in an end-to-end fashion. We formulate the problem of training such downsampling module optimisation of sampling density distributions over the input images given their low low-resolution views. To defend against degenerate solutions (e.g. over-sampling trivial regions like the backgrounds), we propose a regularisation term that encourages the sampling locations to concentrate around the object boundaries. We find the downsampling module learns to sample more densely at difficult locations, thereby improving the segmentation performance (see Figure "Ours"). Our experiments on benchmarks of high-resolution street view, aerial and medical images demonstrate substantial improvements in terms of efficiency-am and-currayyy trade-off compared both uniform downsampling and two recent advanced downsampling techniques.
INTRODUCTION
Many computer vision applications such as auto-piloting geospatial analysis and medical image processing rely semantic segmentation of ultra-high resolution images. Exemplary applications include urban scene analysis with camera array images (> 25000 x 14000 pixels) et al., 2020). geospatial analysis with satellite images (> 5000 x 5000 pixels) (Maggiori al., 2017) and histopathological analysis with whole slide images (> 10,000 x 10,000 pixels) (Srinidhi et al., 2019). Computational challenges arise when applying deep learning segmentation techniques on those ultra-high resolution images.
To speed up the performance, meet memory requirements or reduce data transmission latency, standard pipelines often employ preprocessing step that uniformly downsamples both input images and labels, and train the segmentation model at lower resolutions. However, such uniform downsampling operates under an unrealistic assumption that all pixels are equally important, which can lead to under-sampling of salient regions in input images and consequently compromise the segmentation accuracy trained models. Marin et al. (2019) recently argued that better downsampling scheme should sample pixels more densely near object boundaries, and introduced a strategy that adapts the sampling locations based on the output of separate edge-detection model. Such "dgge-based" downsampling technique has achieved the state-of-the-ar performance in low-cost segmentation benchmarks. However, this method is not optimised directly maximise the downstream segmentation accuracy; importantly, the pixels near the object boundaries are not necessarily the most informative about their semantic identities; for instance, textual cues indicating forest or cancer regions are less dependent on or cell boundaries, respectively.
Video demos available https: pestts://xxxssee http///xxxggxgtttih/a do//earn-dowwe eern-doonsampll e, gi thub. 1

---

Published as a conference paper at ICLR 2022
11505 X 2000
230 x 40
Ours
Ground Truth
Unitorm
Unlabelled
Mild cancer
Severe cancer
Suspicious
Figure 1: Semantic segmentation with learnable downsampling operation on cancer histology images. Top row: Our deformation module learns to adapt the sampling locations spatially (blue dots on the images) according to their utility for the downstream segmentation task and produces deformed downsampled image (bottom-left corner), which leads more accurate low-resolution segmentation. Middle row; high-resolution segmentation reversed from low-resolution predictions. Bottom row: Compare to uniform downsampling (referred to as "Uniform") "), our deformed downsampling samples more densely at difficult regions (yellow arrows) and ignore image contents that are less important (green arrows). We also demonstrate our method on the street and aerial benchmarks (video and code link).
We begin this work with an experiment to investigate the limitations of the commonly employed uniform downsampling, and the current state-of-the-art technique, namely the "edge-based" downsampling (Marin et al., 2019). Motivated by the undesirable properties of these approaches, we then introduce deformation module, a learnable downsampling operation, which be optimised together with the given segmentation model in an end-to fashion. Figure 2 provides an overview of the proposed method. The deformation module downsamples the high-resolution image over non-uniform grid and generates deformed low-resolution image, leading to its name. Moreover, to defend against degenerate solutions (e.g. over-sampling input images and labels trivial regions propose a regularisation term that encourages the sampling locations to
like the backgrounds), concentrate around the object boundaries. We demonstrate the general utility of our approach on three datasets from different domains-Cityscane (Cordts et al., 2016) and DeepGlobe (Demir et al., 2018)) and one medical imaging dataset-a where we observe consistent improvements in the segmentation trade-off.
performance and the
METHODS
In this section, we first formulate the basic components associated with this study in Section 2. We then perform motivation experiment in Section 2.2 to illustrate the impact of manual tuned objective of the basgee" downsampling approach (Marin al., 2019), particularly the sampling density around edge, segmentation performance and its spatial variation across the image. Motivated by the finding, we propose the deformation module, the learnable downsampling module for segmentation, in Section 2.3. To ease the joint optimisation challenge of oversampling at the trivial locations, regularisation term is proposed in Section 2.4.
2.1 PROBLEM FORMULATION
Sampling method. Let X #xXCC be high-resolution image of arbitrary size H.W.C. A sampler G takes X input and computes downsampled image X = =(X), where X RixuxC
2

---

Published as a conference paper at ICLR 2022
High-Resolution Image Low-Res Verson
Deformed Sampler G,(X. (,(X.D((Xx))
Downsamplod Image gg
Low-Ros Prediection
Non-Uniform Upsampling I[G (5, (X)) gullan
2612-100
X₁, g
S.(X) k
S Segmentation Module
Deformation Module
Figure 2: Architecture schematic. For each high- high-reeolution image X x we compute its lower reso- lution version X R Rha x The deformation module, Do. parametrised by 0 takes the low-resolution image X, as input and generates deformation map d = (x(X,,), where d R Kwya that predicts the sampling density at each pixel location. Next, the deformed sampler Ga is constructed by taking both X and as input and computes the downsampled image X G. d)) where X E EgxxxxCC and sampling locations shown as red dots masked on the image. The downsampled image X is then fed into the segmentation network to estimate the corresponding segmentation probabilities P = S. (X). During training the label Y is downsampled with the same deformed sampler to get Ga ,d) and [0.6] are jointly optimised by minimise the segmentation specific loss C. (0, Y) = (0, upsampled to the original space through deterministic reverse sampler G () and interpolation function II()).
P. Y). )- At inference time, the low-resolution prediction S. (X) is non-uniformly
Consider a relative coordinate system' such that X(u, v) is the pixel value of X where u, V E [0,1]. And an absolute coordinate system such that X[i is the pixel value of X at coordinates (i. for ie (1,2....h..je (1.2.....) Essentially, the sampler G computes mapping between (i. and (u, v).
Practically, sampler contains two functions {g".g* such that:
x (.).)
(1)
Segmentation and non-uniform upsampling. In this work we discuss model agnostic downsampling and upsampling methods, therefore any existing segmentation model can be applied. Here we denote the segmentation network Sg, parameterised by o, that takes an input a downsampled image X and makes a prediction P= (X) in the low-resolution space. During training label is downsampled by the same sampler G to get Y = G(Y) to optimise the segmentation network S& by minimising the segmentation specific loss C. (o:P, Ý)) At testing, the upsampling process consists of reverse sampler G-!() that reverse mapping each pixel coordinates (i. f) from the sampled image X back to coordinates (u, v) in the high- high-resolution domain. Then an interpolation function II() is applied to calculate the missing pixels to get the final prediction P = interpolation used as II() in this work.
)
The nearest neighbour
Disentangle the intrinsic u upsamplingerree error. test time, typical evaluation of downsampled segmen- tation with Intersection over Union (IoU) performed prediction P and the label Y. Hence ToU Y) incorporated both segmentation error and upsampling error. The upsampling error is caused by distorted downsampling, hence to improve the interpretability of the jointly learned downsampler, we propose disentangle the intrinsic upsampling error from the segmentation error by assuming a perfect segmentation at the downsampled space and calculating the error introduced after upsampling. In specific we calculate IoU (o Y). where Y' III (G((Y)) indicting the same non-uniform downsampling and upsampling processes applied to the label Y.
upsampling, between the final
2.2 MOTIVATIONAL STUDY: INVESTIGATING THE BARRIER OF THE SOTA
The based" approach (Marin et al., 2019), separately train a non- non-uniform down- Ge minimizing two competing energy terms: "sampling distance to semantic boundaries" and "sampling distance to uniform sampling locations", as the first and second term in equation 2, where b(ū is the spatial coordinates of the closest pixel on the semantic boundary. temperature term X used to balance the two energies, whose value is empirically recommended to I by Marin et al. (2019) and decided the sampling density around the edge.
A relative instcad of absolute coordinate system is selected for sampling he calculated in continues space. The "uniform" approach will have sampler G., ((0())))=--))/----)alll
1)).
3

---

Published as a conference paper at ICLR 2022
Hext pecipnllannll www.et ofok
t0 30
Goord Trush
loceesing Seripiig Dasily Artend
Tatls
Mexyi
Cerressry Sempting Diverty
(a)
betych aepinng dua bendd
shogle
(c)
Figure 3: Optimal sampling density varies over locations. We simulating a set "edge-based" samplers each at different sampling density around edge, and evaluate class-wise performance in (a) LoU YY) and (b) LoU. Best performed sampling density varies for each class as in (a) and (b). Such variation also been observed visually in (c), where predictions with various sampling density for example classes illustrated. Sampling locations masked red dots and hest performed prediction is highlighted. Motivational experiments performed on binary segmentation on 1 1/10th subset of Cityscapes (Cordts et al., 2016) (excludes under-represented classes) with downsampled size of 64 x 128 given input of 1024 x 2048.
- LJ
+A Ge(. --''----'E
(2)
The first part of our work performs empirical analysis to investigate a key question: "How does the empirically tuned temperature term À affect the segmentation performance?", We hence perform binary segmentation the Cityscapes dataset (Cordts et al., 2016). We generate set of simulated "edge-based" samplers each at different sampling densities around the edge (i.e. A) and evaluate by either directly calculating ToU (Y Y) given each fixed simulated sampler (Figure 3 (a)) or train segmentation network with each fixed simulated sampler and evaluates its accuracy by ToU (Figure (b)). The results show there is no "one-size-fin fits-allll sampling density configuration that leads to the best performance for all individual classes, neither for intrinsic upsampling error (IoU Y) as in Figure 3 (a))) nor segmentation error (IoU as Figure 3 (b)), which is also verified visually in Figure (c). These observations imply that the SOTA "edge-based" sampling scheme with pre-set sampling density around the edge is sub-optimal, highlighting the potential benefits of more intelligent strategy that can adapt sampling strategy to informative regions according to local patterns.
2.3 DEFORMATION MODULE: LEARN NON-UNIFORM DOWNSAMPLING FOR SEGMENTATION
Motivated by the above findings, we introduce the deformation module, data-driven sampling method that adapts sampling density at each location according to its importance to the downstream segmentation task. Figure 2 provides schematic of the proposed method.
Following earlier definitions in equation 1, to construct the deformed sampler Ga, we need to define the two sampling functions {gh) g who inputs the downsampled coordinates (i. in x and the learned deformation map d calculate the sampling location (u, v) in X, such that u[i.j] j...). jii.j] denser miihh-importancc regions correspond to the salient regions the deformation map A naive approach is treating d sampling probability distribution and performing the stochastic sampling. However one has to estimate gradients through the reparameterisation trick and traverse stochastic
j. The idea is to construct the sampling functions to sample X
The simulation details of the "edge-based" samplers deseribed in Appendix A.5.
4

---

Published as a conference paper at ICLR 2022
sampling over all locations is not computationally efficient. Inspired by Recasens et al. (2018) we consider the sampling location of each low-resolution pixel (i.j) is pulled by each surrounding pixel (i,j") by an attractive force, and the final sampling coordinates of (i, j) can then be considered as a weighted average of the coordinates of each surrounding pixel (i' (i.))) The weight at (i .j is defined to be: 1) proportional to the sampling density d(i .J ); 2) degrade away from the centre pixel j) by a factor defined by distance kernel ((tt)))(('' .J ) and 3) only applies within certain distance. Practically the distance kernel is fixed Gaussian, with a given standard deviation a and square shape of size 2o 1, and the sampling coordinates of (i.j) can then be calculated in the convolutional form by the two deformed sampler functions {g2-ga} defined as equation 3. The standard deviation a decided the distance the attraction foree can act on. While the magnitude of the attraction force is learned through the deformation map. We perform ablation studies to investigate the impact of a in Section A.
(i...d)) En S(i (L.D E J (L.).
sa(i.idd) - Ex S C'
(3)
Er I t'K (C.))
This formulation holds certain desirable properties that fit our goal: 1) it computes sampling locations proportional to the learned deformation map that indicting varying sampling density at different different regions, the need of which was illustrated in the motivational studies in Section 2.2; 2) the fixed Gaussian distance kernel can naturally fit CNN architecture, preserve differentiability and be efficient to calculate. This way the sampling density at each location is optimised by segmentation performance rather than sep- arately trained based on manual designed objective as the "edge-based" approach (Marin et al., 2019)).
2.4 REGULARISATION
THE JOINT-TRAINED SAMPLING
Our method is similar to Recasens et al. (2018), which however is only explored for image classification tasks. Transferring the method proposed in Recasens et al. (2018) to segmentation not only needs reformulating the problem as described in Section 2.1, optimising the joint at the levell not trivial. Given the segmentation loss (B. Y) (8, xXX, Y) is calculated in the row-uusoluiion space for minimal computation. The downsampling parameters 0 can be optimised to trivial solutions which encourage oversampling both input image X and label Y at easy locations like the background to reduce loss, which contradicts our goal. We experimentally verified the naive adaptation of the method from Recasens et al. (2018) segmentation (red bars in Figure 4) does not perform satisfactionally. To discourage the network learning trivial solutions, we propose regularisation edge loss by encouraging the deformation map similar to a simulated target which leads to denser sampling around the object boundaries, inspired by Marin et al. (2019) that object edge may be informative for segmentation. Different to the approach of Marin et al. (2019) we don't 't manually tune optimal target deformation map but treat it as a regularisation term to carry its main message that edge information is useful, then let the joint training system adapts optimal sampling location according to the segmentation objective.
In specific, we calculate the target deformation map d, = fedge yous ))+ from the uniformly downsampled segmentation label Y,, E Xhaxxaxx x1 which is at the same size as the deformation map d. The regularisation edge loss is defined as L. (irXir, Y,,) = (...ssddddd)).... which calculates the mean squared error (MSE) between the predicted deformation map mapd= d (X(r.)) and the target deformation map de. To this end, we jointly optimise the parameters {6.Φ} corresponding to the deformation module and the segmentation module respectively by the single learning objective as equation 4, where the first term is the segmentation specific loss (e.g. cross entropy + I.2 weight-decay) and the second term is the edge loss. We add weight term 7 for both losses that have comparable magnitudes. We note different to X in cquation 2, y can better adapt sampling location to difficult regions because it balancing the edge-loss and segmentation loss (end task loss) in an end-to-end setting, while À is balancing the two manual designed sampling targets in separately trained downsampling network. We also evaluate the impact of y in Section A.I.
Yir)
A 'Sedge() (.) is an edge detection filter by convolution of a specific 3 x kernel [[-1, -1.-1111-11111. 1-1,1 -1 I] and fgaus(s (-) is Gaussian hlur with standard deviation 5= 1 to encourage sampling close to the edge. To avoid the edge filter pickup the image boarder, we padding the border values prior applying the edge filter.
5

---

Published as a conference paper at ICLR 2022
3 RELATED WORKS
Convolutions with fixed kernel size restricted information sampled ("visible") for the network. Various multi-scale architectures have shown the importance of sampling multiple-scale information, either at the input patches (He et al., 2017; Jin et al., 2020) or feature level (Chen et al., 2014; 2016; Hariharan et al., 2015), but less efficient because of multiple re-scaling or inference steps. Dilated convolution (Yu & Koltun, 2015) shift sampling locations of the convolutional kernel at multiple distances to collect multiscale information therefore avoided inefficient re- ssalllinn.. Such shifting however limited to fixed geometric structures of the kernel. Deformable convolutional networks (DCNs) (Dai et al., 2017) learn sampling offsets to augment each sampling location of the convolutional kernel. Although DCNs share similarities with our approach, they are complementary to ours because we focus on learning optimal image downsampling as pre-processing so that keeps computation to a minimum at segmentation and a flexible approach can be plug-in to existing segmentation networks.
Learned sampling methods have been developed for image classification, arguing better image-level prediction can be achieved by an improved sampling while keeping computation costs low. Spatial Transformer Networks (STN) (Jaderberg et al., 2015) introduce a layer that estimates a parametrized affine, projective and splines transformation from input image to recover data distortions and thereby improve image classification accuracy. Recasens et al. (2018) proposed to jointly learn a baased network and "zoom-in" to salient regions when downsampling
input image for classification. Talebi & Milanfar (2021) jointly optimise pixel value interpolated (i.e. super-resolve) each fixed downsampling location for classification. However, an end-to-end trained downsampling network has not been proposed so far for per-pixel segmentation. Joint optimising the downsampling network for segmentation more challenging than the per-image classification, as we experimentally verified, due to potential oversampling at trivial locations and hence we propose to simulate sampling target to regularise the training.
On learning the sampling for image segmentation, post- post-processing approaches (Kirillov et al., 2020; Huynh et al., 2021) refining samplings at multi-stage segmentation outputs are complementary to ours, but we focus on optimising sampling at inputs so that keeps computation to a minimum. Non-uniform grid representation modifying the entire input images to alleviate the memory cost, such as meshes (Gkioxari et al., 2019), signed distance functions (Mescheder et al., 2019), and octrees (Tatarchenko et al., 20. However none of those approaches is optimised specifically for segmentation. Recently Marin et al. (2019) proposed to separately train an "edge-based" downsampling network, to encourage denser sampling around object edges, hence improving segmentation performance at low computational cost and achieving state-of-the-art However, the basee- approach is sub-optimal with sampling density distribution fixed to manual designed objective, the distance to edge, rather than segmentation performance. We verify this point empirically in Section 2.2 and hence propose our jointly trained downsampling for segmentation.
4 EXPERIMENTS AND RESULTS
In this section, we evaluate the performance of our deformed downsampling approach on three datasets from different domains as summarised in Table 1, against three basclines. We evaluate the segmentation performance with LoU and verify the quality of the sampler by looking at the intrinsic upsampling error with IYU(Y', Y). We show quantitative and qualitative results in Section 4.1. We further compare with reported results from Marin et al. (2019) in Section 4.2 with additionally experiments performed under the same preprocessing with the square central crop of input images.
Model and training. The deformation module defined as a small CNN architecture comprised of 3 convolution layers. The segmentation network is defined as deep CNN architecture, with HRNe1V2-WAR (Sun et al., 2019) used in all datasets, and PSP- net (Zhao et al., 2017) as a sanity check in the Cityscapes dataset. We employ random initialisation like as in He al. (2015), the same training scheme (Adam (Kingma & Ba, 2014), the focal loss (Lin et al., 2017) as the segmentation loss and MSE loss for the eoge-loss with full details in Appendix A.7) unless otherwise stated.
Baselines. We compare two versions of our method either with single segmentation loss -OuuseSingle loss") or adding the edge-loss as equation 4 ("Ours-Joint loss") against three baselines: 1) the "uniform"

---

Published as a conference paper at ICLR 2022
downsampling; 2) the "edge-based" and 3) "interpolation", the jointly learned method for pixel value interpolation at fixed uniform downsampling locations (Talebi & Milanfar, 2021), as an additional performance reference in Section 4.2, whose implementation details are given in Section A.2.
Table I: Dataset summary. More details in Appendix A.6
Dataset
Content
Resolution (pixels)
Number of Classes
Cityseape (Cordts et al., 2016) DeepGlobe (Demir et al., 2018) PCa-Histo (local)
Urban seenes
2048 x 1024
19
Acrial scenes
2448 x 2448
6
Histopathological
1968= 9392
6
4.1 QUANTITATIVE AND QUALITATIVE PERFORMANCE ANALYSIS
We plot mIoU and mJoU(YY, Y) in Figure 4 comparing our method against "uniform" and "edge-based" baselines on all three datasets. Figure 5 further investigate ToU at the per-class level, aiming understand the impact of different object sizes reflected by different class frequencies. We look at the performance along with a set of different downsampling sizes in Figure 6, which is key matrix indicating whether downsampling strategy can enable better trade-off between performance and computational cost.
In Figure 4, "Ours- "Ours-Single loss", as a basic version of our proposed method with single segmentation loss, performs better than the "uniform" " baseline on 2/3 tested datasets. Adding the proposed edge loss, "Ours-Joint loss" consistently performs best on all datasets with 3% 10% higher absolute m ooU over the "uniform" baseline. The performance improvement from "Ours-Single loss" to "Ours-Joint loss" represents the contribution of the edge loss within the proposed joint training framework. The variation of improvements over datasets suggests that the informativeness of edge is data-dependent, hence an end-to- mnd-to-end system is needed better adapt sampling to content. Besides, the segmentation error (mloli (Y', Y)) does not always agree with the upsampling error (m/oU), indicting separately trained sampler with manual designed sampling objective may fail.
Chyscaper (m6c1))
Reel
Kfo Sugletar
(8)
(b)
(c)
Figure 4: Comparing mloU and mioU(Y', Y) of our joint trained downsampler, either with single segmentation loss ("Ours-Single loss") or additional edge loss ("Ours-Joint loss"), versus two haseline downsampling methods on three datasets.
At class-wise, as in Figure 5, "Ours-Joint loss" performs to to 25% better in absolute IoU over baselines. Results also suggest the joint trained method can further generalise eddge-based" sampling for better segmentation performance. (Ours-Joint loss" not only improves the performances over all the low- frequency classes representing small objects, making use of the edge information as the rdge-based! baseline does, but also further generalise such improvements in the high-frequency classes representing large objects, where badee- approach has been found difficult to improve over "uniform".
Good downsampling can enable a better trade-off between performance and computational cost. In Figure 6, our method shows can consistently save up to 90% of computational cost than the "uniform" baseline across a set of different downsampling sizes (details are given in Appendix Table 8) over all three datasets. We notice mIoU does not increase monotonically on DeepGlobe dataset (Demir et al., 2018) with increased cost in Figure 6 (b), this indicates high resolution is not always preferable but a data-dependent optimal tradeoff exists and an and-to-end adjustable downsampler is needed. The mIo YY) results show our method can adjust sampling priority depending on the
'Note this is an approximation of the method from Marin et al. (2019), which is the best- rest-performed result from set segmentation networks each trained with a simulated "edge-based" samplles represents using different X, full details given in Appendix A.5.
7

---

Published as a conference paper at ICLR 2022
(kovtungldd menuuughddii 300-300)
Wi oe Prcpeety
Figure 5: wliss IoU three datasets. IoU gain indicates improvements from method over "uniform" baseline. Classes are ordered with increasing pixel frequency which also indicated in each plot.
available computational resource, by compromising the upsampling error but focusing on improving segmentation at the low-cost end as in Figure 6 (b) and (c). Quantitative evidence all altogether shows our method can efficiently learn where to "invest" the limited budget of pixels at downsampling to achieve the highest overall return in segmentation accuracy.
5-13
(4)
64
Figure 6: Cost-performance trade-offs in mloU and m Y) on three datasets. Cost determines the amount of computation required at segmentation. The red lines indicates at same performance our method can save 33%. 90% and 70% from the "uniform" baseline on the three datasets respectively.
Visual results from Figure and Figure 8 confirms our method integrates the advantage of the "edge-based" approach by sampling denser small objects while further generalise with better sparse sampling on larger objects (see caption for details). Visual evidence also suggests the object edges are not always informative (see window edges in the wall class of Figure 7 misguided the edge-based sampler ander-sampling of surrounding textures and lead to miss-pree phiss-predictio, and similarly as bushes in the green box of Figure 8). We therefore further discuss how boundary accuracy affects segmentation accuracy in the next section.
Ground Truth
Ours
Edge- based
Uniform
691- 11151
40
10
Figure 7: Examples on Cityscapes (Cordts et al., 2016) comparing our method against both baselines, where segmentation is performed on 16 times downsampled images (at each dimension). Predictions masked over and sampling locations shown in white dots. Yellow/ red arrows indicated regions denser/ sparser sampling helped to segment rider (red)/ sky (blue) classes, respectively.
4.2
A FAIR COMPARISON TO THE SOTA AND BOUNDARY ACCURACY ANALYSIS
In this section we perform experiments at same experimental condition on Cityscapes dataset and compare to reported results from Marin et (2019). The "interpolation" (Talebi & Milanfar, 2021) baseline is also compared. Figure 9 (a) shows method outperforms all three baselines at all trade-off downsample sizes by upto 4% in absolute mlou over the second best result and achieves better cost-performance orf saving upto 47% calculation. Figure 9 (a) also suggests "interpolation" approach is less effective at small downsampling while its benefits increase as downsampling size increases. Visual comparison to "interpolation" baseline provided in Appendix Section A.2.
We introduced the edge-loss to regularise the training of the joint system, inspired by Marin et al. (2019). Here we measure boundary precision and ask 1) how does it contribute overall segmentation
8

---

Published as a conference paper at ICLR 2022
Input Image
Ground Truth
Uniform
Figure 8: Qualitative example on DeepGlobe dataset (Demir al., 2018) where segmentation performed on 8 times downsampled image (at each dimension). Predictions are masked over and sampling locations are shown red dots. Ycllow/ green boxed region indicated regions denser/ sparser sampling helped segmenting water (blue)/ forest (purple) classes, respectively.
accuracy? and 2) how differently do the joint trained system balance boundary accuracy and overall segmentation accuracy than the separately trained system (Marin al., 2019)? We adopt trimap following Kohli et al. (2009); Marin et al. (2019) computing the accuracy within a band of varying width around boundaries in addition to moU, for all three datasets in Figure 9 (b). We found: 1) the "edge-based" baseline is optimal close to the boundary, but its performance does not consistently transfer overall mloU/: 2) Our joint learned downsampling shows can identify the most beneficial sampling distance to invest sampling budget and lead to the best overall mIoU: 3) The most beneficial sampling distances learned by our approach are data-dependent that can close to the boundary (i.e.
Cityscapes) or away (i.e. DeepGlobe and PCa-Histo); 4) it also suggests our method is particularly useful when the labelling is based on higher-level knowledge, i.e. Deepglobe and PCa-Histo dataset.
Cooll pezformanee trde off cave Citycagun eorarn
Cityscaps tdveessmppeed size
DeepClnde (domesamplo size 200x200)
Elisto (dowasazpped -20002)))
gedge-Jus
male: busel meas jast
8.3
Abee Bovoannnengg (nD0l 4
4.
wnd
58
3.42
0.
033
3.
Jus Jnss.
dige bocel
0.2
Dad pasked
-4.
avendi
-210
Tiisup mitll ploii
(a)
Figure 9: (a) Experiments performed comparable condition to reported "dgge-based" results by Marin (2019) on Cityscapes dataset (Cordts et al., 2016), i.e. applying central 1024 x 1021 crop pre-processing and train with PSP-net (Zhao al., 2017); (b) trimap analysis of absolute accuracy near semantic boundaries and overall mfo gain over "uniform" baseline the three datasets.
CONCLUSION
We introduce approach for learning to downsample ultra high-resolution images for segmentation tasks. The main motivation is to adapt the sampling budget to the difficulty of segmented pixels/regions hence achieving optimal cost ress-performance trade-off. We empirically illustrate the SOTA method (Marin et al., 2019) been limited by fixing downsampling locations to manual designed sampling objective, and hence motivate our end-to-end adaptive downsampling method. We illustrate simply extending the learn to downsample method from image classification (Recasens et al., 2018) to segmentation does not work, and propose to avoid learning trivial downsampling solutions by incorporating edge-loss to regularise the training. Although our edge-loss, despite a simpler approximation, share the same spirit with Marin et al. (2019) we demonstrate our jointly trained system generalises sampling more robustly especially when object boundaries are less informative, hence consistently leading to a better cost-performance trade-off. Our method is light weighted and can be fiexibly combined with the existing segmentation method without modifying the architecture.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to thank our reviewers for valuable discussions and feedback the work and manuscript. We acknowledge the EPSRC grants EP/R006032/1, EP/M020533/1, the CRUK/EPSRC grant NS/A000069/1, and the NIHR UCLH Biomedical Research Centre which supported this research.
REFERENCES
Liang- Chigh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv: 1412. 2014.
Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention scale: Scale-aware semantic image segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3640-3649, 2016.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3213-3223 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li,, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
like Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge parse the earth through satellite images. In The IEEE Conference on Computer Vision and Patlern Recognition (CVPR) Workshops, June 2018.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Proceedings of the IEEE International Conference on Computer Vision. pp. 9785-9795, 2019.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-gr fine-grained localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 447-456, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017. 2017.
Chuong Huynh, Anh Tuan Tran, Khoa Luu, and Minh Hoai. Progressive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16755-16764, 2021.
2021.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. arXiv preprint arXiv: /506 02025, 2015.
Chen Jin, Ryutaro Tanno, Moucheng Xu, Thomy Mertzanidou, and Daniel Alexander. Foveation for segmentation of mega- pixel histology images. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 561-571. Springer, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/VV conference on computer vision and pattern recognition, pp. 9799-9808, 2020.
Pushmeet Kohli, Philip HS Tomr et al. Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 3233)3302-324, 2009.
10

---

Published as a conference paper at ICLR 2022
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3226-32222 -2229, IEEE, 2017.
Dmitrii Marin, Zijian He, Peter Vajda, Priyam Chatterjee, Sam Tsai, Fei Yang, and Yuri Boykov. Efficient segmentation: Learning downsampling near semantic boundaries. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2131-2141, 2019.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Adria Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba. Learning to zoom: a saliency- based sampling layer for neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 51-66, 2018.
Chetan L Srinidhi, Ozan Ciga, and Anne L Martel. Deep neural network models for computational histopathology: A survey, arXiv preprint arXiv: 12378 2019.
Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang- High-resolution representations for labeling pixels and regions. arXiv preprint arXiv: 1904.04514, 2019.
Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. arXiv preprint arXiv:2103.000 099509 2021.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for nigh-resolutton 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10888-0966, 2017.
Xueyang Wang. Xiya Zhang, Yinheng Zhu, Yuchen Guo, Xiaoyun Yuan, Liuyu Xiang, Zerun Wang, Guiguang Ding, David Brady, Qionghai Dai, et al. Panda: A gigapixel-level human-centric video dataset. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 32682268330 0278,, 2020.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:151..07122 2015.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.

---

Published as a conference paper at ICLR 2022
A APPENDIX
Here we provide additional results and various ablation studies and implementation details that have not been presented in the main paper.
CONTENTS
A.1 Sensitivity analysis on hyperparameters
12
A.2 Implementation and visual illustration of the "interpolation" " baseline
13
A.3 More visual examples at extreme downsampling rates
13
A.4 Investigate the learned class distribution
14
A.5 Simulated ledge-based" results
15
A.6 Datasets
16
A.7 Network Architectures and Implementation Details
17
A.1 SENSITIVITY ANALYSIS ON IIPPERPARAEETERS
We perform sensitivity analysis four hyperparameters in our system, Cityscapes (Cordts et al., 2016) and DeepGlobe (Demir et al., 2018) datasets Table to Table 5. In specific, Table investigate the standard deviation of the kernel & in equation 3, which impact the distance the attraction force can be acted on. Table 3 and Table investigates the impact of kernel size and input size on the proposed deformation module, which impact model capability and amount of context the deformation module. Table 5 investigates edge-loss weight in equation 4 which impact the weight of the edge loss. In general our method has been stable and robust across all tested hyperparameters (with absolute mIoU variation within 3%).
Table 2: The sensitivity analysis of the hyperpa- Table The sensitivity analysis of the hyper- rameter of the kernel & in equation 3: the standard deviation a.
parameter of the kernel size in the proposed deformation module.
Cityscapes (1024²
962 pixels)
Cityscapes (1024 to 962 pixels)
a (pixels)
15
26
32
kernel size mloU
3x3
5 x
7 7 0.35
mToU
0.35
0.34
0.32
0.36
0.35
DeepGlobe (2448² to 100? pixels)
DecpGlobe (2448² to 1002
pixels)
a (pixels)
20
25
33
kernel size
3x3 0.45
5x5 0.45
7x7 0.46
mloU 0.43
0.45
0.43
mloU
Table 4:
The sensitivity analysis of different low-res input Table The sensitivity analysis of the edge-loss weight in equation 4
sizes to the proposed deformation module.
Cityscapes (1024 to 642 pixels)
Cityscapes (10242 to 642)
low-res input size
322
482
642
802
962
7
50
100
200
mloU
0.35
0.35
0.34
0.33
0.36
mloU
0.33
0.35
0.35
DeepGlobe
(2448² to 50 pixels)
DeepGlobe (2448² to 100?)
low-res input size
502
752
100a
200
300
7
50
100
200
mloU
0.44
0.42
0.45
0.42
0.43
mloU
0.45
0.45
0.43
12

---

Published as a conference paper at ICLR 2022
A.2 IMPLEMENTATION AND VISUAL ILLUSTRATION OF THE "INTERPOLATION BASELINE
The "interpolation" baseline is implemented by replacing our deformation module with the resizing network proposed by Talebi & Milanfar (2021) and without using the deformed sampler. The number of residual blocks (r) and the number of convolutional filters (v1) are two key hyperparameters resizing network, hence we perform each of our experiments with suggested hyperparameter combinations by Talebi & Milanfar (2021), and select the best performed results to represent the "interpolation" baseline, with all results provided in Table 6.
Table 6: The "Interpolation" baseline performance with different combinations of the number of residual blocks (r) and the number of convolutional filters (n) for the resizing network, at each the three downsampling sizes the cityscapes dataset (with the central crop of size 1024 x 1024 pixels). Results are measured in m mloU.
Blocks
Downsampling size
Filters
64x 64
n= 16
0.29
0.27
0.27
0.27
=== 32
0.29
0.29
0.29
0.29
128 x 128
n 16
0.42
n/a
n/a
0.43
32
0.43
n/a
n/a
0.44
256 x 256
16
0.54
n/a n/a 0.54
32
0.55
n/a
n/a 0.55
To better understand why our method works better than the "interpolation" baseline small downsam- pling sizes as in Figure 9 (a), we plot visual examples in Figure 10. The visual examples confirm jointly learning where to sample more effective strategy when sampling budget is limited, while with more sampling budgets available the learning to "interpolation" approach would also work (Figure 9 (a)).
input/ label
downsampled/ prediction (ours)
downsampled/
input/ label
downsampled/ prediction (ours)
downsampled/ prediction (interpolation)
prediction
(interpolation)
Figure 10: Visual comparison of our jointly trained downsampling against "interpolation" baseline (Talebi & Milanfar, 2021) on Cityscapes dataset where inputs size 1024 1024 pixels are downsampled to 64 x 64 pixels. Sampling locations are masked over prediction as white dots.
A.3 MORE VISUAL EXAMPLES EXTREME DOWNSAMPLING RATES
A visual overview of applying our method on Cityscapes are given in Figure 11. To further verify how our method performs at extreme downsampling rate on histology images, in Figure 12, we show visual examples when downsampling rate of 50 times at each dimension is applied to the PCa-Histo dataset. Consistent with quantitative results, our method can still perform well comparing to ground truth and significantly better than the "uniform" baseline at such an extreme downsampling rate. Our method performs especially well for the most clinically important classes, Gleason Grade 3 and Gleason Grade 4. the separation of these two classes is often referred to the threshold from healthy to cancer. The sampling locations shown as blue dots in Figure 12 also indicate our method learned to sample densely clinical important and challenge regions (e.g. cancerous nuclei and tissues) while sampling sparsely
13

---

Published as a conference paper at ICLR 2022
Input Image
2000 x 640
Ours
Ground Truth
Uniform
Figure 11: Semantic segmentation with learned deformed downsampling on Cityscapes. Our method learns non non-uniffrm downsampling (white dots on the images) high- regouresotttion image (top-row) row) to preserve information that guides segmentation and ignore image content that does not contribute (green arrows) thereby producing a deformed downsampled image (bottom- corner) which leads to more accurate low-resolution segmentation. This strategy helps identify small regions such as the people and traffic signs the figure; see masked ground truth and segmentation reversed from low-resolution predictions in the middle-row. Compared to "Uniform" baseline our method ("Ours") sample more densely at important semantic regions (yellow arrows) and lead to more accurate segmentation (see bottom-row).
less informative regions (e.g. umlabelled//backrruund)) which lead consistent improvement over uniform sampling in segmentation accuracy.
A.4 INVESTIGATE THE LEARNED CLASS DISTRIBUTION
To verify whether our deformation module has learned downsample denser at important regions/classes effectively, we monitor class frequency change after downsampling, as shown in Figure 13.
The results indicated the deformation module learned to adjust sampling strategy so that more pixels "invested" to important regions/classes and less from the less informative regions/classes, and lead to optimal overall segmentation accuracy. For example, the deformation module learned to sample less (shown as negative sampling frequency ratio) in the "Background" class in the PCa-Histo dataset (Figure 13 (a)) and "road" and "sky" classes in the cityscapes (Cordts et al.. 2016) dataset (Figure 13 (b)), in both cases those classes can be considered as potential "background" with common knowledge. However, our method also shows the ability to adjust sampling strategy on more complex dataset where "background class" is less obvious. For example, in DeepGlobe (Demir et al., 2018) dataset there no obvious "background class", but the "Urban" class has been learned to sample much denser than other classes (Figure 13 (c)), and leads to boosted segmentation accuracy (see Figure 4 and Figure in the main paper). The learned distribution on the DeepGlobe dataset also verified our argument that the manual designed sample denser around object edge strategy as the "edge-based"" (Marin et al., 2019) method may fail when object boundary less informative, which is justified in the main results.
14

---

Published as a conference paper at ICLR 2022
5775 x 1984
116
Ours
Ground Truth
Uniform
Unlabelled
Benign
Gleason Grade
Gleason Grade 4
Atrophy
HGPIN
Figure 12: Qualitative example on PCa-HHiso dataset (I) where segmentation performed on times (at each dimension) downsampled image. Predictions are masked over and sampling locations are shown in blue dots.
fequarcy staanal toage (Clpuapo))
Clase
stlaiaal Emage (Douplikbb))
Aego
[3]
[C]
Figure 13: Class frequency change the ratio between the sampled and original image. Higher value indicting more sampling pixels are "invested" one class with the leamedd sampler. Results suggest our method can learn to sampler denser at semantic important classes while ignoring those not contribute. Results are from three datasets: (a) a local prostate cancer histology dataset (PCa- (PCa-Histo) the dynamic downsampled size of 80 800 pixels (b) Cityscapes (Cordts et al., 2016) the downsampled size of 64 x 128 pixels and DeepGlobe (Demir al., 2018) at the downsampled size 300 x 300 pixels.
A.5 SIMULATED "EDGE-BASED" RESULTS
We direct simulate a set of "edge-based" samplers each with different sampling density around edges. For each simulated "edge-base" downsampler, we expand sampling locations away from the label edge different distances in Gaussian manner mimic the effect of X sampling as shown by Marin et al. (2019). In specific, we calculate edge from label then apply gaussian blur to generate simulated deformation map guide sampling framework. We fix each simulated edge-haaedd"
15

---

Published as a conference paper at ICLR 2022
downsamplers to train segmentation network, with all results provided in Figure 14, among which best tesst-erformed in each dataset is selected as the edge-basedd" baseline.
Cliyao - devoningllll size 64x128
DacyGlite- dewesonphlllllooooooooooooooo..m size 300
PCa-Histo ECoHHsb-dowtsmmppn dowrsamplod siez 500500
1.40
6:20
4:30
8:42
8.40
3.40
8-00
6:34
8.50
9,39
8,500
6:11
8.500
Miz
137 0.30
(8)
(b)
(c)
Figure 14: The segmentation accuracy with a set simulated "odge-based" downsamplers. Each bar is the
segmentation accuracy of a segmentation network trained with one simulated "edge-based" downsampler. Fach downsampler is simulated with a different sampling density around edges, represented by blur radius. The low blur radius indicates dense sampling around edges. The best-performed result selected and referred to as the "edge-based" result presented in the main paper.
A.6 DATASETS
In this work, we verified our method on three segmentation datasets: DeepGlobe aerial scenes segmentation dataset (Demir al., 2018), Cityscapes urban scenes segmentation dataset (Cordts et al., 2016) and Histo medical histopathological segmentation dataset.
The DeepGlobe (Demir al., 2018) dataset has 803 nigh-resolution (2448 x 2448 pixels) images of acrial scenes. There are 7 classes of dense annotations, 6 classes among them are used for training and evaluation according to Demir al. (2018). We randomly split the dataset into the train, validate and test with 455, 207, and 142 images respectively.
The Cityscapes (Cordts et al., 2016) dataset contains 5000 high-resolutien (2048 1024 pixels) urban scenes images collected across 27 European Cities. The finely-annotan images contain 30 classes, and 19 classes among them are used for training and evaluation according to Cordts et al. (2016). The 5000 images from the Cityscapes are divided into 2975/500/155 1555 images for training, validation and testing.
The PCa-Histo: dataset contains 266 ultra-high resolution whole slide images, from 33 distinct biopsies. The size of the images ranged from 250800 pixels to 40880000 pixels. Each pixel is annotated into one out of six classes: Unlabelled, Benign, Gleason Grade 3, Gleason Grade 4, Atrophy, HGPIN (see Table 7 for explanations of each class). Among the six classes, informative classes are underrepresented compared to Unlabelled classes (see Table 7). We random split the dataset into 200 training, 27 validation and 39 test images. This dataset has varying size, with height (the short axis) range between 229 to 2000 pixels (1967.5 215.5 pixels), width (the long axis) range between 3386 to 20440 pixels (9391.9 + 44733.8 8 pixels) and Aspect Ratio (AP) range between 1.69 to 33.9 (5. 3.554. The downsampling size of this dataset is calculated as follows: with a dynamic downsample size of for example S0 x 800, for each image, we calculate two candidate downsample sizes by either downsampling the short axis to 80 or the long axis to 800 while in either case we keep AP unchanged. Then we select the downsampling size that leads fewer total pixels. For example, for an image with size 368 x 7986 pixels, we have candidate downsample sizes of 40 x 868 pixels or 36 x 800 pixels, and we will use downsample size of 36 x 800 in our experiment.
Table 7: Pixel frequency distribution of PCa-Histo dataset and explanation of cach class
Label
Background
Benign
Gleason Grade 3
Gleason Grade 4
Atrophy
HGPIN
% pixels in dataset
93.99
0.97
1.23
3.35
0.26
0.20
Explanation
all unlahelled pixels
healthy
mild
aggressive cancer
healthy noise
precursor legion
cancer
The prostate biopsies of the PCa-Histo dataset are collected from 122 patients. The identity of the patients is completely removed, and the dataset is used for research. The biopsy cores were sliced,
16

---

Published as a conference paper at ICLR 2022
mounted on glass slides, and stained with H&E. From these, one slice of each of 220 cores digitised using a Leica SCN400 slide scanner. The biopsies were digitised at 5, 10 and 20 magnification. At 20 magnification (which is the magnification used by histopathologists to perform Gleason grading), the pixel size is 0.55 um?
A.7 NETWORK ARCHITECTURES AND IMPLEMENTATION DETAILS
Architecfures:
The deformation module defined as a small CNN architecture comprising layers each with 3 3 kernels follower by BatchNorm and Relu. The number of kernels in each respective final 1 convolutional layer is applied reduce the dimensionality and a
layer is (24, softmax layer for normalisation are added at the end. All convolution layers initialised following He initialization (He et al., 2015). The segmentation module was defined as a deep CNN architecture, (Sun et al., 2019) applied in all datasets, and PSP-net (Zhao et al., 2017)
with HRNetV2-W sanity check the Cityscapes dataset. The segmentation network HRNetV2 2-4488 is pre-trained on the Imagenet dataset provided by Sun et al. (2019). In all settings, the size of the deformation map is either same or twice the size of the downsampled image.
we employ the same training scheme unless otherwise stated. We optimize parameters using Adam (Kingma & Ba, 2014) with an initial learning rate of x 1003 and 3 =0.,, and train for 200 epochs on DeepGlobe, 250 epochs on PCa-Histo dataset, and 125 epochs on Cityscapes dataset. We use batch size of 4 for the Cityscapes dataset (Cordts al., 2016) and the DeepGlobe dataset (Demir et al., 2018) and a batch size of for the PCa-Histo dataset. We We employ the step decay learning rate policy. We scale the edge loss 100 times is the same scale the segmentation loss. During the training of the segmentation network we do not include upsampling stage but instead, downsample the label map. For all datasets, we take the whole image downsample as input, unless otherwise stated. During training, we augment data by random left-right flipping. All networks are trained on 2 GPUs from an internal cluster (machine models: gtx 1080ti, titanxp, titanx, rtx2080ti, p100, v100, rtx6000), with syncBN.
Training: For experiments, we
Downsampling sizes: Downsampling configurations for the cost-performance trade-offs
experiments are summarised in Table 8.
Table 8: Downsampling sizes of the cort-performance trade-off's experiments
Cityscapes (original size 1024 x 2048 pixels)
downsampled sizes (pixels)
16 x2
32 x4
64x 128
128
fraction of pixels sampled from original
2e-4
1e-3
3.9e-3
1.6e-2
DeepGlobe (original size 2148 pixels)
downsampled sizes (pixels)
50 50
100 100
200
300
fraction of pixels sampled from original
4e-4
1.7e-3
6.7e-3
1.5e-2
PCa-Histo (original size 1968 0068+2211 216 9392
pixels)
downsampled sizes (pixels)
40
80x
fraction of pixels sampled from original
4.1e-4
1.7c-3
17


Reviews:
[{"review_id": "wr6OD2ScYsn", "reviewer": "Reviewer_wnXt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is well motivated based on detailed description of the limitations of the closest recent work.\n+ The results are detailed and well explained.\n\n- \"Consider a relative coordinate system such that I[u, v] is the pixel value of I where u, v∈[0,1].\" is not clear that why the coordinates need to be normalized between [0,1]\n- The term b[i,j] in Eq.2 is not explained. Although its an equation from another paper, it would help to have some definition of this variable.\n- Eq 3 needs to be explained a bit better. Also what is the kernel k in Eq.3 and how is it defined in the paper implementation.", "summary_of_the_paper": "Semantic segmentation using deep learning techniques becomes challenging if the input images are of high resolution. The image resolution can be lowered and the network learned, but downsampling based on uniform grid pattern on the high resolution image can lead to network not being able to learn semantic information in high frequency regions, i.e. the loss of information is uniform over the image. In order to alleviate this problem, the downsampling can be done based on a non-uniform grid on the high resolution image which is content adaptive where more samples are selected at the high frequency/important regions while low frequency regions are sampled sparsely. Previous work has computed this non-uniform downsampling in a deep learning framework as an independent task. This paper proposes to jointly learn the downsampling grid estimation task and the particular task of semantic segmentation. They show improved IoU metric over semantic segmentation results on standard benchmarks.", "summary_of_the_review": "The paper shows that jointly doing content driven subsampling and the high level task (semantic segmentation here) can lead to better results rather than doing them independently. This idea may be propagated to other high level vision tasks which are limited by image size of training data. Thus, the reviewer is inclined favorably towards this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_toGIQr4BiG", "reviewer": "Reviewer_wmVs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This paper proposes a new deformed sampling module that explicitly downscale the image into target resolution. The idea of making the network to learn the sampling density is very intuitive, and it is also adaptable to classical sampling functions.\n- Besides, the author noticed the importance of regularization and added the edge loss in training. \n- The proposed method is end-to-end trainable, and can be plugged into different backbone networks.\n\nWeakness:\n\nStill, I have complex feeling about this paper: the motivation is really good: following all the previous works that try to improve the segmentation performance by allocating the computations to most important regions, the proposed module does make a lot of sense, and it's plug-and-play. But when I read the experiment results, I find that it is not the module itself that brings the most difference, but the joint loss. From Fig.5, it can be seen that adding the module trained with single loss is not as good as the uniform one at the most time. This implies that the most important contribution of this paper is the joint loss.\n\nThough, if this paper wants to illustrate the importance of joint loss that avoids the trivial solution, there are still some missing points to support this claim:\n- Not enough ablation studies. If applying the joint loss on other methods, how the performance would be influenced? \n- More analysis of why the single segmentation loss would not work. After reading this paper, I still do not understand: why single loss would lead to a trivial solution?\n\nI read the appendix and find the downsampling module consists of 3 layers of CNN with 3x3 kernel. Since the other works in 2.1 all aim to cover multiple scale information, does it matter that this module only has this limited spatial window? And, Fig. 12 shows that the sampling density suddenly increases on the edge of the picture, even if the content is still solid-color. Why would this happen? Is this learned module vulnerable to noises in image?\n\nBesides, since this work adopts a learned CNN module to predict the sampling density, and then conduct rescaling by interpolation, I wonder why not directly makes the network to directly downsample/super-resolve the input? Such ideas are applied in \"Learning to Resize Images for Computer Vision Tasks\". How does the proposed method compare with this paper?\n\n\nMinor errors:\n- In the caption of Fig. 5, the last line should note \"(c) a local prostate cancer histology dataset...\".", "summary_of_the_paper": "This paper aims to improve the performance of segmenting ultra high-resolution images by replacing the commonly used downsampling functions with the learned deformation sampling module. The biggest difference of the proposed method is, instead of sampling all areas uniformly, the sampling density is estimated based on the contents. The experiment results on the Cityscape, DeepGlobe and PCa-Histo datasets show some improvements on mIoU and cost-performance trade-offs compared with uniform downsampling.", "summary_of_the_review": "Overall, this paper has good motivation and proposes an interesting method. But after reading the whole paper, I felt that it is not the proposed method, but the joint loss that contributes most to the performance improvement. Besides, this paper lacks of in-depth explanation of the performance drop when applying the single loss, and comparisons of several similar methods. So I would give score 5 for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JDy2AXHXqzj", "reviewer": "Reviewer_eqeg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well-written. The problem, motivation, idea, and method are clearly stated. Sufficient visualization is provided to make the paper easier to read.\n\n2. Comprehensive experiments and analyses are conducted. The results on three publicly available datasets are reported.\n\n\n\nWeaknesses:\n\n1. As the authors said, the proposed method is similar to [Recasens et al. 2018], so the novelty of the proposed method is somewhat limited. However, the authors show that directly applying [Recasens et al. 2018] to segmentation tasks is not necessarily effective. Hence, I agree with the authors that the proposed method is somewhat novel. \n", "summary_of_the_paper": "This paper address the semantic segmentation problem on high-resolution images. Existing methods uniformly downsample the original image to a small version to meet the memory requirement, while the uniform downsampling is suboptimal. The authors propose a deformed downsampling method in this paper and mainly compare it with the previous edge-based downsampling, showing better performance. Although the proposed method is somewhat similar to one previous method, while directly applying which on segmentation task does not improve the performance. This demonstrates the importance of the modifications proposed by the authors in this paper.  ", "summary_of_the_review": "The paper is well-argued, and the proposed method is effective. Novelty is somewhat limited but can meet the ICLR bar. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D6gZr7J4mM", "reviewer": "Reviewer_EMME", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the papers are:\n1. It extends the learn to zoom approach for classification to segmentation tasks. A deformable module is applied for downsampling the original image.\n2. The upsampling processing is incorporated in the training phase.\n3. The edge information is also used to guild the downsampling step which improves the segmentation performance. \n\nThe overall presentation of the method is great. I have some questions about the paper. \n1. The loss function consists of segmentation loss and edge loss. A weight term is used to balance these two terms. Is this the same problem as mentioned in Marin et. al (2009) in the barrier of the SOTA section? \n2. It would be better to add more text to generate the downsampled image (ex. R^{52x100}) from low-res (ex. R^{13x25}).\n\nMissing references:\nKirillov, Alexander et. al., Pointrend: Image segmentation as rendering, CVPR 2020\nHuynh, Chuong et. al., Progressive Semantic Segmentation, CVPR 2021", "summary_of_the_paper": "The authors present a method for ultra-high resolution image segmentation. It follows the learn to zoom approach to non-uniform downsample the original high resolution image. Furthermore, it adds edge-based loss to guide the downsample process. The upsampling step is also included in the training stage. The proposed method is evaluated on several public datasets and achieves promising results. Effectiveness of components are verified in the experiments. ", "summary_of_the_review": "The method mainly extends the learn to zoom approach for classification to segmentation tasks. Importantly, the edge loss is applied to guild the learning based downsample. The authors present extensive experiments and analysis to assess the components of the method. This method brings new ideas or approaches to handle high resolution images, especially in segmentation tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper receives positive reviews. The reviewers find the technical novelty and contributions are significant enough for acceptance at this conference. The authors' rebuttal helps address some issues. The area chair agrees with the reviewers and recommend it be accepted at this conference.
Decision: Accept (Poster)