Label super-resolution networks

1 INTRODUCTION . Semantic image segmentation is the task of labeling each pixel in an input image X = { xij } as belonging to one of L fine-scale application classes , Y = { yij } , y ∈ { 1 , . . . , L } . In weakly supervised segmentation , instances in the training set only contain partial observations of the target ground truth labels , e.g. , summary of class labels instead of pixel-level labels . We aim to solve a variant of this problem where coarse-scale , low-resolution accessory classes , Z = { zk } ; z ∈ { 1 , . . . , N } , are defined for sets of pixels in the input images , where we are given the joint distribution P ( Y , Z ) between the accessory class labels and the application labels . Specifically , a training image X is divided into K sets Bk , each with an accessory class label zk , and our models are trained to produce the high-resolution application labels yij . For example , in Figure 1 , a high-resolution aerial image is shown alongside the low-resolution ground truth land cover map ( defined over accessory classes ) and the target high-resolution version ( defined over application classes ) . We aim to derive the high-resolution land cover map based on the aerial image and low-resolution ground truth . Compared to other weakly supervised image segmentation techniques , the formulation of the problem we aim to solve is more general : it applies both to existing weakly supervised image segmentation problems , as well as to other problems with different characteristics of weak labels . The more general formulation is necessary for tasks such as land cover mapping from aerial imagery and lymphocyte segmentation from pathology imagery . In these applications , coarse labels do not necessarily match the fine-scale labels , as shown in Figure 1 . The distinction between the fine-scale application and coarse-scale accessory classes is necessary for situations in which the ground-truth information that is known about an image does not match with the application classes that we aim to ∗jojic @ microsoft.com label the image with , but instead suggests a distribution over the application labels . State-of-the-art methods for weakly supervised semantic segmentation exploit the structure of weak labels in ways that are not applicable in our examples : we can not create bounding boxes around land cover object instances ( Dai et al . ( 2015 ) ; Papandreou et al . ( 2015 ) ) – we consider data that is generally given at scales much larger than the objects being segmented and does not carry foreground-background morphology – nor use coarse approximations of ground-truth segmentation ( Krähenbühl & Koltun ( 2011 ) ; Hong et al . ( 2015 ) ) . Other work attempts to match a class “ density function ” to weak labels ( Lempitsky & Zisserman ( 2010 ) ) , but it mainly targets localization and enumeration of small foreground objects with known sizes . Existing Weak supervision approaches also often involve expensive steps in inference , such as CRFs or iterative evaluation ( Chen et al . ( 2015 ) ) , which are impractical on large datasets . At the same time , thorough analyses of training algorithms only exist for models that are not sufficiently expressive for the applications we consider ( Yu et al . ( 2013 ) ) . While our formulation of the problem allows us to specifically address the previously mentioned land cover mapping and lymphocyte segmentation , it can also be applied to more traditional segmentation tasks such as foreground/background segmentation as we explore in Appendix . F. Our proposed method is illustrated in Figure 2 . Briefly , a standard segmentation network will output probabilistic estimates of the application labels . Our methodology summarizes these estimates over the sets Bk , which results in an estimated distribution of application labels for each set . These distributions can then be compared to the expected distribution from the accessory ( low-resolution ) labels using standard distribution distance metrics . This extension is fully differentiable and can thus be used to train image segmentation neural networks end-to-end from pairs of images and coarse labels . Land cover mapping from aerial imagery is an important application in need of such methodology . Land cover maps are essential in many sustainability-related applications such as conservation planning , monitoring habitat loss , and informing land management . In Section 3.1 we describe land cover mapping in detail and show how our method creates high-resolution land cover maps solely from high-resolution imagery low-resolution labels , at an accuracy similar to that of models trained on high-resolution labels . We further show how to train models with a combination of low- and highresolution labels that outperform the high-res models in transfer learning tasks . As low-resolution labels are much easier to collect , and indeed exist over a much wider geographic area in our land cover mapping application , the ability to combine low- and high-resolution labels is an important feature of our proposed methods . In a second example ( Section 3.2 ) , we segment tumor infiltrating lymphocytes from high-resolution ( gigapixel ) pathology images . Understanding the spatial distribution of immune cells , such as lymphocytes in pathology images , is fundamental for immunology and the treatment of cancer ( Finn ( 2008 ) ; Thorsson et al . ( 2018 ) ) . Here , coarse labels are probabilities of lymphocyte infiltration ( having two or more lymphocytes ) on 100×100 pixel regions , given by an automatic classifier ( Saltz et al . ( 2018 ) ) . Our super-resolution model trained on coarse labels performs the same as a lymphocyte classifier trained with high-resolution ( cell-level ) supervision ( Hou et al . ( 2018 ) ) . To summarize , as our first contribution , we propose a label super-resolution network which utilizes the distribution of high-resolution labels suggested by given low-resolution labels , based on visual cues in the input images , to derive high-resolution label predictions consistent to the input image . Our second contribution is that we evaluate our method extensively on the application of land cover segmentation and conclude that when there are not enough representative high-resolution training data , our method is much more robust than a model trained on high-resolution training data only , since our method utilizes more training data with weak labels . We show the generality of our method on the lymphocyte segmentation task and the task of segmenting foreground given object bounding boxes ( in Appendix F ) . 2 CONVERTING A SEMANTIC SEGMENTATION NETWORK INTO A LABEL SUPER-RESOLUTION NETWORK . A semantic segmentation network takes pixels X = { xij } as input and produces a distribution over labels Y = { yij } as output . If φ are learned network parameters , this distribution is factorized as : p ( Y |X ; φ ) = ∏ i , j p ( yij |X ; φ ) , ( 1 ) Each p ( yij |X ; φ ) is a distribution over the possible labels , y ∈ { 1 , . . . , L } . Typically , a network would be trained on pairs of observed training images and label images , ( Xt , Y t ) , to maximize : φ̂ = argmax φ log ∏ t p ( Y t|Xt ; φ ) = argmax φ ∑ t ∑ i , j log p ( ytij |Xt ; φ ) . ( 2 ) In this paper , we assume that we do not have pixel-level supervision , Y t , but only coarse accessory ( low-resolution ) labels zk ∈ { 1 , . . . , N } given on sets ( blocks ) of input pixels , Bk . We also assume a statistical joint distribution over the number of pixels c ` of each application label ` ∈ { 1 , . . . , L } occurring in a block labeled with an accessory ( low-resolution ) label z , pcoarse ( c1 , c2 , . . . , cL|z ) . Our extension of semantic segmentation networks is described in the following three sections . Using coarse labels as statistical descriptors . In computer vision applications , pixel-level labeled data is typically expensive to produce , as is the case of high resolution land cover mapping where high-resolution labels only exist for limited geographic regions . On the other hand , coarse lowresolution labels are often easy to acquire and are readily available for larger quantities of data . Coarse labels can provide weak supervision by dividing blocks of pixels into categories that are statistically different from each other . To exploit this we must formally represent the distribution of high-resolution pixel counds in these blocks , pcoarse ( c|z ) . For example , in the case of land cover mapping with four types of high-resolution land cover classes1 , the descriptions of labels from the National Land Cover Database ( NLCD ) – at 30 times lower resolution than available aerial imagery ( Homer et al . ( 2015 ) ) – suggest distributions over the high-resolution labels . For instance , the “ Developed , Medium Intensity ” class – see Table 3 in the appendix – is described as “ Areas with a mixture of constructed materials and vegetation . Impervious surfaces account for 50 % to 79 % of the total cover ” . While such a designation does not tell us the precise composition or arrangement of high-resolution labels within a particular “ Developed , Medium Intensity ” label , it does describe a distribution . One mathematical interpretation of this particular example is cimperv ∼ unif ( 0.5 , 0.8 ) , cforest + cfield = 1− cimperv , cwater ≈ 0 . In practice these descriptions should be interpreted in a softer manner ( e.g. , with Gaussian distributions ) that can account for variance in real-world instances of the coarse classes2 . Label counting . Assume pcoarse ( c|z ) , a connection between the coarse and fine labels , has been represented . Suppose we have a model that outputs distributions over high-resolution labels , p ( Y |X ) given inputsX . We must summarize the model ’ s output over the low-resolution blocksBk . Namely , a label counting layer computes a statistical representation θk of the label counts in each block Bk . If we sampled the model ’ s predictions yij at each pixel , the count of predicted labels of class ` in block Bk would be c ` = 1 |Bk| ∑ ( i , j ) ∈Bk δ ( yij = ` ) . ( 3 ) By averaging many random variables , these counts c ` will follow an approximately Gaussian distribution , pnet ( c ` , k = c|X ) = N ( c ; µ ` , k , σ2 ` , k ) , where µ ` , k = 1 |Bk| ∑ ( i , j ) ∈Bj p ( yij = ` |X , φ ) , σ2 ` , k = 1 |Bk|2 ∑ ( i , j ) ∈Bk p ( yij = ` |X , φ ) ( 1−p ( yij = ` |X , φ ) ) . ( 4 ) These two parameters for each label ` constitute the output of each block ’ s label counting layer θk = { µ ` , k , σ2 ` , k } L ` =1 . Note that treating each count c ` as an independent Gaussian variable ( given the input X ) ignores the constraint ∑ ` c ` = 1 , and more exact choices exist for modeling joint distributions pnet ( { c ` } |X ) ; however , we do have ∑ µ ` = 1 and thus E [ ∑ ` c ` ] = 1 . In practice , this approximation works well . Statistics matching loss . The coarse labels z provide statistical descriptions for each block pcoarse ( { c ` } |z ) , while the label counting modules produce distributions over what the segmentation network sees in the block given the high-res input image X , pnet ( { c ` } |X ) . The statistics matching module computes the amount of mismatch between these two distributions , D ( pnet , pcoarse ) , which we then use as an optimization criterion for the core segmentation model . Namely , we set C ` = argmax c ` [ ∑ log pnet ( c ` |X ) log pcoarse ( c ` |z ) ] 1Water , forest , field , and impervious surfaces . 2For example , the label “ water ” is in fact found in the “ Developed ( medium intensity ) ” in around 1 % of such pixels in Maryland , and the frequency of impervious surfaces may not lie between 50 % and 79 % due to occasional misclassification and misalignment in the dataset . and seek to maximize D ( pnet , pcoarse ) = ∑ l log pnet ( C ` |X ) , that is , the likelihood of the distribution of labels C that represents the optimal compromise between what the segmentation network expects ( given the image X ) and what the joint distribution dictates ( given the coarse label z ) . In particular , if the distributions pcoarse ( c ` |z ) are also represented as products of Gaussians , i.e. , pcoarse ( { c ` } |z ) = ∏ ` N ( c ` ; η ` , z , ρ2 ` , z ) , ( 5 ) then C ` = ρ2 ` , zµ ` + σ 2 ` η ` , z σ2 ` + ρ 2 ` , z , ( 6 ) D ( pnet , pcoarse ) = log pnet ( C ` |X ) ∼ const− 1 2 ( µ ` − η ` , z ) 2 σ2 ` + ρ 2 ` , z − 1 2 log 2πσ2 ` , ( 7 ) a function that is differentiable in the output of the label counting layer θ = { µ ` , σ2 ` } . In turn , these are differentiable functions of the input image X . Thus , the network can be trained to minimize the sum of the expressions ( 7 ) over all blocks k in the input image3 .

Summary: This paper presents a deep-learning based method for super-resolving low-resolution labels into high-resolution labels given the joint distribution between those low- and high- resolution labels. This is useful for many semantic segmentation tasks where high-resolution ground truth data is hard and expensive to collect. Its main contribution is a novel loss function that allows to minimize the distance between the distribution determined by a set of model outputs and the corresponding distribution given by low-resolution label over the same set of outputs. The paper also thoroughly evaluates the proposed method for two main tasks, the first being a land cover mapping task and the second being a medical imaging problem.