Meta-Learning to Guide Segmentation

1 INTRODUCTION . Many tasks of scientific and practical interest require grouping pixels , such as cellular microscopy , medical imaging , and graphic design . Furthermore , a single image might need to be segmented in several ways , for instance to first segment all people , then focus on a single person , and finally pick out their face . Learning a particular type of segmentation , or even extending an existing model to a new task like a new semantic class , generally requires collecting and annotating a large amount of data and ( re- ) training a large model for many iterations . Interactive segmentation with a supervisor in-the-loop can cope with less supervision , but requires at least a little annotation for each image , entailing significant effort over image collections or videos . Faced with endless varieties of segmentation and countless images , yet only so much expertise and time , a segmentor should be able to learn from varying amounts of supervision and propagate that supervision to unlabeled pixels and images . We frame these needs as the problem of guided segmentation : given supervision from few or many images and pixels , collect and propagate this supervision to segment any given images , and do so quickly and with generality across tasks . The amount of supervision may vary widely , from a lone annotated pixel , millions of pixels in a fully annotated image , or even more across a collection of images as in conventional supervised learning for segmentation . The number of classes to be segmented may also vary depending on the task , such as when segmenting categories like cats vs. dogs , or when segmenting instances to group individual people . Guided segmentation extends fewshot learning to the structured output setting , and the non-episodic accumulation of supervision as data is progressively annotated . Guided segmentation broadens the scope of interactive segmentation by integrating supervision across images and segmenting unannotated images . As a first step towards solving this novel problem , we propose guided networks to extract guidance , a latent task representation , from variable amounts of supervision ( see Figure 1 ) . To do so we meta-learn how to extract and follow guidance by training episodically on tasks synthesized from a large , fully annotated dataset . Once trained , our model can quickly and cumulatively incorporate annotations to perform new tasks not seen during training . Guided networks reconcile static and interactive modes of inference : a guided model is both able to make predictions on its own , like a fully supervised model , and to incorporate expert supervision for defining new tasks or correcting errors , Image Support Features Support Task Representation Guided Inference Annotations Support Image NegativesPositives Support Features Task Encoding Output D ec od er Query Features Conditional Inference Support Annotations Query Image OutputQuery Image Query Features D ec od er NegativePositive Figure 1 : A guide g extracts a latent task representation z from an annotated image ( red ) for inference by fθ ( x̄ , z ) on a different , unannotated image ( blue ) . like an interactive model . Guidance , unlike static model parameters , does not require optimization to update : it can be quickly extended or corrected during inference . Unlike annotations , guidance is latent and low-dimensional : it can be collected and propagated across images and episodes for inference without the supervisor in-the-loop as needed by interactive models . We valuate our method on a variety of challenging segmentation problems in Section 5 : interactive image segmentation , semantic segmentation , video object segmentation , and real-time interactive video segmentation , as shown in 2 . We further perform novel exploratory experiments aimed at understanding the characteristics and limits of guidance . We compare guidance with standard supervised learning across the few-shot and many-shot extremes of support size to identify the boundary between few-shot and many-shot learning for segmentation . We demonstrate that in some cases , our model can generalize to guide tasks at a different level of granularity , such as meta-learning from instance supervision and then guiding semantic segmentation of categories . 2 RELATED WORK . Guided segmentation extends few-shot learning to structured output models , statistically dependent data , and variable supervision in amount of annotation ( shot ) and numbers of classes ( way ) . Guided segmentation spans different kinds of segmentation as special cases determined by the supervision that constitutes a task , such as a collection of category masks for semantic segmentation , sparse positive and negative pixels in an image for interactive segmentation , or a partial annotation of an object on the first frame of a clip for video object segmentation . Few-shot learning Few-shot learning ( Fei-Fei et al. , 2006 ; Lake et al. , 2015 ) holds the promise of data efficiency : in the extreme case , one-shot learning requires only a single annotation of a new concept . The present wave of methods ( Koch et al. , 2015 ; Santoro et al. , 2016 ; Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Bertinetto et al. , 2016 ; Finn et al. , 2017 ; Ravi & Larochelle , 2017 ; Snell et al. , 2017 ) frame it as direct optimization for the few-shot setting : they synthesize episodes by sampling supports and queries , define a task loss , and learn a task model for inference of the queries given the support supervision . While these works address a setting with a fixed , small number of examples and classes at meta-test time , we explore settings where the number of annotations and classes is flexible . Our approach is most closely related to episodically optimized metric learning approaches . We design a novel , efficient segmentation architecture for metric learning , inspired by Siamese networks ( Chopra et al. , 2005 ; Hadsell et al. , 2006 ) and few-shot metric methods ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) that learn a distance to retrieve support annotations for the query . In contrast to existing meta-learning schemes , we examine how a meta-learned model generalizes across task families with a nested structure , such as performing semantic segmentation after meta-learning on instance segmentation tasks . Segmentation There are many kinds of segmentation , and many current directions for deep learning techniques ( Garcia-Garcia et al. , 2017 ) . We take up semantic ( Everingham et al. , 2010 ; Liu et al. , 2011 ) , interactive ( Kass et al. , 1988 ; Boykov & Jolly , 2001 ) , and semi-supervised video object segmentation ( Pont-Tuset et al. , 2017 ) as challenge problems for our unified view with guidance . See Fig . 2 for summaries of these tasks . For semantic segmentation Shaban et al . ( 2017 ) proposes a one-shot segmentor ( OSLSM ) , which requires few but densely annotated images , and must independently infer one annotation and class at a time . Our guided segmentor can segment from sparsely annotated pixels and perform multi-way inference . For video object segmentation one-shot video object segmentation ( OSVOS ) by Caelles Under review as a conference paper at ICLR 2019 tra in bird monitor input image target mask input image + annotation target mask target mask input query input support et al . ( 2017 ) achieve high accuracy by fine-tuning during inference , but this online optimization is too costly in time and fails with sparse annotations . Our guided segmentor is feed-forward , hence quick , and segments more accurately from extremely sparse annotations . Chen et al . ( 2018 ) impressively achieve state-of-the-art accuracy and real-time , interactive video object segmentation by replacing online optimization with offline metric learning and nearest neighbor inference on a deep , spatiotemporal embedding ; however , they focus exclusively on video segmentation . We consider a variety of segmentation tasks , and investigate how guidance transfers across semantic and instance tasks and how it scales with more annotation . For interactive segmentation , Xu et al . ( 2016 ) ; Maninis et al . ( 2018 ) learn state-of-the-art interactive object segmentation , and Maninis et al . ( 2018 ) only needs four annotations per object . However , these purely interactive methods infer each task in isolation and can not pool supervision across tasks and images without optimization , while our guided segmentor quickly propagates supervision non-locally between images . 3 GUIDED SEGMENTATION . Akin to few-shot learning , we divide the input into an annotated support , which supervises the task to be done , and an unannotated query on which to do the task . The common setting in which the support contains K distinct classes and S examples of each is referred to as K-way , S-shot learning ( Lake et al. , 2015 ; Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ) . For guided segmentation tasks we add a further pixel dimension to this setting , as we must now consider the number of support pixel annotations for each image , as well as the number of annotated support images . We denote the number of annotated pixels per image as P , and consider the settings of ( S , P ) -shot learning for various S and P . In particular , we focus on sparse annotations where P is small , as these are more practical to collect , and merely require the annotator to point to the segment ( s ) of interest . This type of data collection is more efficient than collecting dense masks by at least an order of magnitude ( Bearman et al. , 2016 ) . Since segmentation commonly has imbalanced classes and sparse annotations , we consider mixed-shot and semi-supervised supports where the shot varies by class and some points are unlabeled . This is in contrast to the standard few-shot assumption of class-balanced supports . We define a guided segmentation task as the set of input-output pairs ( Ti , Yi ) sampled from a task distribution P , adopting and extending the notation of Garcia & Bruna ( 2018 ) . The task inputs are T = { { ( x1 , L1 ) , . . . ( xS , LS ) } ∪ { x̄1 , . . . , x̄Q } ; xs , x̄q ∼ Pl ( RN ) } Ls = { ( pj , lj ) : j ∈ { 1 . . P } , l ∈ { 1 . .K } ∪ { ∅ } } where S is the number of annotated support images xs , Q is the number of unannotated query images x̄q , and Ls are the support annotations . The annotations are sets of point-label pairs ( p , l ) with |Ls| = P per image , where every label l is one of the K classes or unknown ( ∅ ) . The task outputs , that is the targets for the support-defined segmentation task on the queries , are Y = ( y1 , . . . , yQ ) , yq = { ( pj , lj ) : pj ∈ x̄q } Our model handles general way K , but for exposition we focus on binary tasks with K = 2 , or L = ( + , − ) . We let Q = 1 throughout as inference of each query is independent in our model . 4 GUIDED NETWORKS . Our approach to guided segmentation has two parts : ( 1 ) extracting a task representation from the semi-supervised , structured support and ( 2 ) segmenting the query given the task representation . We define the task representation as z = g ( x , + , − ) , and the query segmentation guided by that Inference Pool representation as ŷ = f ( x̄ , z ) . The design of the task representation z and its encoder g is crucial for guided segmentation to handle the hierarchical structure of images and pixels , the high and variable dimensions of images and their pixelwise annotations , the semi-supervised nature of support with many unannotated pixels , and skewed support distributions . We examine how to best design the guide g and inference f as deep networks . Our method is one part architecture and one part optimization . For architecture , we define branched fully convolutional networks , with a guide branch for extracting the task representation from the support with a novel late fusion technique ( Section 4.1 ) , and an inference branch for segmenting queries given the guidance ( Section 4.2 ) . For optimization , we adapt episodic meta-learning to image-to-image learning for structured output ( Section 4.3 ) , and increase the diversity of episodes past existing practice by sampling within and across segmentation task families like categories and instances .

Summary: To my knowledge, this paper is probably the first one to apply few-shot learning concept into high-level computer vision tasks. In this paper's sense, segmentation. It proposes a general framework to few from the very few sample, extract a latent representation z, and apply it to do segmentation on a query. Cases of semantic, interactive and video segmentation are applied. Experiments are very thorough.