Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning
Authors: Tsung-Wei Ke
Jyh-Jing Hwang
Stella Yu
Venue: ICLR 2021 Poster (2021)
Keywords: weakly supervised representation learning
representation learning for computer vision
metric learning
semantic segmentation

Abstract: Weakly supervised segmentation requires assigning a label to every pixel based on training instances with partial annotations such as image-level tags, object bounding boxes, labeled points and scribbles. This task is challenging, as coarse annotations (tags, boxes) lack precise pixel localization whereas sparse annotations (points, scribbles) lack broad region coverage. Existing methods tackle these two types of weak supervision differently: Class activation maps are used to localize coarse labels and iteratively refine the segmentation model, whereas conditional random fields are used to propagate sparse labels to the entire image.

We formulate weakly supervised segmentation as a semi-supervised metric learning problem, where pixels of the same (different) semantics need to be mapped to the same (distinctive) features. We propose 4 types of contrastive relationships between pixels and segments in the feature space, capturing low-level image similarity, semantic annotation, co-occurrence, and feature affinity They act as priors; the pixel-wise feature can be learned from training images with any partial annotations in a data-driven fashion. In particular, unlabeled pixels in training images participate not only in data-driven grouping within each image, but also in discriminative feature learning within and across images. We deliver a universal weakly supervised segmenter with significant gains on Pascal VOC and DensePose. Our code is publicly available at https://github.com/twke18/SPML.

Published as a conference paper at ICLR 2021
UNIVERSAL WEAKLY SUPERVISED SEGMENTATION BY PIXEL-TO-SEGMEN CONTRASTIVE LEARNING
Tsung- Wei Ke
Jyh-Jing Hwang
Stella X. Yu
UC Berkeley ICSI
(twke, jyh, stel layu) @berkell edu
ABSTRACT
Weakly supervised segmentation requires assigning label to every pixel based on training instances with partial annotations such as image-level tags, object bound- ing boxes, labeled points and scribbles. This task challenging, as coarse an- notations (tags, boxes) lack precise pixel localization whereas sparse annotations (points, scribbles) lack broad region coverage. Existing methods tackle these two types of weak supervision differently: Class activation maps used to localize coarse labels and iteratively refine the segmentation model, whereas conditional random fields are used to propagate sparse labels to the entire image.
We formulate weakly supervised segmentation as a cemi-supervised metric learn- ing problem, where pixels of the same (different) semantics need to be mapped the same (distinctive) features. We propose types contrastive relationships between pixels and segments in the feature space, capturing low-level image simi- larity, semantic annotation, co-occurrence, and feature affinity. They act as priors; the pixel-wise feature can be learned from training images with any partial anno- tations data-driven fashion. particular, unlabeled pixels training images participate not only in data-driven grouping within each image, but also in discrim- inative feature learning within and across images. We deliver a universal weakly supervised segmenter with significant gains on Pascal VOC and DensePose. Our code is publicly available at fhtttts://// / com/ twke 8/MMM
INTRODUCTION
Consider the task of learning semantic segmenter given sparsely labeled training images (Fig. I Each body part is labeled with a single seed pixel and the task is segment the entire person
weak supervision
image
baseline
ours
ground-truth
Figure 1: Our task learns a segmenter given partially labeled training images and applies it to test images. common baseline propagate labels within an image based on feature similarity. We model it as semi-supervised metric learning and learn the pixel-wise feature by contrasting it within and across images. Our results are fuller and more accurate, approaching the ground-truth.

---

Published as a conference paper at ICLR 2021
image
image tags
bounding boxes
labeled points
scribbles
Person Motorbike
SOTA methods
CAM + refine box-wise CAM
CRF loss
CRF loss
our method
single moxeleto-segment contrastive learning loss formulation
our relative gain
+8.6%
+4.7%
+24.7%
+1.4%
We propose unified framework for weakly supervised semantic segmentation with dif- ferent types of annotations. We demonstrate consistent performance gains compared to the state-of- the-art (SOTA) ) methods: Chang et al. (2020) for image tags, Song et al. (2019) for bounding boxes, and Tang et al. (2018b) for points and scribbles. For tags and boxes, Class Activation Maps (CAM) (Zhou et al. 2016) are often used to localize semantics an initial mask and iteratively refine the segmentation model, whereas for labeled points and scribbles, Conditional Random Fields (CRF) used to propagate semantic labels to unlabeled regions based on low-level image similarity.
Figure 2:
by individual body parts, even though the ground-truth segmentation not known during training. This task is challenging, not only single body part could contain several visually distinctive areas (e.g., head consists of eyes, nose, mouth, beard), but two adjacent body parts could also have the same visual appearance (e.g., upper arm, lower arm, and hand have the same skin appearance). Once the segmenter is learned, it can be applied to a test image without any annotations.
This task belongs to family of weakly supervised segmentation problems, the goal of which is to assign a label to each pixel despite that only partial supervision is available during training. It addresses the practical issue of learning segmentation from minimum annotations. Such weak super- vision takes many forms, e.g., image tags (Kolesnikov & Lampert A01 & Kwak 2018 Huang
al. 2018 Lee et al.) 2019). bounding boxes (Dai et al.) 2015 Khoreva et al.) [017 Song al.) 2019 keypoints (Bearman et al. 2016), and scribbles (Lin et al. TTng et al.) [0 8a b). Tags
and boxes are coarse annotations that lack precise pixel localization whereas points and scribbles are sparse annotations that lack broad region coverage.
Weakly supervised semantic segmentation can regarded as a semi-supervised pixel classification problem: Some pixels or pixel sets have labels, most don't 't, and the key is how to propagate and refine annotations from coarsely and sparsely labeled pixels to unlabeled pixels.
Existing methods tackle two types of weak supervision differently: Class Activation Maps (CAM) Zhouu et al. 2016) are used to localize coarse labels, generate pseudo pixel-wise labels, and iter- atively refine the segmentation model, whereas Conditional Random Fields (CRF) (Krähenbühl & Koltun 2011) are used to propagate sparse labels to the entire image. These ideas can be in- corporated
additional unsupervised loss on the feature learned for segmentation (Tang al.) 2018b): While labeled pixels receive supervision, unlabeled pixels in different segments shall have distinctive feature representations.
We propose a Semi-supervised Pixel-wise Metric Learning (SPML) model that can handle all these weak supervision varieties with a single pixel-to-eegment contrastive learning formulation (Fig. 2 Instead of classifying pixels, our metric learning model learns a pixel-wise feature embedding based on common grouping relationships that can be derived from any form of weak supervision.
Our key insight is to integrate unlabeled pixels into both supervised labeling and discriminative feature learning. They shall participate not only in data-driven grouping within each image, but also in discriminative feature learning within and more importantly across images. Intuitively, labeled pixels receive supervision not only for themselves, but also for their surround pixels that share visual similarity. On the other hand, unlabeled pixels are not just passively brought into discriminative learning induced by sparsely labeled pixels, they themselves are organized based on bottom-up grouping cues (such grouping by color similarity and separation by strong contours). When they are examined across images, repeated patterns of frequent occurrences would also form a cluster that demand active discrimination from other patterns.
2

---

Published as a conference paper at ICLR 2021
We capture the above insight in single pixel-wise metric learning objective for segmentation, the goal of which is to map each pixel into a point in the feature space so that pixels in the same (different) semantic groups close (far) in the feature space. Our model extends SegSort (Hwang) et al. 2019) from its fully supervised and unsupervised segmentation settings to a universal weakly- supervised segmentation setting. With single consistent feature learning criterion, such model sorts pixels discriminatively within individual images and sorts segment clusters discriminatively across images, both steps minimizing the same feature discrimination loss.
Our experiments on Pascal VOC (Everingham et al. 2010) and DensePose (Alp Güler et al. 2018 demonstrate consistent gains over the state-of-the-ar (SOTA), and the gain is substantial especially for the sparsest keypoint supervision.
RELATED WORK
Semi-supervised learning. Weston et al.) (012)) treats it as a joint learning problem with both labeled and unlabeled data. One way is capture the underlying structure of unlabeled data with generative models (Kingma et al.) 2014 Rasmus et al. 2013). Another way to regularize feature
learning through a consistency loss, e.g., adversarial ensembling (Miyato et al. 2018)) imitation
learning and distillation (arvainen & Valpola, 2017). cross-view ensembling (Clark et al. 2018
These methods are most related to transductive learning (Joachims) 2003 Zhou et al. [2004] Fergus
et al. 2009 Liu et al. 2019), where labels are propagated to unlabeled data via clustering in the pre- trained feature space. Our work does transductive learning in adaptively learned feature space.
Weakky-superrised semantic segmentation. Partial annotations include scribbles (Lin et al. 2016
Tang et al.
2018a b.
Wang al., (2019), bounding boxes (Dai et al. 2015:: Khoreva et al.
2017
Song et al.
2019), points (Bearman et al. 2016), or image tags (Papandreou et al. 2015; Kolesnikov
& Lampert| 2016; & Kwakk 2018 Huang al. 2018 [i et al.
[0188 Lee al. 2019 Shimoda
& Yanai |an
et 20.1 Yao & Gong |020 Chang et al.
2020 Araslanov & Roth [2020]
Wang et al.] [020 Fan et al.) 2020, Sunn al. 2020). Xu et al.
(2015) formulates all types of
weak supervision as linear constraints on SVM. Papandreou et al.
2015
bootstraps segmentation
predictions via EM-optimization. Recent works (Lin et al.) 2016 Kolesnikov & Lampert, [016]]
Pathak et al.] 2015) typically use CAM (Zhou et al. 2016)) to obtain an initial dense mask and then train a model iteratively. GAIN et et al.
2018) utilizes image tags or bounding boxes to refine these class-specific activation maps. Sun et al. (2020) considers within-imag relationships and explores
the idea of co-segmentatiin. Fan et al. (2020) estimates the foreground and background for each category, with which the network learns generate more precise CAMs. Regularization is enforced at either the image level (Lin et al.] 2016); Kolesnikov & Lampert 2016] Pathak et al., (013)) or the feature level (Tang al. 2018a b) to produce better dense masks. We incorporate this concept into adaptive feature learning and train the model only once. All types of weak annotations are dealt with in single contrastive learning framework.
Non-parametrie segmentation. Prior to deep learning, non-parametric models (Russell et al. 2009
Tighe& & Lazebnik 2010 Liu et al. 2011) usually use designed features with statistical or graphical models to segment images. Recently, inspired by non-parametric models for recognition (Wu et al., 2018b a). SegSort (Hwang et al. 2019) captures -e-egmeet relationships via a wixel-wise embedding and develops the first deep non-parametric semantic segmentation for supervised and unsupervised settings. Building upon SegSort, our work has the flexibility a non-parametric model at capturing data relationships and modeling subclusters within a category.
3 SEMI-SUPERVISED PIXEL-WISE METRIC LEARNING METHOD
Metric learning develops a feature representation based on data grouping and separation cues. Our method (Fig. segments image by learning pixel-wise embedding with a contrastive loss between pixels and segments: For each pixel i, we learn a latent feature o()) such that i is close to its positive segments (exemplars) and far from its negative ones in that feature space.
In the fully supervised setting, we can define pixel i's positive and negative sets, denoted by C+ and C- respectively, as pixels in the same (different) category. However, this idea is not applicable
3

---

Published as a conference paper at ICLR 2021
oversegmentatiion with embeddings
image
Embedding
Space
pixel
segmenfation CNN
Negative
Positive
Figure 3: Overall method diagram. We develop pixel-wise embeddings with contrastive learning between pixels and segments. We derive various forms of positive and negative segments for each pixel. Our goal to attract (blue inward arrows) the pixel with positive segments, while repelling (red outward arrows) it from negative segments in the feature space.
(a) low-level image similarity
(b) semantic annotations
(c) semantic co-occurrence
(d) feature affinity
sofa
cat sofa table
chair
person
horse
Figure 4: Four types of pixel-to-segment attraction and repulsion relationships. A pixel is attracted (repelled by) segments: a) of similar (different) visual appearances such as color texture, b) of the same (different) class labels, с) in images with common (distinctive) labels, d) of nearby (far-away) feature embeddings. They form different positive and negative sets.
to weakly- or un-supervised settings where the label not available on every pixel. In the laheled points setting, C+ and (- would only contain few exemplars according to the sparse pixel labels.
Our basic idea is enlarge the sets of C+ and C- to improve the feature learning efficacy. By exploring different relationships and assumptions in the image data, we are able to generate abundant positive and negative segments for any pixel at the same time, providing more supervision in latent feature space. We propose four types of relationships between pixels and segments (Fig.
1. Low-level image similarity: We impose spatial smoothness prior the pixel-wise feature to keep pixels together in visually coherent regions. The segment pixel belongs to based on low-level image cues a positive segment to pixel any other segments are negative ones.
2. Semantic annotation: We expand the semantics from labeled points and scribbles to pseudo- labels inferred from image- or box-wise CAM. The label of segment can be estimated by majority vote among pixels; if it is the same as pixel i's, the segment is positive segment to i.
expand the semantics by assuming that pixels in similar semantic contexts tend to be grouped together. If a segment appears in image that shares any of the semantic classes as pixel i's image, is a positive segment to and otherwise negative one.
Semantic co-occurrence:
4. Feature affinity: We impose featural smoothness prior assuming that pixels and segments of the same semantics form cluster in the feature space. We propagate the semantics within and across images from pixel i its closest segment in the feature space.

---

Published as a conference paper at ICLR 2021
3.1 PIXEL-TO- SEGMENTT CONTRASTIVE GROUPING RELATIONSHIPS
Our goal is to propagate known semantics from labeled data C to unlabeled data 24 with the afore- mentioned priors. C. and u denote the sets of segment indices respectively. We detail how augment positive / negative segment sets using both C and 24 for each type of relationships (Fig.
Low-level image similarity. To propagate labels within visually coherent regions, we generate a low-level over-segmennttion. Following SegSort (Hwang et al. 2019). we use the HED contour detector (Xie & Tu, 2013) (pre-trained on BSDS5Ó0 dataset (Arbelaez al. 20TO)) and gPb-owt- uem (Arbelaez et al., (2100) to generate a segmentation without semantic information. We define i's positive and negative segments as i's own segment and all the other segments, denoted as V and V- respectively. We only consider segments in the same image as pixel i's. We align the contour-based over-segmeentaions with segmentations generated by K-Means clustering as in SegSort.
Semantic annotation. Image tags and bounding boxes do not provide pixel-wise localization. We derive pscudo labels from image- or box-wise CAM and align them with oversegmentaaions induced by the pixel-wise feature. Pixel i's positive (negative) segments are the ones with the same (different) semantic category, denoted by C+ and C respectively. We ignore all the unlabeled segments.
Semantic co-occurrence. Semantic context characterizes the co-occurrences of different objects, which can be used as a prior to group and separate pixels. We define semantic context as the union of object classes in each image. Even without the pixel-wise localization of semantic labels, can leverage semantic context to impose global regularization on the latent feature: The feature should separate images without any overlapping object categories.
Let O+ (O-) denote the set of segments in images with (without) overlapping categories as pixel i's image. That is, if the image of pixel and another image share any semantic labels (Fig. 4e: (cat, sofa, table, chair} for the pixel in the Row 2 image vs. {sofa] for the Row 1 image), then all the segments from that image are positive segments i and included in O+; otherwise they are considered negative segments in O (Fig. 4e: all the segments in the Row 3 image). In particular, all the segments in pixel i's image are in O+ of i. This semantic context relationship does not require localized annotations yet imposes regularization on pixel feature learning.
Feature affinity. Our goal to learn a pixel-wise feature that indicates semantic segmentation. It is thus reasonable to assume that pixels and segments of the same semantics form cluster in the feature space, and we reinforce such clusters with featural smoothness prior: We find nearest neighbours in the feature space and propagate labels accordingly.
Specifically, we assign a semantic label to each unlabeled segment by finding its nearest labeled segment in the feature space. We denote this expanded labeled set by C. For pixel i, we define its positive (negative) segment set Č+ (C)) according to whether a segment has the same label as i.
Our feature affinity relationship works best when: 1) the original labeled set is large enough to cover the feature space, 2) the labeled segments are distributed uniformly in the feature space, and 3) the pixel-wise feature already encodes certain semantic information. We thus only apply to DensePose keypoint annotations in our experiments, where each body part is annotated by a point.
3.2 PIXEL-WISE METRIC LEARNING LOss
SegSort (Hwang et al. 2019) is an end-to-end segmentation model that generates pixel-wise fea- ture map and resulting segmentation. Assuming independent normal distributions for individual segments, SegSort seeks a maximum likelihood estimation of the feature mapping, so that the fea- ture induced partitioning in the image and clustering across images provide maximum discrimination among segments. During inference, the segment label is predicted by K-Nearest Neighbor retrievals.
The feature induced partitioning in each image is calculated via spherical K-Means cluster- ing (Banerjee et al.) 2005). Let e; denote the feature vector at pixel i, which contains the mapped feature o(t) and i's spatial coordinates. Let 2; denote the index of the segment that pixel i belongs to, R. the set of pixels in segment s, and μa, the segment feature calculated as the spherical cluster centroid of segment s. In the pnneccaatio-Maximizaan
(EM) procedure for spherical K-means, the E-step calculates the most likely segment pixel i belongs to: = arg max, uue,, and the M-Step E.CR e,
updates the segment feature as the mean pixel-wise feature: μ. 1 1EvER e.1l
5

---

Published as a conference paper at ICLR 2021
image
semantic annotation
semantic
Positive
Negelivve
Ignore
imge similerity
affinity
a) training data
b) existing methods
c) our SPML
Figure 5: Our method uses labeled and unlabeled portions of the training data more extensively. a) Training images and their labeled scribbles are sparse and incomplete. b) Existing methods train pixel-wise classifier using only labeled pixels and propagate labels within each image. c) Our method leverages four types of pixel-to-sepee -eemmmett semantic relationships to augment the labeled sets, includes unlabeled pixels (fuller segments than just thin scribbles) and unlabeled segments (e.g. desk outlined in magenta), forms dynamic contrastive relationships between segments (e.g. the desk can be positive, negative, to be ignored to the sofa in different relations.
Let s denote the resulting segment that pixel belongs to per spherical clustering. The posterior probability of pixel in segment can be evaluated over the set of all segments S as:
exp(1 μ e;)
p sle., Eres exp(x p.,e,)
where is concentration hyper-parameter. SegSort minimizes the negative log- likelihood loss:
exp(k ec)
LieeSoot (i) log s/e,, log Eies exp(x
(2)
SegSort adopts soft neighborhood assignment (Goldberger et al.) 2005) further strengthen the grouping of same-category segments. Let C+ (C-) denote the index set of segments in the same (different) category pixel except the segment belongs We have:
EiEc+ exp(mu(e)
LseeSort + -,,,t,-=== -- - Σ p(z: = tle, log Electue- xxp(xuuee))
(3)
+EC+
For our weakly supervised segmentation, the total pixel-to-segment contrastive loss for pixel con- sists of 4 terms, for each of the 4 pixel-to-segment attraction and repulsion relationships:
L(1) Looggotttt + (i, V+ V-) Ac (i,Ct,C*)
+ Is 01,00,0000 (0) + AA SeggSort+ (1, (1,C+,C-)),
where Ao Fig. shows how metric learning method utilizes labeled and unlabeled pixels and segments more extensively than existing classification methods: Our pseudo-labeled sets are fuller than labeled thin scribbles and include unlabeled segments; there other than semantic annotations; our segments participate in contrastive learning with dynamic roles in different relations. By easily integrating a full range of pixel-to-see -o-smmmett attraction and repulsion relationships from low-level image similarity to mid-level feature affinity, and to high-level semantic co-occurrence, we go far beyond the direct supervision from semantic annotations.
more relationships
EXPERIMENTS ON PASCAL VOC AND DENSEPOSE
Datasets. Pascal VOC 2012 Everingham et al. (2000) includes 20 object categories and one back- ground class. Following Chen et al. (2017). we use the augmented training set with 10,582 im- ages and validation set with 1,449 images. We the scribble annotations provided by Lin et al.)

---

Published as a conference paper at ICLR 2021
Dataset
Annotation
X
o
AA
Pascal
image tags
0.3
1.0
0.0
boxes
0.3
1.0
0.0
points scribbles
1.0
1.0
0.0
0.1
0.5
0.0
DensePose
points
0.1
0.0
0.5
Table 1: Hyper-parameters for different types of annotations on Pascal and DensePose dataset.
2016) for training. DensePose (Alp Güler et al., 2018) a human pose parsing dataset based on MSCOCO (Lin et al., 2014). The dataset is annotated with 14 body part classes, We extract the keypoints from the center of each part segmentation. The training includes 26,437 images and use minival2014 set for testing, which includes 1,508 images. See Appendix for more details.
Architecture, training and testing. For the experiments on PASCAL VOC, we base our archi- tecture DeepLab (Chen et al. (017) with Re (He et al.
2016) as the backbone network. For the experiments on DensePose, we adopt PSPNet Zhao et al. 2017) as the backbone network.
Our models pre-trained ImageNet (Deng et al., 2009) dataset. See Appendix for details our inference procedure and hyper-paramette selection for training and testing.
For each type of annotations and dataset, formulate four types of pixel-to-segment contrastive relationships and jointly optimize them in a single pixel-wise metric learning framework (Fig. 3) Table the data-driven selection of hyperparameters Är, Åo and λ for different task settings.
Pascal: Image tag annotations. Table shows that, without using additional saliency labels, our method outperforms existing methods with saliency 4.4%, and those without saliency by 5.1%.
Pascal: Bounding box annotations. Table 2 shows that, with the same DeepL.ab/ResNet101 back- bone network, our method outperforms existing methods by 3.2%.
Pascal: Image lags
Saliency
val
test
Huang et al. (0018)
61.4
63.2
Pascal: Bounding boxes
val
test
Lee et al. (2019)
64.9
65.3
Khoreva al. (2017)
69.4
Zhang al. 2019
66.3
66.5
Song et al. (019)
70.2
Yao & Gong 2022
67.1
67.2
Chang et al.
2020
66.1
65.9
Our SPML
73.5
74.7
Our SPML
69.5
71.6
Table 2: Pascal VOC 2012 dataset with image tag (left) and bounding box (right) annotations.
Pascal: Scribbles
CRF
Full
Weak
WvF
100
Tang
8018a
75.6
72.8
96.3
Tang ct al.
2018a
76.8
74.5
97.0
50
Tang cl al.
2018b
75.6
73.0
96.6
Tang al.
2018b
76.8
75.0
97.7
Wang
2019)
75.6
73.2
96.8
ou 5PML
Wang
2019
76.0
99.0
TTngg et
SPML
74.2
97.5
-4th (2016)
Our SPML
77.3
76.1
98.4
100%
Pascal: Verying acriiiees arrolaaiinn
Table Pascal VOC 2012 dataset using scribble annotations. Left: mloU on validataion (white) and test (gray) set. WvF denotes relative mloU w.r.t full supervision. Right: Relative mloU performance w.r.t full supervision on different lengths of scribbles.
Pascal: Scribble annotations. Table shows that, our method consistently delivers the best per-
formance among methods without or with CRF post-processing We get 74.2% (76.1%) mIoU, achieving 97.5% (98.4%) of full supervision performance in these two categories respectively.
Varying sparsity of scribble and point annotations. Exploiting metric learning with different relationships in the data frees from the classification framework and delivers more
Pascal:

---

Published as a conference paper at ICLR 2021
Person
21
(R) Image
(b) Annotation
(c)
Baseline
(d) Our SPMI.
(c) Full Supervision
(1) Ground Truth
Figure 6: Our results on Pascal and DensePose under various weak supervision settings are consis- tently better aligned with region boundaries and visually closer to fully supervised counterparts.
(a)
(hj Besdine
(c) Semmatic Arnstatonn
Image Simiarly
Foruar affiniy
(d) Full experisii
(c) Grund Trch
Figure 7: Our segmentation results get better with more types of regularizations, We compare visual results by adding more regularizations. As we introduce more relationships for regularization, observe significant improvement our results are visually closer to fully supervised counterparts.
powerful approach that requires fewer annotations. Table 3 shows that, scribbles from 100%, 80%, 50%,30% to 0% (points), we reach 97.5% 97.5%. 96.3%, 96.5% and 93.7% of full supervision performance. Compared to the full scribble annotations, our accuracy only drops 3.7% with point labels and is significantly better than the baseline.
shorten the length of
Point annotations. We train our baseline using the code released by Tang et al. (2018b). Table g shows that, our method without CRF post-processing out- performs the baseline by 12.9% mloU, reaching 77.1% of full supervision performance with only point supervision.
DensePose:
DensePose: Points
mIoU
WvF
Tang et al. (2018b)
31.3
51.9
Our SPMI.
44.2
77.1
Visual quality and ablation study. Fig. 6 shows that results are better aligned with region boundaries and visu- ally closer to fully-supervised counterparts. Fig. shows that our results improve significantly with different relationships for more regularization. See Ap- pendix for more details and ablation studies.
Table DensePose minival 2014 set.
Summary. We propose a novel wcakly-supervised semantic segmentation method via Semi- supervised Pixel-wise Metric Learning. based four common types of pixel-to-segment attrac- tion and repulsion relationships. It is universally applicable to various weak supervision settings, whether the training images are coarsely annotated by image tags or bounding boxes, sparsely annotated by keypoints or scribbles. Our results on PASCAL VOC and DensePose show consistent and substantial gains over SOTA, especially for the sparsest keypoint supervision.
A cknowledgements. This work was supported, in part. by Berkeley Deep Drive and Berkeley AI Research Commons with Facebook. This work used the Extreme Science and Engincering Dis- covery Environment (XSEDE), which is supported by National Science Foundation grant number ACI- -51-154 Specifically, it used the Bridges system, which supported by NSF award number ACI- (CI-1445 at the Pittsburgh Supercomputing Center (PSC).
8

---

Published as a conference paper at ICLR 2021
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4981 1-1990, 2018.
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7297-7306, 2018.
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2020.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hi- erarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898-91 2010.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hyper- sphere using von mises-fisher distributions. Journal of Machine Learning Rexeurch, 2005.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference computer vision, pp. 549-565. Springer, 2016.
Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming- Hsuan Yang. Weakly-supervised semantic segmentation via sub-category exploration. Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2020.
Liang- Chic-Chhchhhhh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2017.
Kevin Clark, Thang Luong, and Quoe V Le. Cross-view training for semi-supervissed arXiv preprint arXiv:1809.08
2018.
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolu- tional networks for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1635-1643, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Junsong Fan, Zhaoxiang Zhang, Chunfeng Song. and Tieniu Tan. Learning integral objects with intra-class discriminator for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Rob Fergus, Yair Weiss, and Antonio Torralba. Semi-supervised learning in gigantic image collec- tions. In Advances in neural information processing systems, pp. 322-530, 2009.
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Ruslan R Salakhutdinov. Neighbourhood components analysis. In NIPS, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Zilong Huang, Xinggang Wang. Jiasi Wang. Wenyu Liu, and Jingdong Wang. Veakly-supervise semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7014-7023, 2018.

---

Published as a conference paper at ICLR 2021
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chich Chen. Segsort: Segmentation by discriminative sorting of segments. In ICCV, 2019.
Thorsten Joachims. Transductive learning via spectral graph partitioning. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp- 290 290-297..0 2003.
Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 876-885, 2017.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Alexander Kolesnikov and Christoph Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In European Conference on Computer Vision, pp. 695-711 Springer, 2016.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In Advances in neural information processing systems, pp. 107-117,70 2011.
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proceedings of the JEEE conference on computer vision and pattern recognition, pp. 5267-5276, 2019.
Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng. Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9215-9223, 2018.
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolu- tional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3159-3167, 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Bin Liu, Zhirong Wu, Han Hu, and Stephen Lin. Deep metric transfer for label propagation with limited annotated data. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparametric scene parsing via label transfer. PAMI, 2011.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and sumi-super learning. IEEE transactions on pattern analysis and machine intelligence, (1): 1979-1993,, 2018.
Nicholas A Nystrom, Michael J Levine. Ralph Z Roskies, and J Ray Scott. Bridges: uniquely flexible hpc resource for new communities and data analytics. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructuue, pp. 1-8, 2015.
George Papandreou, Liang-Chier Chieh Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi- supervised learning of deep convolutional network for semantic image segmentation. In Pro- ceedings of the IEEE international conference on computer vision, pp. 1742-1750, 2015.
Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1796-1804, 2015.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Hami Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in neural information processing systems, pp. 3546-3554, 2015.
10

---

Published as a conference paper at ICLR 2021
Bryan Russell, Alyosha Efros, Josef Sivic, Bill Freeman, and Andrew Zisserman. Segmenting scenes by matching image composites. In NIPS, 2009.
Wataru Shimoda and Keiji Yanai. Self-supervised difference detection for weakly-supervised se- mantic segmentation. In Proceedings of the IEEE International Conference Computer Vision, pp. 5208-5217, 2019.
Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region mask- ing and filling rate guided loss for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3136-3145, 2019.
Guolei Sun, Wenguan Wang. Jifeng Dai, and Luc Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In ECCV, 2020.
Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Nor- malized cut loss for -saeey-uuee enn segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1818-1827, 2018a.
Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for weakly-supervised enn segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 507-522, 2018b.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Joseph Tighe and Svetlana Lazebnik. Superparsing: scalable nonparametric image parsing with superpixels. In ECCV, 2010.
John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D Peterson, et al. Xsede: accelerating scientific discovery. Computing in science & engineering, 16(5):62-74, 2014.
Bin Wang, Guojun Qi, Sheng Tang, Tianzhu Zhang. Yunchao Wei, Linghui Li, and Yongdong Zhang. Boundary perception guidance: A scribble-supervieed semantic segmentation approach. International Joint Conference Artificial Intelligence, 2019.
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. Proceedings of the IEEE International Conference Computer Vision, 2020.
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural networks: Tricks of the trade, pp. 639-655. Springer, 2012.
Zhirong Wu, Alexei A Efros, and Stella Yu. Improving generalization via scalable neighborhood component analysis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 685-701, 2018a.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018b.
Saining Xie and Zhuowen Tu. Holistically-neste nested edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1403-1403, 2015.
Jia Xu, Alexander G Schwing, and Raquel Urtasun. Learning to segment under various forms of weak supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3781-3790, 2015.
Qi Yao and Xiaojin Gong. Saliency guided self-attention network for weakly and semi-supervised semantic segmentation. IEEE Access, 8:14 1441-11442) 2020.
Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, and Kaizhu Huang. Reliability does matter: An end-to-end weakly supervised semantic segmentation approach. arXiv: 191 1. 1.08039, 2019.
arXiv preprint
11

---

Published as a conference paper at ICLR 2021
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881-2890, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929. 2016.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learn- ing with local and global consistency. In Advances in neural information processing systems, pp- 321-328, 2004.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
We propose single pixel-to-segment contrastive learning loss formulation for weakly supervised semantic segmentation. We explore different types of visual relationships to group and separate pixels within and across images. We demonstrate state-of-the-arr performance using proposed method with different types of annotations. Here, we include more details on the following aspects:
We present the visual results of our method in A.1
We showcase the semantic cues generated by CAM for image tag and bounding box anno- tations
A.2
We illustrate the data pre-processing used for DensePose dataset in A.3
We describe the details our experimental settings, syperr-parameters and inference pro- cedure
A.4
We present the ablation study regarding hyper- paremeerrmeeerss in A.5
We present mloU performance with varying sparsity scribble annotations on Pascal dataset in A.6
We present per-category results with Pascal and DensePose dataset in A.7
A.1 VISUALIZATION
We present the visual results on VOC (with image tags, bounding boxes and scribbles) and Dense- Pose (with keypoints) dataset in figure We observe that our segmentation results are better aligned with image boundary. When visual evidence is prominent, weakly-supervised results better than the fully-supervised counterpart.
We then demonstrate the efficacy of each visual relationship figure 9 notation, low-level image similarity and feature affinity progressively, we observe consistent im- provement of our results. The predicted segmentation becomes more coherent and better aligned with image boundary. We lastly showcase that our method implicitly encodes semantic contexts. In figure TO We observe that retrieved segments appear in the similar semantic context as the query segments. For examples, given a bottle next to desktop, our model retrieves bottles also next to desktop; a set of sofas in a living room can be retrieved using one sofa query example; screens of desktop can also be retrieved likewise.
By adding semantic an-
A.2 SEMANTIC ANNOTATIONS
Since image tag and bounding box annotations do not provide any of precisely localized semantic information, we adopt CAM (Zhou et al. 2016) to produce localized semantic cues. Without using additional saliency labels, Me be the activation map of class c.
the classifier trained by Wang al. (2020) generate CAM. Let
For image tag annotations, we follow Ahn Kwak (2018) to normalize Me of the entire image Mc within the range between 0 and 1, where M mas. M. The background confidence May can then be estimated by Mbg = (1 max. Mc)", where a the hyper-paramest adjusting background confidence. In our experiments, we set to 6 and confidence threshold to 0.2. The low-confidence pixels are considered as unlabeled regions.
For bounding box annotations, we simply normalize the CAM logits within each bounding box to the range between 0 and We then set confidence threshold 0.5 for selecting foreground pixels and unlabeled regions. We restrict all the regions outside bounding boxes "background". See figure for more visual examples.
A.3 DATA PRE-PROCEE PROCESSIIN FOR DENSEPOSE DATASET
We next illustrate our pre-processing to generate training labels given keypoint annotations in Dense- Pose dataset. As shown in figure 12 first assume Gaussian heat map from every keypoint. By
13

---

Published as a conference paper at ICLR 2021
thresholding, we derive 3 regions from every Gaussian blob: labelled, unknown and background gion. In labelled region, pixels are annotated as each body part. We then propagate labels, including background class, pixels in the unknown region. The std of Gaussian heat map is estimated from instance size, and we use ground-truth information in our paper.
A.4 HYPER- PARAMETRRS AND EXPERIMENTAL SETUP
Architecture and training. For all the experiments on VOC, we hase our architecture as DeepLab (Chen et al. (017)) with ResNet101 (He et al.. 2016) backbone network. For the exper- iments on DensePose dataset, we adopt PSPNet (Zhao et al. (017) as backbone network. We only models pre-trained on ImageNet (Deng et 2009) dataset.
Aeroplane
Person Table
Image
(c) Baseline
SPML
(e) Full Supervision
(i) Ground Truth
Figure 8: Visual comparison of baseline method (c). our SPML (d) and fully-supervised SegSort (c) on VOC and DensePose. On VOC (top rows), our baseline method is based on Lee et al. 2019 Song et al. Tang et al. (2018b) for image tag, bounding box and scribble annotations, respectively. DensePose (bottom rows), our baseline [Tag et al. (2018b). The results from our weakly-supervised model is visually very close to its fully- fully-supervised counterpart, or even better when visual cues are prominent.
14

---

Published as a conference paper at ICLR 2021
(A) Image
(b) Baacine
(c) Seraatic Amosaaion
Image Siniaray
Fesure affinty
(d) Full superviiin
tel Ground Truss
Figure 9: Our segmentation results get better with more types of regularizations, We compare visual results by adding more regularizations. As we introduce more relationships for regularization, observe significant improvement and our results are visually closer to fully supervised counterparts.
We next describe the hyper-parametens used for each experiment. On Pascal VOC dataset, we set "batchsize" to 12 and 16 for scribble point and image tag bounding box annotations. On Dense- Pose dataset, "batchsize" is set to 16. For all the experiments, we train our models with 512 512 "cropsize". " Following Chen et al. (017). we adopt poly learning rate policy by multiplying base learning rate by 1 maga
We set initial learning rate to 0.003, momentum to 0.9. For the hyper-parameters in SegSort framework, we use unit-length normalized embedding of dimension 64 and 32 VOC and DensePose, respectively. We iterate K-Means clustering for 10 iterations and generate 36 and 144 clusters on VOC and DensePose dataset. We set the concentration parameter к different values for semantic annotation, low-level image similarity. semantic co-occurrence and feature affinity, respectively. Moreover, År, λo and A are set to different values according to different types of annotations and datasets. Àc set to 1 among all the experiments. The detailed hyper-parameter settings are summarized in table[5] We train for 30k and 45k iterations on VOC and DensePose dataset for all the experiments. We use additional memory banks cache up previous 2 batches. For conducting experiments, we take advantage of XSEDE infrastructure (Towns et al., 2014) that includes Bridges resources (Nystrom et al. 2015).
Figure 10: Visual examples of nearest neighbor segment retrievals. We observe that retrieved seg- ments (right) appear in the similar semantic context as the query segments (left). For examples, given a bottle next to a desktop, our model retrieves bottles also next to desktop.
15

---

Published as a conference paper at ICLR 2021
(a) Image
(b) CAM (label)
(c) CAM (box)
(d) Scribble
(c) Ground Truth
Figure 11: Visual examples of semantic annotations used on VOC. For image tag bounding box annotation, use the classifier trained by Wang et al. (2020) to infer CAM as semantic annotation.
These semantic annotations are noisy, which do not precisely localize on the objects.
Inference and testing. fix the learned pixel-wise embedding and train an additional softmax classifier for inference. Iterative training is adopted to bootstrap the semantic segmentation predic- tion. Notably, we do not propagate gradients to the segmentation CNN from the softmax classifier.
For scribbles points bounding boxes, first learn initial softmax classifier S₁ from the cor- responding weak annotations. Following |Ahn & Kwak (2018), we apply random walk to refine the semantic logits M generated by S,. The transition probability matrix T is formulated as follows: where B and y are 20 and respectively. The label propagation given Thi = EE exp(ye;e;
exp((ee
by: = M, where M' denotes refined semantic logits. The random walk process is iterated for 6 times. Next, we obtain the corresponding pseudo labels V.c. arg max M'. The pseudo labels are used train the final softmax classifier S2 for predicting semantic segmentation.
For image tag annotations, we adopt both within within-image and across-image label propagation to gener- ate optimal pseudo labels. Starting with CAM logits M, we conduct within-iman image label propagation thru random walk and obtain refined pseudo labels ) ) Across-image label propagation is carried out by nearest neighbor search thru whole training set. We refer to SegSort (Hwang et al., 2019) for more details. We then obtain refined pseudo labels V and train the initial softmax classifier St. Similarly, we use S₁ to predict pscudo labels ) from the training images. Followed by nearest neighbor search, obtain our final pseudo labels ) and train the final semantic classifier S2.
Figure 12: Preparing training labels DensePose dataset. From left to right are input image, our training labels and ground-truth truth mask. For each keypoint, Gaussian heat map is applied to determine labelled, unknown and background region. The white region denotes unknown pixels, to which we propagate labels from annotated or background region.
16

---

Published as a conference paper at ICLR 2021
The inference procedures for different annotations summarized in algorithm | and respec- tively. For image tags, we adopt multi-scale and horizontally flipping as data augmentation for predicting semantic segmentation. For scribbles / points / bounding boxes, do not employ data augmentation during the final inference.
Dataset
Annotation
R1
A
KC
o
Ko
A
AAA
batchsize
VOC
scribbles
0.1
16
1.0
0.5
12
0.0
12
points
1.0
16
1.0
1.0
0.0
12
boxes
0.3
16
1.0
6
1.0
8
0.0
16
image tags
0.3
16
1.0
1.0
0.0
16
DensePose
points
0.1
16
1.0
0.0
0.5
Table 5: Hyper-parameters for different types of annotations on Pascal and DensePose dataset.
Algorithm 1: Inference procedure for semantic segmentation using scribble point / bounding box annotations.
Input: Fixed mixel-wise embedding of the input image and weak annotations Vweak.
Output: Semantic segmentation prediction Vpred-
/* Train the initial softmax classifier +/
Train the softmax classifier S1 using Vweak-
/* Train the final softmax classifier +/
Predict semantic logits from initial softmax classifier: M S1(e).
Calculate pixel-wise transition probability matrix T from e.
Refine semantic logits by random walk propagation: M T o ... T M.
Derive pseudo labels from refined semantic logits: Vee = arg max, M'
Train the softmax classifier S2 using V.c.
Predict final semantic segmentation 3 Predd from S2.
Algorithm 2: Inference procedure for semantic segmentation using image-level tags.
Input: Fixed pixel-wise embedding e of the input image and CAM logits M.
Output:
Semantic segmentation prediction P'red- /* Train the initial softmax classifier +/
Calculate pixel-wise transition probability matrix T from
Refine CAM by random walk propagation: M T T.M.
Derive pseudo labels from refined CAM: D'am arg max, Mi-
Predict new pseudo labels ) from Veam using nearest neighbor retrievals.
Train the softmax classifier S₁ using )/1 Vnn.
/x Train the final softmax classifier */
Predict pseudo labels y from initial softmax classifier Sj.
Predict new pseudo labels y22 from ) using nearest neighbor retrievals.
Train the softmax classifier S₃ using y
Predict final semantic segmentation Ypred from Sg.
A.5 ABLATION STUDY OF HYPPERPARRMETER
We conduct ablation study over different regularizations on Pascal VOC dataset. As shown in table
we achieve the most optimal performance on Pascal VOC dataset with \, = 0.1 and λo = 0.5. We also observe performance drops 0.4 of mloU by adding feature affinity regularization. We argue that scribble/box/pont annotations are not uniformly distributed across object instance and background, and results in noisy label propagation.
17

---

Published as a conference paper at ICLR 2021
o
mloU
A
o
mloU
A
o
A
mloU
0.3
0.5
73.7
0.1
1,0
74.1
0
0
0
71.2
0.1
0.5
74.2
0.1
0.5
74.2
0.1
0
0
72.8
0.05
0.5
73.5
0.1
0.1
74.1
0.1
0.5
0
74.2
0.5
71.7
0.1
0
72.8
0.1
0.5
0.1
73.8
Table 6: Ablation study of different weighting parameters for each objective function on Pascal VOC validation dataset.
Method
Backhone
CRF
Full
100%
80%
50%
30%
0%%
Lin et (2016)
CeepLab-MSe-Lagee-r
68.5
63.1
61.8
58.5
54.3
51.6
lang et al. (20T8b)
Decplab-MMSc--aggeFFe
68.7
66.0
65.5
64.2
62.7
57.2
Our SPML
DecpLab/ResNet101
76.1
74.2
74.2
73.3
73.4
71.3
Our SPML
DeepLab//RsNetl01
77.3
76.1
75.8
74.8
75.0
73.2
Table 7: mloU performance Pascal VOC 2012 validation set on different lengths of scribble.
4mm
hree
bailc
bax
63r
63
00g
papun
pam
dhapp
301a
19an
Tong atal
20100
13.
15.0
15.
10.9
R11
101
153
337
K77
313
157
377
31.9
01.0
31.
735
Our SPlIt
85.8
37.6
82.8
69.6
75.9
89.3
82.8
89.7
38.6
85.7
96.7
83.9
80.1
75.1
8488
539
63.7
49.1
80.0
74.4
74.2
2010
56.7
7778
01.
751
913
155
637
858
811
191
38.
814
767
00
59.0
38.4
56.0
726
77.
900
838
91.0
40.
58.3
57.5
87.7
828
79.1
86.5
57.1
87.
98.5
81.2
76.9
76.1
Table 8: Per-class results on Pascal VOC 2012 validation set. White- and gray-colored background denotes using without- and with- CRF post- post-processing for inference.
Ammedaatoos
bike
Hmdd
temile
minil
$1.7
81.6
S3.7
Scitlls
37.0
36.7
12.3
60.
79.7
18.5
645
1011
31.6
86.3
63.1
89.1
31.8
83.01
16.0
10.5
85.8
00.3
75.3
73.0
M6.4
Poins
83.5
07.0
75.4
61.9
7488
86.4
81.2
86.9
179
85.3
62.4
87.2
86.2
81.1
K1.1
64.3
85.1
99.1
74.0
66.1
74.0
Boses
84.1
36.5
86.7
53.6
75.7
83.7
84.8
80.6
19
86.4
57.
89.2
88.0
82.6
80.3
54.7
88.2
53.9
76.7
71.6
74.7
Taps
92.1
38.7
80.0
56.9
73.7
85.7
81.0
86.7
33.9
87.7
40.8
86.8
84.9
81.3
77.3
53.2
86.5
90.1
64.8
56.4
71.6
Table 9: Per-class results on Pascal VOC 2012 testing set. CRF postpprocessing is used for inference.
A.6 MEAN IOU PERFORMANCE WITH VARYING SPARSITY SCRIBBLES.
We report absolute mloU performance by varying sparsity of scribbles on Pascal VOC 2012 valida- tion The results are summarized in table Our results are much better with sparser annotation.
A.7 PER-CATEGORY MIOU PASCAL VOC AND DENSEPOSE DATASET.
We next present per-category results Pascal VOC and Denspose dataset. In table 8 compare with Tang et al. (2018)) on VOC validation set. Without- and with CRF post-processing, our method outperform the baseline method among experiments on VOC testing set, using DeepLab as backbone network. In table we can retrieve most performance w.r.t full supervision. We also compare per-category results DensePose dataset in table I0 We train our baseline method using the code released by Tang et al. (2018b). We outperform the baseline method by large margin every category.
categories by large margin. We further conduct
Method
bg
10000
Riland
Litand
Llok
RToot
KThigh
Linggh
RLag
LLag
LAm
RAcm
Liam
k1
MLoL
Tang et
20183
872
28.3
373
36.0
18.9
19.5
21.2
20.8
16.1
16.6
33.9
35.3
35.6
37.6
25.2
31.3
519
Our S29IL
95.8
32.
48.1
43.2
22.8
22.2
36.8
35.6
27.1
27.6
42.1
45.3
42.
45.5
726
44.2
77.1
Table 10: Per-class results DensePose minival 2014 set with keypoint annotations. White- and gray-colored background indicates using full and point supervision.
18


Reviews:
[{"review_id": "tTFY4-YUaPv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper talks about a novel weakly-supervised semantic segmentation (WSSS) approach which leverages a single pixel-to-segment contrastive learning formulation. The key idea is to map each pixel into a point in the feature space so that the pixels in the same semantic categories are embedded closely. It is interesting to note that they have also incorporated the analysis of unlabeled pixels across the images to harvest their patterns/clusters for better discrimination.\n\nPros:\n- Motivation was well described.\n- It is interesting to see that four different types of pixel-to-segment (where same segmentation entities were sometimes regarded as different categories) relationships were leveraged in a combined manner to eventually pull up the performance.\n- Experiments are reasonably carried out both qualitatively and quantitatively.\n\nCons/Questions:\n- Would there be a reasonable explanation of why a big performance gain was acquired for the \"labeled points\" case while the jump was relatively small on other weak supervision cases (image tags, bounding boxes, scribbles)?\n- A paper by Sun et al. (Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation, ECCV 2020) also claims that their approach is novel in that they make use of the intra-image information which is somewhat similar to the proposed approach. A paper by Fan et al. (Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation, CVPR 2020) seems highly relevant to the proposed approach, especially with the CAM-driven WSSS approaches. \nSince these recent papers have not been included in the reference of the submitted manuscript, it would be interesting to elaborate on how the proposed approach is unique/strong when compared.\n- For Table 2, what is the accuracy measure for the numbers shown?\n- It is hard to tell how the lambda values were chosen. (Table 1)\n- Some descriptions/explanations are rather redundant and verbose which makes it hard to read.\n\nTypo:\nIn Section 1, \"have have\"  --> \"have\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZTbEayq8pm-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a unified framework for weakly supervised semantic segmentation which is compatible with different types of annotations including image tag, bounding box, points and scribbles. With different kinds of labels, the method derives positive and negative segments for each labels and via metric learning the network learns to predict class label for each pixel. While the proposed method obtained STOA or close to SOTA performance on VOC2012 and densepose dataset, the reviewer feels that the novelty of the proposed method is not significant enough. A unified framework for semantic segmentation is appealing but it isn't new. E.g. Guided Attention Inference Network, TPAMI 2019 proposed a way to use gradient-based attention weakly supervised semantic segmentation framework which is compatible with image tags, bounding boxes and pixel-level annotations. In their work, different kinds of annotations are converted into supervision on class-specific attention maps and no parameters need to be tuned based on the ratio/types of annotations are used in the training, whereas the proposed method here needs different parameter choices for different lables/dataset. In the reviewer's eyes the submission follows a similar direction, although the framework design and objectives are different. The reviewer hope the authors can clarify their contribution and add discussions comparing to other similar approaches. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "T9QuyHfckA6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a metric learning-based semi-supervised semantic segmentation approach. In the proposed method, unlike the conventional semantic segmentation scheme that cast semantic segmentation as pixel-wise classification, they formulate semantic segmentation as pixel-segment contrastive learning. To that end, the authors introduced different positive and negative samples mining mechanisms such as low-level image similarity, semantic annotation, semantic co-occurrence, and feature similarity. Furthermore, the proposed method leverage unlabeled data in discriminative feature learning both within and across images. The validity of the proposed method is demonstrated on Pascal Voc and Dense Pose benchmark datasets.\n\n##########################################################################################\n\n*Strength: The formulation of pixel-to-segment based contrastive learning is intriguing. In addition to that, the authors introduced a new insight into collecting positive and negative samples that would be leveraged in contrastive learning. Moreover, the samples involve pixels that come from both intra and inter images.\n\n\n#################################################################################\n\n*Weakness:- The sub-modules used in the proposed approaches are adopted from existing methods thus there is no enough novelty in there.\n-The resulting improvement achieved is impressive however the work lacks theoretical novelty.\n\nQuestion: As some of the semantic relationships are basically generating pseudo-labels, for example, Feature Affinity. Have you conducted any experimental analysis to assess the impact of accumulated error that comes from these Pseudo labels?\n\n######################################################################################\n\n*Reason for score:\nThe proposed work is more of engineering, and it does not have any theoretical novelty; however, formulating semantic segmentation in a contrastive learning context is interesting. Furthermore, the authors demonstrate the effectiveness of the proposed method by showing improvements over the state of the art methods. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QcnkjfLgdEQ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nWeakly-supervised image/object segmentation can naturally be formulated as a pixel-level semi-supervised problem. To solve the insufficient supervision problem in weakly-supervised segmentation, the paper proposes to use four kinds of heuristics to provide supervision for training pixel-level metric learning. The takes the advantages of image segments generated from edge detectors (HED & gPB), then low-level image similarity, semantic annotation, semantic co-occurrence, and feature affinity.\n\nThe paper has obvious advantages summarized as follows.\n1. It unifies various weakly-supervised segmentation tasks by proposing pixel-level metric learning.\n2. The paper obtains clear SOTA results and significantly promotes the development of weakly-supervised segmentation.\n\nThe weakness of the paper is its clarity. Lots of details are missing in the paper, e.g., I cannot find how to apply semantic co-occurrence applied in training segmentation models using image classification models. I think this is not the authors' fault. The problem is caused due to there are so many contents in the paper. Thus, I think source codes should be released and well-organized to ensure the reproducibility of the paper.\n\nBesides, the terms in the papers can be re-considered. (1)  Although contrastive learning is very popular now, I think the method is better termed as metric learning - its original name. (2) I think there are some redundancy in semantic annotation, semantic co-occurrence, and feature affinity. The relation between them should be clarified and justified.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-CIOT0L8ObO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a unified framework for weakly-supervised semantic segmentation that can take various types of weak labels as the input, e.g., points, scribbles, boxes, image tags. The authors formulate it as a contrastive learning framework by considering pixel-to-segment relationships, i.e., for each pixel, finding positive and negative segments to perform contrastive learning objectives. Specifically, the paper introduces four types of relationships between pixels and segments, i.e., image similarity within the image, semantic relationship, semantic co-occurrence, and feature affinity across images. In experiments, results with various weak labels show SOTA performance on the PASCAL and DensePose datasets.\n\nPros:\n\nThe paper is well written and is easy to follow. The proposed unified framework using the idea of pixel-to-segment contrastive learning is interesting for the weakly-supervised semantic segmentation task. \n\nThe proposed four types of pixel-to-segment relationships seem mostly reasonable and effective for different types of weak label inputs.\n\nExtensive experiments are conducted to demonstrate the effectiveness of the proposed framework and show strong performance on the PASCAL and DensePose datasets.\n\nCons:\n\nAlthough using pixel-to-segment contrastive learning is interesting for weakly-supervised semantic segmentation, the technical contribution is a bit limited as it mostly derives from the recent work in Hwang et al., 2019.\n\nIt is not clear to fully understand the effectiveness of the introduced four types of relationships. For example, for semantic co-occurrence, it is doubtful to form positive pairs simply based on the co-occurrence, as the features in two images could be very different even they share at least one same label. This situation could be more significant if performing on more challenging datasets that contain diverse scenes and more categories, e.g., COCO, ADE20K. The authors should make comments on this or provide more analysis for reasoning this design choice, e.g., one easy thing to consider could be weighting this relationship based on the co-occurrence rate.\n\nIt appears that feature affinity is less effective or not used in some settings. Although the authors explain the reason in the appendix (due to noisy label propagation), it may not be a principal way to integrate this feature affinity into any dataset. Moreover, Table 1 shows different configurations for different label types, and it could also vary across different datasets. Although there are ablation studies provided in Table 5 and 6 of the appendix, it is difficult to draw conclusions on how to choose the proper weights. In this way, it makes the practical usage harder to scale up to more challenging datasets.\n\nAnother concern is the motivation and usefulness of the unified framework. In the literature, there exist specific frameworks optimized for different types of weak labels. Previous frameworks could not use all the weak labels at the same time, which is on the contrary the advantage of the proposed method. Therefore, it is of great interest to see whether the method can simultaneously leverage different weak labels and achieve better performance, e.g., using image tags + boxes against only boxes.\n\nOverall, the paper presents an interesting and effective framework. I would like to hear the feedback from the authors about the above-raised points and may update the rating (my current rating is more like a borderline).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article proposes a novel weakly supervised segmentation method that unifies several annotation types using contractive/metric learning. This method clearly outperforms the current SOTA. While the unified framework itself is not novel enough, the reviewers agree that the contrastive loss formulation is interesting and the extensive experiments show its effectiveness. Overall, I consider that this unified framework is well engineered, the formulations are insightful, and the results advance the SOTA of weakly supervised segmentation. Accordingly, I propose to accept this paper at ICLR 2021.
Decision: Accept (Poster)