Contextual Image Parsing via Panoptic Segment Sorting
Authors: Jyh-Jing Hwang
Tsung-Wei Ke
Stella Yu
Venue:  (2021)
Keywords: metric learning
context encoding
context discovery
image parsing
panoptic segmentation

Abstract: Visual context is versatile and hard to describe or label precisely.  We aim to leverage the densely labeled task, image parsing, a.k.a panoptic segmentation, to learn a model that encodes and discovers object-centric context.  Most existing approaches based on deep learning tackle image parsing via fusion of pixel-wise classification and instance masks from two sub-networks.  Such approaches isolate things from stuff and fuse the semantic and instance masks in the later stage.   To encode object-centric context inherently, we propose a metric learning framework, Panoptic Segment Sorting, that is directly trained with stuff and things jointly.  Our key insight is to make the panoptic embeddings separate every instance so that the model automatically learns to leverage visual context as many instances across different images appear similar.  We show that the context of our model's retrieved instances is more consistent relatively by 13.7%, further demonstrating its ability to discover novel context unsupervisedly.  Our overall framework also achieves competitive performance across standard panoptic segmentation metrics amongst the state-of-the-art methods on two large datasets, Cityscapes and PASCAL VOC.  These promising results suggest that pixel-wise embeddings can not only inject new understanding into panoptic segmentation but potentially serve for other tasks such as modeling instance relationships.

Under review as a conference paper at ICLR 2021
CONTEXTUAL IMAGE PARSING VIA PANOPTIC SEGMENT SORTING
Anonymous authors Paper under double-blind review
ABSTRACT
Visual context is versatile and hard to describe or label precisely. We aim to leverage the densely labeled task, image parsing, a.k.a panoptic segmentation, learn a model that encodes and discovers object-centric context. Most existing approaches based on deep learning tackle image parsing via fusion of pixel-wise classification and instance masks from two sub-networks. Such approaches isolate things from stuff and fuse the semantic and instance masks in the later stage. To encode boject-centric context inherently, we propose metric learning framework, Panoptic Segment Sorting, that directly trained with stuff and things jointly. Our key insight is to make the panoptic embeddings separate every instance so that the model automatically learns to leverage visual context as many instances across different images appear similar. We show that the context of our model's retrieved instances is more consistent relatively by 13.7%. further demonstrating its ability discover novel context unsupervisedly. Our overall framework also achieves competitive performance across standard panoptic segmentation metrics amongst the state-of-the-ar methods on large datasets, Cityscapes and PASCAL VOC. These promising results suggest that pixel-wise embeddings can not only inject new understanding into panoptic segmentation but potentially serve for other tasks such as modeling instance relationships.
INTRODUCTION
Visual context versatile and hard to describe or label precisely, yet is critical for humans (Medin & Schaffer 1978) to recognize objects quickly. More importantly, objects in different contexts carry different meanings. For example, pedestrians walking in crosswalks should receive more attention than sidewalks to driver. However, it is almost impossible to categorize objects with different contexts as the change can be subtile yet dramatic: A pedestrian is more likely in danger if walking in front of a than by car, where both a person and a appear together. We are thus motivated to propose a model that automatically encodes and discovers object visual context by leveraging a densely labeled task, panoptic segmentation.
Panoptic segmentation (Kirillov et al. 2019b), a.k.a., image parsing (Tu et al. 2005). Is to segment an image into its constituent visual patterns with both semantic and instance labels. The major challenge lies in delineating different instances while associating them with semantic categories. For example, one has segment two side-by-side cars apart while still being able to classify them as the same category. Most existing approaches (Kirillov et al. 201 9a Xiong et al. 2019 Yang et al., 2019 Cheng et al. 2020 Li al. 2020) tackle these two aspects via two sub-networks, instance and semantic segmentation branches. The advantage of such approaches is that each branch can cater to one aspect and achieves high performance. Additional modules for integrating things and stuff are needed to resolve the disagreements between two branches. Yet for object visual context, things and stuff are two integral parts. Hence, we aim to propose a framework that unifies these two seemingly competing aspects and thus encodes visual context inherently.
Our framework is inspired by the perceptual organization view (Biederman 1987): Humans perceive scene by breaking it down into visual groups and structures; repeated structures then associated for cognitive recognition. Our key insight is to separate everything first and group visually similar components later. The grouping takes place within an image and across images. Within an image,

---

Under review as a conference paper at ICLR 2021
image
panoptic embedding
panoptic prediction
panoptic label
query
retrieval within the image
retrieval images
image query
retrieved
Figure 1: Top from left to right: input image, panoptic embeddings, panoptic predictions, and panoptic lahels. We overlay panoptic embeddings with the resultant over-segmentation boundaries. Middle extracting panoptic embeddings from CNN the resultant iver-segmentation, we use the segment prototype features to find nearest neighhors, within the image (middle) or across images (right), of each query segment (in red). These retrieval results probe what's 's learned in the embedding space, Bottom row: an example of contex specific instance retrieval results, where pedestrians crossing intersection discovered unsupervisedly.
After
visually similar segments are merged to form instances; across images, visually familiar segments are associated to create semantics, as illustrated in Fig.
We carry this idea by building an end-to-end trained pixel-wise embedding framework. Each pixel an image is mapped via CNN to a feature in latent space, and nearby features indicate pixels belonging the same instance. This framework is therefore non non-parametric model at the segment and instance levels as its complexity scales with number of segments and instances, i.e., exemplars. Particularly, forcing all the instances separate, the model has to utilize all the possible visual and semantic information. The model thus learns to separate instances by not only their appearances but also their surroundings, or visual context. major difference between our model and others is the metric learning perspective: Our model trains with a contrastive loss that captures pixel- to-segment relationships while others trains with pixel-wise classification that predicts category or instance directly. As result, the learned panoptic embeddings can discover instances under similar context, as in Figure [ bottom row.
Specifically, we adapt the Segment Sorting approach (Hwang et al. 2019b) to panoptic segmentation by sorting segments according to both of semantic and instance labels, hence dubbed Panoptic Segment Sorting (PSS). Such trained pixel-wise embeddings thus encode both semantic and instance information. We then predict each segment's semantic label by simply mapping and classifying its prototype feature with a softmax classifier. We also propose corresponding clustering algorithm to merge segments into instances with a nearest neighbor criterion (Sarfraz et al., 2019). To al- leviate the problem of instances with various scales, we further equip our framework with hybrid scale exemplars during training and dynamic partitioning during inference. Finally, we facilitate the merging process with seeding branch that predicts the center of each instance.
As result, we demonstrate that the contexts of instances retrieved by our panoptic embeddings are more consistent relatively by 13.7% while achieving competitive performance amongst the state- of-the-art on two datasets, Cityscapes (Cordts et al., 2016) and PASCAL VOC (Everingham et al.) These promising results suggest that Panoptic Segment Sorting pixel-wise embeddings can not only inject new understanding into panoptic segmentation but potentially serve as a foundation for other tasks such as discovering novel contexts or modeling instance relationships.
2 RELATED WORK
Image parsing and panoptic segmentation. The task image parsing is first introduced TT al. (2055), where they formulate the solution in Bayesian framework and construct parsing
2

---

Under review as a conference paper at ICLR 2021
graph output. Since then, lot of work has attempted solve holistic scene understanding Zhu & Mumford (2007); Malisiewicz & Efros (2008); Tighe & Lazebnik Rabinovich et al.
2007 Yao et al." 2012)). ). Recently, Kirillov et al. (2019b) reintroduce image parsing in the context
of deep learning with large-scale datasets and new evaluation metric, renaming the task as panoptic segmentation as unify the well-developed semantic and instance segmentation. Many research
efforts (Li et al., 22018b Kirillov et al. 2019a Xiong et al. 2019: Porzi et al. 2019] Yang et al. 2019)
Liu et al. 2019 Li et al. [2119] 2018a Gao al., 2019 Chen et al., 2020 Wu al. 2020; Wang
et al. 2020; Li et al., 2020) have followed quickly. The common approaches embrace the concept
of unifying instance and semantic segmentation by integrating the time-tested object proposal and
segmentation framework popularized by Mask R-CNN (He et al. 2017).
Instance segmentation. This task generally approached by two camps of solutions: top-down or bottom-up. The top-down approaches (Dai et al. 20166: Li et al.) 2017 Dai et al. 2017; He et al.) 2017 Chen et al. 2018a Liu et al. 20T88) adopt a two-stage framework where the bounding boxes
proposed by detection network (Ren et al. 20155 and the segmentation masks are produced by
an add-on head. The bottom-up approaches (Carreira & Sminchisescu [011 Arbellaz et al. 2014
Pinheiro al. 2015; 2016 Bai & Urtasun 2017:
Liu et al. 2017a Kirillov al.) 2017 Newell
et al.
2017 7 Fathi et al., 2017] Kendall et al.
2018 Liu et al. 2018b Papandreou et al., 20188] Zhou
al. 2019) predict and encode pair-wise relationships in various forms and segment the instance
accordingly.
Instance context. Instance contexts and relationships explored mainly enhance the detection performance. Earlier work (Malisiewicz & Efros 2009) models the appearances and 2D spatial con- text as graph. Recently researchers integrate graphs (Chen et al. 22018 8c) or spatial memory (Chen) & Gupta, 2017) into the deep learning framework. The distinction of our work is that our model does not explicitly model contexts yet is able discovers novel contexts automatically.
Semantic segmentation. Current state-of of-the--rrtttttoo arr semantic segmentation approaches develop from fully convolutional networks (Long et al., [2015; Chen al., 2016), with various innovations. In- corporating contextual information (Ronneherger et al.) [0015] Yu & Koltun 2016 Xie et al. [0166] Zhao et af.] 2017; [hen et al.] [017]] 2018b). and encoding pair- wise relationships (Zheng et al., 2015 Bertasius et 22.1/2016] Liu al. 2017b, Bertasius et al., 2017 Maire et al.) 2016] MMstajabi ct al. 2018; Kong & Fowlkes 2018 Ke et al. 2018 Hwang et al. 2019a b) are the two major
research lines.
Non-parametric segmentation. Prior to deep learning's emergence, non- parametrii models (Rus-) sell et al.) 22009; Tighe & Lazebnik 2010 Liu et al. (2011) usually use hand-craft features with sta- tistical models or graphical models to segment images with pixel-wise labels. Deep metric learning methods (Fathi et al. 2017 Neven et al. 2019) for instance segmentation emphasize the simplicity and fast computation. More recently, inspired by non-parametric models (Wu et al.) 2018h a) for image recognition, SegSort (Hwang et al. 2019b), upon which our work is built, captures pixel-to- segment relationships via pixel-wise embeddings, proposing the first deep non-parametric semantic segmentation both supervised and unsupervised settings.
3
METHOD
Our end-to-end framework consists of major SegSort branch and a seeding branch, both of which share one backbone network that generates multi-scale pixel-wise features. The SegSort branch out- puts pixel-wiss wise panoptic embeddings, which encode both semantic and instance information and are thus used to discover instance-centric context. The over-segmentations induced by the embeddings are then merged into instances and segments are classified by a softmax classifier. The seeding branch predicts the center of instances, which guide the merging process to reduce false positives. The overall framework is illustrated in Figure
This section is organized as follows. We first briefly review the Segment Sorting framework for semantic segmentation in Sec. 3.1 We then describe how to extend for panoptic segmentation in Sec. 3.2. In Sec. 3.3 we further develop dynamic partitioning mechanism to alleviate the problem of varying scales of instances. Finally, we briefly describe the sceding branch in Sec. 3.4 that helps decide the ownership of boundaries.
3

---

Under review as a conference paper at ICLR 2021
image
panoptic embeedding
panoptic segment sorting
semantic prediction
clossity segment
CNN
seed
Instance prediction
merge
seg megtent
Figure 2: The overall end-to-end diagram of our proposed Panoptic Segment Sorting (PSS). We first over- segment image with pixel-wise embeddings, extracted from a CNN. Each segment represented by prototype feature (average of pixel embeddings), which is then used for classifying segments (semantic predic- tions) and/or merging segments into an instance (instance predictions), whose features from the embeddings automatically encode object-centric context. An extra center seeding branch can faciliate the merging pro- cess by designating seed segments. The overall losses include (1) the SegSort loss (Hwang al. (019b) for embeddings, (2) the cross-entropy softmax loss classification, (3) the regression loss for seed locations.
3.1 SEGMENT SORTING
We briefly review the Segment Sorting (SegSort) approach proposed by Hwang al. (2019b). Seg- Sort is an end-to-end optimization framework for non- non-parametric semantic segmentation. It pro- duces pixel-wiss wise semantic embeddings and their corresponding over-segmennation, each segment of which is then, during inference, assigned a semantic category via K-Nearest Neighbor search.
The basic idea of SegSort is assuming independent normal distributions (or Mises-Fishe dis- tributions for normalized embeddings) for individual segments, and seeking a maximum likelihood estimation of the feature mapping, so that the feature induced partitioning in the image and cluster- ing across images provide maximum discrimination among segments. SegSort can be summarized components: spherical k-means clustering (Banerjee et al.] 2005) and maximum likelihood loss formulation with soft nighborhood assignments (Goldberger et al. 2005).
The spherical k-means clustering (Banerjee et al., 2005) alternates the expectation (E) and maxi- mization (M) steps to partition the unit-length pixel- pixel-wise embeddings of an image into K regions (RI...., Rx). The M-step calculates the mean embedding direction of each region, or the proto- LeeR rype μa = 1.Enn, will" The E-step assigns each pixel embedding v, region R3 with nearest
corresponding prototype μk. 2; arg max H. U; where 2; is the segment index that the pixel i assigned. Note that the dot product on the right hand side is equivalent cosine similarity both v and are of unit length. By alternating E- and M-steps, we over-segment image.
After wver-eegmentation. one can derive maximum likelihood loss with soft neighborhood assign- ments (Goldberger et al. 2005) to train the deep neural networks end-to-end. Interested readers are referred the SegSort paper (Hwang et al.) (019b) for detailed derivation. The principle con- nect each pixel with one of its same-class segments, excluding its own segment, and push away all the other segments different classes. We define corresponding probabilities given semantic segmentation ground truth labels follows.
at exp(RRU v.)
p(=: e+ U.. +) Eieo exp(1141, v.)²
p(2, c:v: +) 0,
where is the concentration (around μ) hyper-paramett in the Mises-Fisher distributions, denotes the segment index to which the pixel is assigned, and ct denotes the segment index of any other same-class segment across all images in batch. The final SegSort loss is therefore the
4

---

Under review as a conference paper at ICLR 2021
negative ong-likelhhod of a pixel selecting same-class prototype its neighbor:
Lisessont - log Σ pa(2; = v., Q) = log Exect eXp(Mи v;)
Σi exp(KIL v4)
SECT
where C++ denotes the set of c+ segment indices w.r.t. the pixel i, which is selected by the semantic segmentation ground truth labels. Minimizing this loss equivalent to maximizing the expected number of pixels correctly classified by voting of their nearest neighbor prototypes.
3.2 PANOPTIC SEGMENT SORTING
Since the SegSort loss does not require a fixed number of classes as opposed the conventional cross-entropy softmax loss, way to extend it for instance discrimination is by changing the defini- tion of ground truth labels and its corresponding selections of neighbor prototypes. In other words, we instead consider c+ as the segment index of any other "same-instance" segment. For stuff cat- egories without instances, we consider all the segments in that class have the same instance label. With this modification, the SegSort loss in Eqn. can be used to train panoptic embeddings.
Such trained embeddings, therefore, group each instance against all the other instances, regardless of their semantic categories. Still, since this loss pushes all the instances far away possible, vi- sually similar instances are forced to stay closer on the hypersphere. We thus hypothesize two kinds of additional information are encoded: (1) The embeddings encode the semantic labels inherently as instances of the same class appear similar. To extract such information, we then stack two 1 x convolutional layers on top of segment prototypes, followed by softmax classifier to predict the semantic class of each segment. Note that no conffict between semantic and instance segmentations introduced in this setting as they are built on the same wee-seementation.
(2) The embeddings also encode object-centric context, This is because many instances (especially common objects like cars, persons, bikes, etc. across different images appear similar so the model has to leverage all possible information to push away all the instances of the same class.
Given the panoptic embeddings and the resultant over-segmentations, the challenge to group seg- ments into instances correctly during inference. We need two criteria: 1) how to merge segments, and 2) when to stop the merging. To align with the formulation of the SegSort loss, we adopt nearest neighbor clustering criterion (Sarfraz et al.] 2019) to greedily merge two segments Rm, R, with nearest prototypes, and stop the merging if the distance between two prototypes pms pn is greater than threshold, their dot product is less than threshold Tpp The merging criteria can be summarized as:
R (Rm, R.) (N(m)) or M(w,) = m) and Pmt Tp,
(3)
where } denotes merging segments, NN(-) denotes the index of the nearest neighbor prototype. We sort all the pairs of distances (dot products) of the prototypes in image and consider merging greedily from the closest pair. We also update the new prototype after merging.
3.3 DYNAMIC PARTITIONING FOR HYBRID SCALE EXEMPLARS
The vanilla SegSort partitions an image into a fixed number of regions regardlessly. For semantic segmentation, this setting is reasonable as the number and sizes of homogeneous regions not vary a lot from image to another. However in instance segmentation, scales of objects can change drastically from 100 to 100K pixels. Oftentimes cluttered small instances will fall into one single segment, even worse be included in another big instance. To alleviate this scale problem, propose a hybrid scale setting for training and dynamic partitioning for inference accordingly. The illustrations can be found in Appendix.
During training, we consider regular embeddings and their upscaled embeddings ((1)) by bilinear interpolation. The idea is to use the upscaled embeddings for small instances so that the gradient flows finer. After the spherical k-mean clustering, we calculate segment prototypes using em- beddings in different scales according to the instance sizes. Note that there is still only one prototype for each segment in the SegSort loss, be it either regular upscaled.
During inference, the sizes of instances are unknown and have to be inferred. We notice that if a segment contains multiple small instances multiple parts from big instance, the corresponding
5

---

Under review as a conference paper at ICLR 2021
pixel embeddings usually noisy, resulting in low concentration. Therefore, define ap-
proximated concentration Rg of segment R2 R2 | v.ll [o. 11, where R&g denotes the number of pixels the segment. If this value for a segment falls below a certain threshold Ts. we again partition this segment using the same spherical k-means (here k = 4 usually).
[R
3.4 SEEDING BRANCH
We notice the boundaries between objects sometimes form their own segments, causing false posi- tive instances. To remedy this issue, we build a second branch for predicting instance seeds, which used for guiding the merging process, described in S.2. We define seeds as the centers of in- stances and mark the segments that cover seeds as seed segments. For building this seeding branch, follow closely the instance proposal branch in He al. (2077): Xiong et al. (2019) and the centers of the predicted bounding boxes as the seeds.
Once predict the seeds and the corresponding seed segments, perform seeding variant of merging. The only modification of the merging of {Rm. 3 (in Eqn. 3 is that the segments to merge Rm, &R, are restricted one seed segment and one non-seed segment; the merged segments are then marked as seed segments. Note that the merging only happens between same class seg- ments. After this modification, all the boundary segments are then forced be merged into one of the seed segments. The visualization of the merging processes can be found in the supplementary.
EXPERIMENTS
In this section, we demonstrate the efficacy of our framework through extensive experiments and analysis. We first describe the experimental setup in Sec. 4.] We present the context specific in- stance retrieval results in Sec. 4.2 Finally in Sec. 4.3) we present the panoptic segmentation results. Hyper-parane spper-parameters, ablation study, and more visual results such panoptic predictions, context trieval, and t-SNE (Maaten & Hinton 2008) feature analysis, can be found the Appendix.
4.1 EXPERIMENTAL. SETUP
Datasets. We carry out experiments mainly on two datasets: Cityscapes and PASCAL VOC 2012.
Cityscapes (Cordts et al. 2016) is dataset for semantic urban street scene understanding. 5, 000 high quality pixel-level finely annotated images are divided into training, validation, and testing sets with 2, 975 / 500 1, 525 images, respectively. defines 19 semantic categories containing flat, human, vehicle, construction, object, nature, etc. of which 8 categories have instance labels.
PASCAL VOC 2012 (Everingham et al.) segmentation dataset contains 20 object categories and one background class. The augmented dataset contains 10, 582 (train) / 1, 449 (val) 1, 456 (test) images. All the semantic classes, except for backgrounds, have instance labels.
Network architecture. We use the Feature Pyramid Networks (FPN) (Lin et al. 2017), with ResNet-50 (He et al. 2016) backbone pretrained on ImageNet, to provide the multi-scale pixel- wise features. For each of the seeding and panoptic embedding branch, we follow Xiong al. (2019) by building three layers of deformable convolutional layers (Dai et al.) 2017) (with shared weights across different scales) on top of each scale of FPN features. We then concatenate the multi-scale features, followed by final fusion x convolutional layer. On top of the panoptic embeddings, we stack two 1 convolutional layers for the segment softmax classifier.
4.2 CONTEXT SPECIFIC INSTANCE RETRIEVAL
In this section, we experimentally verify our panoptic embeddings encode the object-centric context automatically.
Discovery of novel context. We retrieve the nearest neighbors of query instances on the Cityscapes validation set using their averaged embeddings. We notice that the retrieved instances are usually in similar context as the query. We showcase five interesting examples in the Appendix, i.e., pedestrians crossing intersecion (also in Figure T) walking next cars, riders riding bikes together next

---

Under review as a conference paper at ICLR 2021
0.
0.15
6.5
0.1
whole image
semantic distribution
Figure Illustration evaluating context similarity between two instaces by comparing their semantic distributions in extended regions. From left to right: whole image, instance of interest, semantic distribution of the middle right extended region. We calculate the symmetric KL-divergence between semantic distributions from corresonning regions the context error.
to cars, and cluttered parked motorbikes. Note that these contexts are not given in the ground truth labels, yet our PSS can discover them unsupervisedly. We believe these examples are relevant in street scene understanding, especially for self-driving vehicles.
Quantitative evaluation. We wonder such phenomena can be measured quantitatively. The chal- lenge lies in the complicated scenarios and the lack of a complete label set. For example, crosswalks, which are labeled as roads, are visually similar as yet functionally different from roads. Furthermore, riding motorbikes next to cars is dangerous but difficult to describe precisely for annotating tasks.
We notice that the semantic category distribution of larger patch captures some of such cases. For example, if there are multiple pedestrians nearby with cars around them, the chance of them walking on a crosswalk is higher. Such motivated, we propose to evaluate the context similarity between query and retrieval instances by comparing their semantic categories in 8 extended regions, as illustrated in Fig.
To be specific, we denote the 8 neighbor regions (with the same size the instance) as B, for j = 1....,& 8. We calculate the semantic distribution each region by the occupancy ratio of each class and denote it PB,- That is, for each class, given semantic label mask Sr, then Pj,
[3yT [B for each category c, where |Byl denotes the area of region B,. We then compare the semantic context distribution of the query p(g) against its i-th retrieval p(r)) by calculating the symmetric KL. divergence between the two,
8 K
CE
ΣΣ i=1
(DKL By p(r.)) + DKL(P(r)) .P()))
8K
where CE is our proposed metric, Context Error, and K is the number of retrievals per query instace. If the reference probability 0. the KL divergence will be invalid; this case, we use small probability 0.1 instead. We set K to 20 nearest neighbors.
We compute Context Error (CE) for each instance category, i.e., we restrict both query and retrieval to be certain instance category. The final CE is the average errors of all instance categories. We compare our PSS against state-of-the-art UPSNet (Xiong et al. 2019) and summarize the results in Tab. We observe PSS performs better in every category and reduces 13.7% relative context errors.
method
person
rider
car
truck
train
mbike
mean CE
UPSNet Xiong el al. 2019
1.15
1.21
0.88
1.20
1.33
1.23
1.21
1.16
PSS
0.96
1.01
0.65
1.12
1.04
1.27
1.11
1.05
1.02 (-13.7% %)
Table 1: Context Errors (CE) the Cityscapes (Cordts 2016)
validation set. We observe PSS performs
better in every category and reduces 13.7% relative context erros.

---

Under review as a conference paper at ICLR 2021
PSS (ours) retrieval >
UPSNet retrieval
Figure 4: Visual comparison for context specific instance retrieval. We show 3 query examples (left) and their top retrieval results by PSS (middle) and UPSNet (Xtong al. 2019) (right), respectively. We observe that retrieved instances by PSS are usually similar context or sometimes even from same training image.
methed
backbone
PQ
PQ
PQ
Li al.
2018b
ResNet-101
47.3
39.6
52.9
Deeper ab Yang ct al
2019
Xception-71
56.5
AUNet (Li et
2019
ResNet-50
56.4
52.7
59.0
SSAP Gao et al.
2019
ResNet-50
56.6
49.2
Panoptic FPN (Kiiillay et al.
2019a
ResNer-50
57.7
51.6
62.2
UPSNet Xiong et al. 2019
ResNer-50
59.3
54.6
62.7
UPSNct (Xtong ct al. 2019
ResNet-50
59.1
54.2
PSS
ResNet-50
58.7
51.7
Table 2: Experimental results on the Cityscapes validation set. Our proposed framework PSS achieves com- petitive performance in PQ and outperforms all the other methods in PQ*. # denotes retraining the model using released code; other results copied from the published papers and denotes missing metrics.
Visual Comparison. Next, we present the visual comparison in between our PSS and UPSNet using three query instances from the same validation image and display 3 retrieved instances for each network in the training set. We observe that our retrieved instances are usually in similar context and are sometimes even from the same training image. indicates that PSS encodes not only the appearances of an instance but also its nearby environment.
Visual Context Cluster Analysis We conduct visual context cluster analysis and visualize the re- sults in Fig. We first collect all the pedestrian prototypes in the Cityscapes training set. We plot their surrounding ground truth mask their t-SNE feature locations and the aggregated density map. We observe interesting clusters such as pedestrians next to car (center) and pedestrians alone on sidewalks (top left). We also notice some rare contexts on the middle left by examining the density map: pedestrian is behind clutter of a motorbike and a bike, which could lead to collision.
4.3 PANOPTIC SEGMENTATION
summarize the main results on the Cityscapes validation set and compare with the state-of-the-as in Table Our PSS achieves competitive performance in PQ (Panoptic Quality, explained in Appendix) and outperforms all the other methods in PQS:, Notably, framework performs particularly well in semantic segmentation related benchmarks.
Main results on Cityscapes.
We summarize the main results on the PASCAL VOC validation set and compare with the of- the-art in Table 3 We show that PSS outperforms Li et al. (2018b) by 2% PQ even with weaker backbone (ResNet-50 vs 101).
Main results on PASCAL VOC.
method
buckbone
PQ
Li al
20180
ResNet- 101
62.7
ResNes-50
Table 3: Experimental results on
Pascal VOC 2012 validation set.
8

---

Under review as a conference paper at ICLR 2021
Density Map
Road
Traffic sign
Truck
Sidewalk
Vegetation
Bus
Building
Terrain
Train
Wall
Sky
Motorcycle
Fence
Person
bicycle
Pole
Rider
Traffic light
Car
Figure 5: Pedestrian-centric visual context cluster visualization (best viewed with zoom-in). We first collect all the pedestrian prototypes in the Cityscapes training set. We plot their surrounding ground truth mask their t-SNE feature locations and the aggregated density map (bottom left). We observe interesting clusters such pedestrians next car (center)
pedestrians alone on sidewalks (top left). contexts on the middle left by examining the density map: a pedestrian is behind a clutter of a motorbike and a bike, which could possihly lead to collision.
notice some rare
SUMMARY
We presented the Panoptic Segment Sorting (PSS) framework for contextual image parsing. We aimed to encode and discover object-centric context automatically by unifying semantic and instance segmentation in the pixel-wise panoptic embeddings. We experimentally demonstrated such trained embeddings automatically encode object-centir context for better instance discrimination. PSS also achieved competitive performance amongst the state-of-the-art when equipped with hybrid scale exemplars, dynamic partitioning, and instance seeding. One future direction is to study how to explicitly disentangle the representations for instance appearances and contexts.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pablo Arbeláez. Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multi- scale combinatorial grouping. CVPR, 2014.
Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. CVPR, 2017.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hyper- sphere using von mises-fisher distributions. Journal of Machine Learning Research, 2005.
Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural fields. In CVPR, 2016.
Gedas Bertasius, Lorenzo Torresani, Stella X Yu, and Jianbo Shi. Convolutional random walk networks for semantic image segmentation. In CVPR, 2017.
Irving Biederman. pereentttoonbyycooooooee a theory of human image understanding. Psycho- logical review, 94(2):115, 1987.
Joao Carreira and Cristian Sminchisescu. Cpme: Automatic object segmentation using constrained parametric min-cuts. TPAMI, 2011.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv: 1606.096060091 2016.
Liang- Blang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.0553 75550777777 2017.
Liang-Chieh Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. In CVPR, 2018a.
Liang- Chigh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. ECCV, 2018b.
Xinlei Chen and Abhinav Gupta. Spatial memory for context reasoning in object detection. In ICCV, 2017.
Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinav Gupta. Iterative visual reasoning beyond convolu- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 7239-7248, 2018c.
Yifeng Chen, Guangchen Lin, Songyuan Li, Omar Bourahla, Yiming Wu, Fangfang Wang, Junyi Feng, Mingliang Xu, and Xi Li. Banet: Bidirectional aggregation network with occlusion han- dling for panoptic segmentation. In CVPR, 2020.
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang. Hartwig Adam, and Liang-Chich Chen. Panoptie-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
M. Everingham. L. Van Gool, C. K. I. Williams. J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- Setwokkor/aalllegee////eeeeeee/ 2oooo
10

---

Under review as a conference paper at ICLR 2021
Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang. Hyun Oh Song. Sergio Guadarrama, and Kevin P Murphy. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277, 2017.
Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In ICCV, 2019.
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Ruslan R Salakhutdinov. Neighbourhood components analysis. In NIPS, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Jyh-Jing Hwang, Tsung-War Bein-WWi Ke, Jianbo Shi, and Stella X Yu. Adversarial structure matching for structured prediction tasks. In CVPR, 2019a.
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang- Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. ICCV, 2019b.
Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X Yu. Adaptive affinity fields for semantic segmentation. In ECCV, 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. Instancecut: from edges to instances with multicut. In CVPR, 2017.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In CVPR, 2019a.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019b.
Shu Kong and Charless Fowlkes. Recurrent pixel embedding for instance grouping. In CVPR, 2018.
Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, and Adrien Gaidon. Learning to fuse things and stuff. arXiv preprint arXiv: 1812.00 192, 2018a
Qizhu Li, Anurag Arnab, and Philip HS Torr. Weakly- and semi-supervised panoptic segmentation. In ECCV, 2018b.
Qizhu Li, Xiaojuan Qi, and Philip HS Torr. Unifying training and inference for panoptic segmenta- tion. In CVPR, 2020.
Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention-guided unified network for panoptic segmentation. In CVPR, 2019.
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-awar aware semantic segmentation. In CVPR, 2017.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparametric scene parsing via label transfer. PAMI, 2011.
Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, and Wei Jiang. An end- to-end network for panoptic segmentation. In CVPR, 2019.
Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance segmentation. In ICCV, 2017a.
11

---

Under review as a conference paper at ICLR 2021
Shu Liu, Lu Qi. Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, 2018a.
Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learn- ing affinity via spatial propagation networks. In NIPS, 2017b.
Yiding Liu, Siyu Yang. Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu. Affinity derivation and graph merge for instance segmentation. In ECCV, 2018b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008
Michael Maire, Takuya Narihira, and Stella X Yu. Affinity cnn: Learning pixel-centric pairwise relations for figure/ground embedding. In CVPR, 2016.
Tomasz Malisiewicz and Alexei A Efros. Recognition by association via learning per-exemplar distances. In CVPR, 2008.
Tomasz Malisiewicz and Alyosha Efros. Beyond categories: The visual memex model for reasoning about object relationships. In NIPS, 2009.
Douglas L. Medin and Marguerite M Schaffer. Context theory of classification learning. Psycholog- ical review, 85(3):207, 1978.
Mohammadreza Mostajabi, Michael Maire, and Gregory Shakhnarovich. Regularizing deep net- works by modeling and predicting label structure. In CVPR, 2018.
Davy Neven, Bert De Brabandere, Mare Proesmans, and Lue Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. In CVPR, 2019.
Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In NeurIPS, 2017.
George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with bottom-up, part- based, geometric embedding model. In ECCV, 2018
Pedro O Pinheiro, Ronan Collobert, and Piotr Dollár. Learning to segment object candidates. In NeurlPS, 2015.
Pedro O Pinheiro, Tsung- Yi Lin, Ronan Collobert, and Piotr Dollár. Learning to refine object segments. In ECCV, 2016.
Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene
segmentation. In CVPR, 2019.
Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge J Belongic. Objects context. In ICCV, 2007.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Bryan Russell, Alyosha Efros, Josef Sivic, Bill Freeman, and Andrew Zisserman. scenes by matching image composites. In NIPS, 2009.
Segmenting
M Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In CVPR, 2019.
12

---

Under review as a conference paper at ICLR 2021
Joseph Tighe and Svetlana Lazebnik. Superparsing: scalable nonparametric image parsing with superpixels. ECCV, 2010.
Joseph Tighe and Svetlana Lazebnik. Finding things: Image parsing with regions and per-exemplar detectors. In CVPR, 2013.
Zhuowen Tu, Xiangrong Chen, Alan L Yuille, and Song-Chun Long-Chun Zhu. Image parsing: Unifying seg- mentation, detection, and recognition. IJC,, 2005.
Haochen Wang, Ruotian Luo, Michael Maire, and Greg Shakhnarovich. Pixel consensus voting for panoptic segmentation. In CVPR, 2020.
Yangxin Wu, Gengwei Zhang, Yiming Gao, Xiajun Deng, Ke Gong, Xiaodan Liang, and Liang Lin. Bidirectional graph reasoning network for panoptic segmentation. In CVPR, 2020.
Zhirong Wu, Alexei A Efros, and Stella Yu. Improving generalization via scalable neighborhood component analysis. In ECCV, 2018a.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. In CVPR, 2018b.
Saining Xie, Xun Huang, and Zhuowen Tu. Top-down learning for structured labeling with convo- lutional pseudoprior. In ECCV, 2016.
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
Tien-Ju Yang, Maxwell D Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chies Chen. Deeperlab: Single-shot image parser. arXiv preprint S.C.v::9922.009999 2019.
Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the scene whole: joint object detection. In CVPR, 2012.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Da- long Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural net- works. In ICCV, 2015.
Xingyi Zhou, Jiacheng Zhuo, and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019.
Song-Chun Zhu and David Mumford. A stochastic grammar of images. Foundations and Trends in Computer Graphics and Vision, 2007.
13

---

and Under review as a conference paper at ICLR 2021
A APPENDIX
We provide metric learning approach to contextual image parsing. is not only unified approach to semantic segmentation and instance segmentation, but most excitingly also the first segmentation method that produces learned feature representation directly applicable for more powerful context- specific image retrieval. Here we include more details on the following aspects:
We present the visual results of novel context discovery Section A.
We present more interesting visual context comparisons in Section A.2
We present the visual results of panoptic segmentation predictions Cityscapes vali- dation set Section A.5
We visualize hybrid scale exemplars and merging processes with and without seeds, for further insights into our algorithm in Section A.3 and A.4 respectively.
We analyze the instance embeddings using t-SNE visualization in Section A.6
We provide detailed description of our experimental setup regarding ablation study in Section A.7
A.1 VISUAL RESULTS OF NOVEL CONTEXT DISCOVERY
As discussed in Sec. 4.2 we retrieve the nearest neighbors of query instances on the Cityscapes validation set using their averaged embeddings. Here, we showcase five interesting examples in Fig. i... pedestrians crossing an intersecion or walking next to cars, riders riding bikes together or next cars, and cluttered parked motorbikes. Note that these contexts are not given the ground truth labels, our PSS can discover them unsupervisedly. We believe these examples are relevant street scene understanding, especially for self-driving vehicles.
A.2 ADDITIONAL VISUAL COMPARISONS FOR CONTEXT SPECIFIC INSTANCE RETRIEVAL
We present interesting visual comparisons for context specific instance retrieval in Fig. It can be observed that PSS captures more context when retrieving instances. For examples, riding bikes along a sidewalk to its left, parked cars with bike passing by, driving cars with pedestrians nearby, riding motorbikes behind a car, etc.
A.3 ILLUSTRATION OF HYBRID SCALE EXEMPLARS
In Section 3.3 in the main paper, we describe the training process with hybrid scale exemplars. In short, when we train the panoptic embeddings with the SegSort loss, we select segment prototypes from embeddings of different scales, decided by their corresponding instance sizes. Particularly, we use upscaled embeddings for prototypes of small instances. We illustrate this process in Fig.
A.4 VISUALIZATION OF MERGING PROCESS
We describe the merging process (without and with seeds) in Section 3.2 and 3.4, respectively. To facilitate the understanding,
visualize few intermediate steps in Fig. 9 We observe that once an instance composes of a single segment, its cosine similarity to its nearest prototype drops significantly. It indicates this merging process with threshold works well with our training objective.
Also note that the merging usually grows from interior pixels towards boundaries.
A.5 PANOPTIC PREDICTIONS
We present the visual results of our panoptic segmentation predictions on the Cityscapes validation set in Fig. I0 We observe that semantic predictions capture well thin objects such as poles and traffic signs, even many of them are labeled ignore. Also, the far away cars (in example 3) and pedestrians (in example 1) are also detected.
14

---

Under review as a conference paper at ICLR 2021
image & query (in red)
retrieved >
(1)
(3)
(4)
(5)
Figure Retrieved the K-Nearest Neighbors for query instances. The image and query are on the left, and to their right are the retrieved instances (in the order left to right, top to bottom). We demonstrate one kind of contexts in each example: 1) Pedestrians crossing an intersection. 2) Podestrians walking next to cars. 3) Riders riding bikes together. 4) Riders riding motorbikes next to cars. S) Parked motorbikes. Note that these contexts given in the ground truth labels, yet PSS discover them automatically. We believe these examples are relevant street scene understanding, especially self-driving vehicles.
15

---

Under review as a conference paper at ICLR 2021
query
PSS (ours) retrieval >
UPSNet -
Figure 7: More visual comparisons for context specific instance retrieval between PSS (ours) and UPSNet. can be observed that PSS captures more context when retrieving instances. For examples, riding bikes along sidewalk its left, parked cars with bike passing by, driving with pedestrians nearby, riding motorbikes behind car, etc.
image
normal embedding
normal instance
CNN
upscale
SegSort loss
small instance
upscaled embedding
Figure 8: Illustration of SegSort training with hybrid scale exemplars in Section 3.3 in the main paper. The embeddings are overlayed with the predicted over-segmentation (with gray boundaries). We emphasize here one normal sized instance (outlined red) and one small instance (outlined in purple), the prototypes of segments of each are collected from corresponding embeddings.
16

---

Under review as a conference paper at ICLR 2021
image
merging (1)
merging (2)
merging (3)
merging (4)
concentration
seed (erging
seed merging (2)
seed merging (3)
seed merging (4)
Figure 9: Visualization of the merging processes, without and with seeds, described in Section 3.2 and 3.4 in the main paper, respectively. On the left are the input images and approximated concentration per segment (overlayed with oversegmentation boundaries). To their right merging processes without seeds (first row) and with seeds (second row). maps indicate the cosine similarity between each segment and its nearest prototype. (The white segments are seed segments.) The merging starts from the pair with the highest cosine similarity. We visualize few steps in the beginning. Note that once instance composes of single segment, its cosine similarity to its nearest prototype drops significantly. indicates this merging process with threshold works well with training objective.
contours outline two segments merge this step. The viridis heat
Image
somantic label
somantic pradiction
Instance label
instance prediction
Figure 10: The panoptic segmentation visual results on the Cityscapes validation set our proposed frame- work, Panoptic Segment Sorting. We observe that our semantic predictions capture well thin objects such as poles and traffic signs, even if many of them are labeled ignore. Also, the far away cars (in example 3) and pedestrians (in example !) also detected.
17

---

Under review as a conference paper at ICLR 2021
We visualize the prototype features of cars the Cityscapes training using ISNE. We observe that the features cluster certain parts types of cars automatically. For example, front wheels appear the bottom left corner and red cars in center. Best viewed with zoom-in.
Figure 11:
A.6 T-SNE VISUALIZATION
We visualize the prototype features of cars the Cityscapes training set using t-SNE in Fig. We observe that the features cluster certain parts or types of cars automatically. For example, front wheels appear the bottom left corner and red cars in the center.
18

---

Under review as a conference paper at ICLR 2021
hybrid dynamic
PQ
PQ
PQ
PQ"
mloU
scoding
classifier
PQ
PQM
PQ
mloU
51.5
32.4
67.4
75.60
44.0
9.3
25.4
35.6
61.66
52.4
7.3
31.5
67.1
76.16
45.5
8.2
27.0
56.6
61.62
52.2
8.4
33.8
67.2
75.16
50.9
12.9
33.4
63.6
75.52
53.7
34.0
66.6
76.02
33.7
11.2
34.0
66.6
76.02
Table 4: Ablation study on hybrid scale exemplars and dynamic partition. 'Hybrid' denotes enabling hy- brid scale exemplars during training and 'dynamic' denotes enabling dynamic partition during inference. The most significant gain is observed for small in- stances where PQ increases by 5. 5... This verifies our claim for alleviating the instance scale problem.
Table 5: Ablation study on seeds and segment soft- max classifier. Two convolutional layers (and a soft- max classifier) to map the features onto semantic la- tent space are still necessary to produce good predic- tions (+7% in PQ). We also observe the seed predic- tions help the merging process and boost the perfor- mance (+-2% in PQ).
A.7 HYPER- PARAMETRR AND ABLATION STUDY
Slyper-prameteer We train the models Cityscapes for 90k iterations. The batch size is set to 8, and the crop size to 704. On VOC dataset, the training iteration is 60k, the batch size 12 and crop size is 512. We adopt the standard poly learning rate policy with the base learning rate, momentum, and weight decay as 0.002, 0.9, and 0.0001, respectively.
For Panoptic Segment Sort, set the hyper-parameters for all the experiments follow. The dimension of panoptic embedding is 64. The concentration is set as 8, and the iterations in K-Means is 10. On Cityscapes, we 196 clusters in K-Means for training. During inference, we set the number of clusters as 500 and the threshold T's of dynamic partition as 0.99. On VOC dataset, the number of cluster is 64. We perform dynamic partition with T's as 0.99 for inference. The small instance area upper bound Ap is set to 2048 and 512 for Cityscapes and VOC dataset.
For the seeding branch, we generate the Gaussian heatmap by setting the radius 16 pixels. We train the pixels within the radius to predict the offsets. The overall loss weights for the SegSort loss, cross-entropy softmax loss, LI and L2 seed location loss are 1, 0.1 and 1.
Post-processine We perform minimal post-processing on the final predictions. For stuff' categories, we set the region with areas less than 2, 048 void. For thing categories, if non-seed segments after merging has area less than 4096, we set them to void as well.
Ablation study on hybrid scale exemplars and dynamic partition. The ablation study is summa- rized in Table We enable and/or disable the hybrid scale exemplars during training and dynamic partition during inference. We train the network with only 30k iterations on Cityscapes for faster experimentation. 'Hybrid" denotes enabling hybrid scale exemplars and 'dynamic' denotes enabling dynamic partition. Each component will boost the performance by 0.7 - 0.9% PQ and by 2.2% PQ both are enabled. The most significant gain observed for small instances where PQ increases by 5.7%. This ablation study verifies our claim for alleviating the instance scale problem.
Ablation study on seeds and segment softmax classifier. The ablation study is summarized in Table 5 We train the network with only 30k iterations on Cityscapes. We enable and/or disable the seeding branch and segment softmax classifier with two convolutional layers.
When the classifier is disabled, we perform K-Nearest Neighbor Search using the prototype features from the panoptic embeddings directly. It shows that two convolutional layers (and a softmax classi- fier) to map the features onto semantic latent space still necessary to produce good classification predictions (+7 ~ 8% in PQ). Also, We observe the seed predictions help the merging process and boost the performance by 1.5 2.8% in PQ.
19


Reviews:
[{"review_id": "yvPxFL8lKrv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n\nThis work is well written and easy to understand. The idea of assembling over-segments sorting, segments merging, dynamic partitioning, and seed selection is interesting, which can be applied to many downstream tasks, like panoptic segmentation and instance relationship modeling.\n\nCons\n\n1 This work emphasizes that the proposed approach can encode and discover object-centric context. Besides, it further designs experiments to compare the context errors between UPSnet and the proposed one. However, I am not quite sure what is the use of object-centric context? Is it meaningful for some applications?\n\n2 Even I consider the proposed idea is interesting, the overall pipeline seems more complex than many end-to-end panoptic segmentation frameworks such as UPSnet and panoptic-deeplab. Besides, the work does not achieve better performance than previous solutions. \n\n3 How about the generalization ability of this work? For example, can PSS be trained on Cityscapes and applied to coco-stuff based on a few-shot setting (i.e. only a few training samples are available)?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MKxMcEnHZZG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper adapts Segment Sorting to panoptic segmentation and proposes a Panoptic Segment Sorting (PSS). The proposed method learns to sort segments according to both of its semantic and instance labels. The semantic label is acquired by simply mapping and classifying prototype feature and instances are formed by a clustering algorithm. A seeding branch is further used to guide merging and avoid false positives.\n\n##########################################################################\n\nPros:\n\n- The overall idea is very interesting, instead of using separate models (or branches) to learn instance segmentation and semantic segmentation, the proposed method directly learns a unified embedding that encodes both instance and semantic information. This make the model more consistent.\n- It is also shown in the experiment that the panoptic embedding is able to encode object-centric context.\n\n##########################################################################\n\nCons:\n\n- It is still not clear to me how does the learned panoptic embedding able to encode object-centric context. Throughout the paper I don't see any specific design to optimize to learn context and the authors also do not provide theoretical explanation. If this is the case, I don't think it is suitable to name it \"contextual image parsing\" and put \"discovering object-centric context\" in the motivation (or the goal), since I think it is not well studied.\n- If the proposed method indeed learns object-centric context, I think it is more reasonable to apply the method to tasks where object-centric context is important, e.g. relation detection.\n\n##########################################################################\n\nTypos:\n\n- Last sentence is not finished in paragraph 2 of Section 1, \"A major\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "CiW4uAd7eOB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros**\n\n* Competitive results on benchmarks. The PQ results are good, with relatively few additional \"tricks\" (assuming they're not simply left out of experimental section).\n\n* Complete ablation studies. This includes swapping in/out some proposed improvements to the underlying SegSort training, plus different components of the method used to construct panoptic segmentations from the oversegmentation.\n\n* Makes additional interesting observations about the behavior of the model/properties of the embeddings, in looking at the \"context.\"\n\n**Cons**\n\n* The motivation is somewhat unclear. The core \"segment sorting\" method is an unsupervised method. The method here only builds on the supervised variant of SegSort and introduces additional components that require supervision. So the main difference from existing panoptic segmentation methods is the use of an intermediate segmentation. The authors propose the embeddings are useful in \"context specific instance retrieval,\" but the importance of this task is unproven. The fact that it seems to improve panoptic segmentation results is more concrete, but it is less clear *why* this method would yield this improvement.\n\n* Description of hybrid scale exemplars is unclear, with very few details.\n\n**Typos**\n\n  * \"objects in different context\" -> \"objects in different contexts\" on page 1\n  * Typo \"differnet\" on p1\n  * Incomplete sentence \"A major...\" on p1\n  * \"equivalent to maximize the expected number\" -> \"equivalent to maximizing the expected number\" on p5\n  * Double periods after \"segment\" at top of p6\n  * \"to be certain [a] instance category\" on p7", "extra_scores": {}, "extra_text": {}}, {"review_id": "8vcUH1mVx95", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a pixel-wise embedding strategy for panoptic segmentation, which aims to learn a pixel representation that encodes both semantic and instance information. To this end, the proposed method builds on top of the Segment Sorting approach and extends its contrastive loss to the instance level by utilizing panoptic supervision. To predict instance segmentation,  the paper also designs a merging process to cluster the pixels into instances, which further employs an object center prediction module for localization and a dynamic partition strategy to cope with scale variation.  This method is evaluated on two panotpic segmentation benchmarks, including Cityscapes and PASCAL VOC 2012. \n\nStrengths:\n- The idea of learning a pixel-wise embedding for encoding instance-specific visual context seems interesting, which can be used for predicting both thing and stuff categories. \n\n- The performance of the proposed panoptic segmentation pipeline is comparable or slightly better than the published SOTA on two benchmarks.  \n\nConcerns:\n- The focus of this work is a bit unclear. It seems trying to tackle two related problems, one is the learning an object instance-specific visual context representation and the other is the panoptic segmentation. However, neither of them is explored in depth or with compelling results in this paper.     \n\n- The technical contribution of this work is a bit incremental. Most of the learning framework is based on the Segment Sorting paper and its extension to instance-level supervision seems straightforward. \n\n-  While learning embedding for object-centeric context is useful for certain applications, it also raise the question on the complexity of such holistic features and how well such a representation can generalize to different scenes.  Due to the compositionality of object and its surroundings, conceptually it seems challenging to learn such representations with limited model capacity. In particular, how well the learned representation is able to cope with an object out of its common context? \n\n- The evaluation of the panoptic embedding is a bit lacking. First, as the method is fully supervised, all different types of object-centric context do exist in the ground-truth label maps. As such, it seems while contexts are not annotated, they should be able to learned by the model via segmentation loss? Second, the CE metric is hard to interpret as it is a KL divergence measure. It would be more clear if common retrieval performance metrics can be adopted to show its improvement. Finally, the paper only compares two representations for object-centric embedding and it would more informative to show the performance of other types of features, such as SegSort, One-stage instance segmentation, etc.  \n\n- For the panoptic segmentation, the proposed pipeline adopts a complex post-processing stage to merge the pixel representations into instance segmentation. Without detailed ablative study, it seems unclear what components contribute to the final performance. Is the learned embedding or a better merging strategy? Moreover, the common practice in panoptic segmentation literature is to conduct evaluation on both COCO and CityScapes, and report detailed metrics on PQ, SQ, RQ. Here the COCO benchmark and some metrics are missing.   \n\nMinor comments:\nSec 1: Paragraph 2: extra words at the end: \"A major\"\n\n=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response and the efforts in the updated draft. Unfortunately,  I still think the results on the object representation is not convincing due to lack of comparisons with other embedding methods, and more needs to be done to study the generalizability of this method for complex non-street scenes.  I retain my original decision for these reasons.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were split (with all scores hovering around borderline) and I found it difficult to reach a conclusion. I like the paper, and agree with the authors that it may offer an interesting "middle ground" between bottom-up and top-down approaches. On the other hand, I was concerned with some of the execution flaws that were brought up in the reviews, in particular, insufficient comparisons to other embedding methods, lack of results on COCO, and to a significant degree, lack of focus in presentation. I think this could be a much stronger paper, and it will benefit from additional time to line up those missing components.
To clarify the concern re: experiments on COCO, since the authors bring up computational constraints: I agree that running these experiments during the rebuttal period is not a reasonable expectation. But the conclusion is that these results should have been in the original submission! Semantic/panoptic segmentation is now a very mature area, and COCO (along with CityScapes) is one of the standard benchmarks. (BTW, "different methods often perform similarly"  on COCO and CityScapes, but not always -- partially due to the significant differences in the statistics of the two datasets). I don't think it's reasonable to have a submission in this area which does not include results on it, since it makes it very hard to assess how much empirical progress is being made.
Decision: Reject