K-Net: Towards Unified Image Segmentation
Authors: Wenwei Zhang
Jiangmiao Pang
Kai Chen
Chen Change Loy
Venue: NeurIPS 2021 Poster (2021)
Keywords: Image Segmentaion
Dynamic Kernels

Abstract: Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at https://github.com/ZwwWayne/K-Net/.


K-Net: Towards Unified Image Segmentation
Wenwei Zhang'
Jiangmiao Pang Kai Chen 3,4 Chen Change Loy" 1X4
Ls-Lab Nanyang Technological University
CLUHK-SenseTiee Joint Lab, the Chinese University of Hong Kong
$ Time Research *Shanghai Al Laboratory
{wenwei001, Entu. edu. sg pangjiangmiao@@maae.. com chenkai @sensetime..................ee.m com
Abstract
Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by group of learnable kernels, where each kernel is responsible for generating a mask for either potential instance or stuff class. To remedy the difficulties of distinguishing various instances, we propose kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state- of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantie segmentation mloU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN MS COCO with 60%-90% faster inference speeds. Code and models will be released at at/httss: //tp://///tttttt/ com/ Setm///owaanee/-Nee..
ADE20K val split with 55.2% PQ and 54.3%
Introduction
Image segmentation aims at finding groups of coherent pixels 48 There different notions in
groups, such as semantic categories (e.g., car, dog, cat) or instances (e.g., objects that coexist in the same image). Based the different segmentation targets, the tasks are termed differently, i.e., semantic and instance segmentation, respectively. There are also pioneer attempts [19 20 to joint the segmentation tasks for more comprehensive scene understanding.
Grouping pixels according to semantic categories can be formulated as a dense classification problem. As shown in Fig. (a), recent methods directly learn set of convolutional kernels (namely semantic kernels in this paper) of pre-defined categories and use them to classify pixels (40) or regions (22). Such framework is elegant and straightffrward. However, extending this notion to instance segmentation is non-trivial given the varying number of instances across images. Consequently, instance segmentation is tackled by more complicated frameworks with additional steps such as object detection [22] or embedding generation [44). These methods rely on extra components, which must guarantee the accuracy of extra components to reasonable extent, or demand complex post- processing such as Non-Maximun Suppression (NMS) and pixel grouping. Recent approaches [34] 55) generate kernels from dense feature grids and then select kernels for segmentation to simplify the frameworks. Nonetheless, since they build upon dense grids to enumerate and select kernels, these methods still rely on hand-crafted post-processing eliminate masks or kernels of duplicated instances.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Semintic
lnstaled
Sementic
Kernels
Kernels
Instance
Kernels
Semantic
Instanee
Panuptic Musks
Masks
Maska
(a) Semamiic Segmentation Fmmework
(b) K-Net Instance Segmenttation
(c)
Panoptic Segmentation
Figure 1: Semantic segmentation (a), instance (h), and panoptic segmentation (c) tasks unified common framework this paper. In conventional semantic segmentation methods, each convolutional kernel corresponds to a semantic class. Our framework extends this notion to make each kernel corresponds to either potential instance or semantic class.
In this paper, we make the first attempt to formulate unified and effective framework to bridge the seemingly different image segmentation tasks (semantic, instance, and panoptic) through the notion of kernels. Our method is dubbed K-Net ('K' stands for kemels).. It begins with set of convolutional kernels that are randomly initialized, and learns the kernels accordance to the segmentation targets at hand, namely, semantic kernels for semantic categories and instance kernels for instance identities (Fig. (b)). A simple combination of semantic kernels and instance kernels allows panoptic segmentation naturally (Fig. (c)). In the forward pass, the kernels perform convolution the image features to obtain the corresponding segmentation predictions.
The versatility and simplicity of K-Net made possible through two designs. First, we formulate K-Net so that dynamically updates the kernels to make them conditional to their activations on image. Such a content-aware mechanism is crucial ensure that each kernel, especially instance kernel, responds accurately to varying objects an image. Through applying this adaptive kernel update strategy iteratively, K-Net significantly improves the discriminative ability of the kernels and boosts the final segmentation performance. It is noteworthy that this strategy universally applics to kernels for all the segmentation tasks.
Second, inspired by recent advances in object detection (4), we adopt the bipartite matching strat- egy [47] to assign learning targets for each kernel. This training approach is advantageous to conventional training strategies [37 [66] as builds a one-to-one mapping between kernels and instances in image. It thus resolves ves the problem of dealing with varying number of instances in an image. addition, it is purely
without involving boxes. Hence, K-Net is naturally NMS-free and box-free, which is appealing real-time applications.
To show the effectiveness the proposed unified framework on different segmentation tasks, we conduct extensive experiments on COCO dataset (38) for panoptic and instance segmentation, and ADE20K dataset [70] for semantic segmentation. Without bells and whistles, K-Net surpasses all previous state-of of-e--of-the-at single-model results on panoptic (54.6% PQ) and semantic segmentation benchmarks (54.3% % mloU) and achieves competitive performance compared to the more expensive Cascade Mask R-CNN [3]. We further analyze the learned kernels and find that instance kernels incline to specialize on objects at specific locations of similar sizes.
2 Related Work
Semantic Segmentation. Contemporary semantic segmentation approaches typically build upon a fully convolutional network (FCN) (40) and treat the task as a dense classification problem. Based this framework, many studies focus enhancing the feature representation through dilated convolution (7-9), pyramid pooling (57 67). context representations [64]65), and attention
mechanisms [32 63 68 Recently, SETR (69 reformulates the task sequence-to-seqqeence
prediction task by using a vision transformer 17). Despite the different model architectures, the approaches above share the common notion of making predictions via static semantic kernels. Differently, the proposed K-Net makes the kernels dynamic and conditional on their activations in the image.
Instance Segmentation. There are two representative frameworks for instance segmentation - 'top- down' and 'bottom-up' approaches. 'Top-down' approaches (2] 15
22 27 33 first detect accurate

---

bounding boxes and generate mask for each box. Mask R-CNN (22) simplifies this pipeline by directly adding a FCN [40] in Faster R-CNN [46]. Extensions of this Iramework add a mask scoring branch (24) or adopt cascade structure 3 S.
first perform semantic segmentation then group pixels into different instances. These methods usually require a grouping process, and their performance often appears inferior to 'top-down' approaches in popular benchmarks [14]38], Unlike all these works, K-Net performs segmentation and instance separation simultaneously by constraining each kernel to predict one mask at time for one object. Therefore, K-Net needs neither bounding box detection nor grouping process, It focuses on refining kernels rather than refining bounding boxes, different from previous cascade methods 33.)
'Bottom-up' methods 30 43 44
Recent attempts [1 55 58) perform instance segmentation in one stage without involving detection nor embedding generation. These methods apply dense mask prediction using dense sliding windows [10] or dense grids (54). Some studies explore polar (58) representation, contour (45). and explicit shape representation (60] of instance masks. These methods all rely on NMS to eliminate duplicated instance masks, which hinders end-to-end training. The heuristic process also unfavorable for real-time applications. Instance kemels in K-Net are trained in an end-to-end manner with bipartite matching and set prediction loss, thus, our methods does not need NMS.
Panoptic Segmentation. Panoptic segmentation (29 combines instance and semantic segmentation to provide a richer understanding of the scene. Different strategies have been proposed to cope with the instance segmentation task. Mainstream frameworks add a semantic segmentation branch [28] 31
55 [5] on an instance segmentation framework or adopt different pixel grouping strategies [TI 61
based on a semantic segmentation method. Recently, DETR (4) tries to simplify the framework
by transformer [5T) but need to predict boxes around both stuff and things classes in training for assigning learning targets. These methods either need object detection or embedding generation to separate instances, which does not reconcile the instance and semantic segmentation a unified framework. By contrast, K-Net partitions an image into semantic regions by semantic kernels and object instances by instance kernels through a unified perspective of kernels.
apply Transformer [51] for panoptic seg- mentation. MaskFormer reformulates semantic segmentation as a mask classification task, which is commonly adopted in instance-level segmentation. From an inverse perspective, K-Net tries to simplify instance and panoptic segmentation by letting a kernel to predict the mask of only one instance or semantic category, which is the essential design in semantic segmentation. In contrast to K-Net that directly uses learned kernels to predict masks and progressively refines the masks and kernels, MaX-DeepLab (52) and MaskFormer [12] rely on queries and Transformer [5] to produce dynamic kernels for the final mask prediction.
Concurrent to K-Net, some recent attempts [2 35 52
Dynamic Kernels. Convolution kernels are usually static, i.e., agnostic to the inputs, and thus have limited representation ability. Previous works [16 25 |5| [6] explore different kinds of dynamic kernels to improve the flexibility and performance of models. Some semantic segmentation methods apply dynamic kernels improve the model representation with enlarged receptive fields (56) or multi-scales contexts (20). Differently, K-Net uses dynamic kernels to improve the discriminative capability of the segmentation kernels more so than the input features of kernels.
Recent studies apply dynamic kernels to generate instance [49|33, or panoptic (34 segmentation predictions directly. Because these methods generate kernels from dense feature maps, enumerate kernels of each position, and filter out kernels of background regions, they either still rely on NMS 49 55) need extra kernel fusion (34] to eliminate kernels masks of duplicated objects. Instead of generated from dense grids, the kernels in K-Net are a set of learnable parameters updated by their corresponding contents in the image. K-Net does not need to handle duplicated kernels because its kernels learn to focus on different regions of the image in training, constrained by the bipartite matching strategy that builds a one-to-one mapping between the kernels and instances.
3 Methodology
We consider various segmentation tasks through a unified perspective of kernels. The proposed K-Net uses a set of kernels to assign each pixel to either potential instance or semantic class (.Tp. To enhance the discriminative capability of kernels, we contribute way to update the static kernels by the contents in their partitioned pixel groups (Sec. 3.2). We adopt the bipartite
3

---

matching strategy to train instance kernels in an end-to-end end manner (Sec. 3.3). K-Net can be applied seamlessly semantic, instance, and panoptic segmentation as described in Sec. 3.4
3.1 K-Net
Despite the different definitions of a mmeaninfful group'. all segmentation tasks essentially assign each pixel to one of the predefined meaningful groups [48]. As the number of groups in image is typically assumed finite, we can set the maximum group number of a segmentation task as N. For example, there are N pre-defined semantic classes for semantic segmentation an image for instance segmentation. For panoptic segmentation. N the total number of stuff classes and objects in image. Therefore, we use N kernels partition image into N groups, where each kernel is responsible to find the pixels belonging to its corresponding group. Specifically, given an input feature map F R& xBBxCxt X of B images, produced by deep neural network, we only need N kernels K E RNNC xC to perform convolution with F to obtain the corresponding segmentation prediction M E BBxNxHXX
most N objects in
as
o * F);
(I)
where C, H, and W are the number of channels, height, and width of the feature map, respectively. The activation function a can softmax function we want to assign each pixel to only one of the kernels (usually used in semantic segmentation). Sigmoid function can also be used as activation function if we allow one pixel belong to multiple masks, which results on N binary masks by setting threshold like 0.5 the activation map (usually used in instance segmentation).
This formulation has already dominated semantic segmentation for years (8 40 67]. In semantic segmentation, each kernel is responsible to find all pixels of a similar class across images. Whereas in instance segmentation, each pixel group corresponds an object. However, previous methods separate instances by extra steps [22 44 instead of by kernlss.
This paper is the first study that explores if the notion of kernels in semantic segmentation is equally applicable to instance segmentation, and more generally panoptic segmentation. To separate instances by kernels, each kernel in K-Net only segments most one object in image (Fig- (b-) In this way, K-Net distinguishes instances and performs segmentation simultaneously. achieving instance segmentation in one pass without extra steps. For simplicity, we call these kernels as semantic and instance kernels in this paper for semantic and instance segmentation, respectively. A simple combination of instance kernels and semantic kernels can naturally preform panoptic segmentation that either assigns a pixel to an instance ID or class of stuff (Fig. (c)).
3.2 Group-Aware Kernels
Despite the simplicity of K-Net, separating instances directly by kernels is non-trin non-trivial. Because instance kernels need to discrim- inate objects that vary in scale and appear- ance within and across images. Without a common and explicit characteristic like semantic categories, the instance kernels need stronger discriminative ability than static kernels
M
M
pk
Class Predictions
K
Kernel Update
Adaprive
R
Kernel Interaction
Kernel Update
Head (f)
To overcome this challenge, we contribute an approach to make the kernel conditional
Convolation
Conv
FC
Identity
Multiplication
Figure 2: Kernel Update Head.
on their corresponding pixel groups, through a kernel update head, as shown in Fig. The kernel update head fi contains three key steps: group feature assembling, adaptive kernel update, and kernel interaction. Firstly, the group feature F for each pixel group is assembled using the mask prediction M,-1- As it is the content of each individual groups that distinguishes them from each other, FK is used to update their corresponding kernel K;- adaptively. After that, the kernel interacts with each other comprehensively model the image context. Finally, the obtained group-aware kemels K; perform convolution over feature map F' to obtain more accurate mask prediction M,. As shown in Fig. 3 this process can be conducted iteratively because finer partition usually reduces the noise in group features, which results in more discriminative kernels. This process is formulated as
K4, M, fi (Mi-1, F).
(2)

---

Semantic Kemels
Instance Kemels
Class Predictions
Class Predictions
Kemel Update Head fz
Kerell Update Headd Js
NxC
NxC
repeat 2
Learned Kernels
Kg NxD
Dynumic Kermess
Dynamic Kernels
OUTPUL
times
Kgt
Mo: NXNXW
Mask Predictions
Mask Predictions
Mask Predictions
My. NXHXW
Mg. NxHxW
Backbone
Nock
Instance
Semantic
Identity
Fearure Map F: CxHxW
Convolution
Convolution Layer
Figure 3: K-Net for panoptic segmentaion. A set of learned kernels first performs convolution with the feature map F to predict masks Mo. Then the kernel update head takes the mask predictions Mo. learned kernels Ko, and feature map input and produce class predictions, group-aware (dynamic) kernels, and mask predictions. The produced mask prediction, dynamic kernels, and feature map are sent to next kemnel update head. This process is performed iteratively to progressively refine the kemelss and the mask predictions.
Notably, the kernel update head with the iterative refinement is universal it does not rely on the characteristic of kernels. Thus, it can enhance not only instance kernels but also semantic kernels. We detail the three steps as follows.
Group Feature Assembling. The kernel update head first assembles the features of each group, which will be adopted later to make the kernels group-aware. As the mask of each kernel in M,-1 essentially defines whether or not a pixel belongs to the kernel's related group, we can assemble the feature FK for K by multiplying the feature map with the Mi-1 as
F
F(u, F) F E RBBNXC
where B is the batch size, N is the number of kernels, and C is the number of channels.
Adaptive Feature Update. The kernel update head then updates the kernels using the obtained F to improve the representation ability of kernels. As the mask M,-1 may not be accurate, which is more common the case, the feature of each group may also contain noises introduced by pixels from other groups. To reduce the adverse effect of the noise in group features, we devise an adaptive kernel update strategy. Specifically, we first conduct element- wise multiplication between FK and K;-1 as
F = Φ1 (F*) @ (2(Kk-1), F E Bx
(4)
where Φ and Φ are linear transformations. Then the head learns two gates, GF and GK, which adapt the contribution from FK and K;- to the updated kernel K, respectively. The formulation
GK = ( GF = =o(
(5)
G= & b + GK & (K;
where on, n = 1,..., 4 are different fully connected (FC) layers followed by LayerNorm (LN) and the Sigmoid function. K is then used kernel interaction.
The gate learned here plays a role like the self-attention mechanism in Transformer (11), whose output is computed as a weighted summation of the values. In Transformer, the weight assigned to each value is usually computed by compatibility function dot- of the queries and keys. Similarly, adaptive kernel update essentially performs weighted summation of kernel features K;- and group features FG, Their weight GK and G' are computed by clement-wiss multiplication, which can be regarded as another kind of compatibility function.
Kernel Interaction. Interaction among kernels is important to inform each kernel with contextual information from other groups. Such information allows the kernel implicitly model and exploit the relationship between groups of an image. To this end, we add kernel interaction process to obtain the new kernels K; given the updated kernels K. Here we simply adopt Multi-Heal Attention (5T) followed by a Feed-Forward Neural Network, which has been proven effective in previous works 51). The output K; of kernel interaction is then used to generate new mask prediction through M. = g: (K,) F. where di is an FC-LN-ReL.U layer followed by an FC layer. K; will also be used to predict classification scores in instance and panoptic segmentation.
5

---

3.3 Training Instance Kernels
While each semantic kernel can be assigned to constant semantic class, there lacks explicit rule to assign varying number of targets to instance kernels. In this work, we adopt bipartite matching strategy and set prediction loss [4] to train instance kernels in end-to-end manner. Different from previous works [4 47] that rely on boxes, the learning of instance kernels is purely mask-driven because the inference of K-Net is naturally box-free.
Loss Functions. The loss function for instance kernels is written as Lx = Acte + AceLcce Adice Ldice,, where Lete is Focal loss (37] for classification, and Lee and Ldsce are CrossEntropy (CE) loss and Dice loss [42] for segmentation, respectively. Given that each instance only occupies a small
region in an image, CE loss is insufficient to handle the highly imbalanced learning targets of masks. Therefore, we apply Dice loss
[2) to handle this issue following previous works 49 54
Mask-based Hungarian Assignment. We adopt Hungarian assignment strategy used in 47 for target assignment to train K-Net in an end-to-end manner. It builds a one-to-one mapping between the predicted instance masks and the ground-truth (GT) instances based on the matching costs. The matching cost is calculated between the mask and GT pairs in a similar manner as the training loss.
3.4 Applications to Various Segmentation Tasks
Panoptic Segmentation. For panoptic segmentation, the kernels are composed of instance kernels Kg and semantic kernels Kg as shown in Fig. 3 We adopt semantic FPN (28) for producing high
resolution feature map F. except that we add positional encoding used in 4 SI 72 to enhance the positional information. Specifically, given the feature maps P2, P3. P4, P5 produced by FPN (36). positional encoding is computed based on the feature map size of P5, and it is added with P5. Then semantic FPN (28 used to produce the final feature map.
As semantic segmentation mainly relies on semantic information for per-pixel classification, while instance segmentation prefers accurate localization information to separate instances, we two separate branches to generate the features Frins and Fsem to perform convolution with Kine and K0 Keem for generating instance and semantic masks M and M respectively. Notably, it is unnecessary to produce 'thing' and 'stuff" masks initially from different branches to produce and empirically
reasonable performance. Such design is consistent with previous practices (8 55
yields better performance (about 1 % PQ).
We then construct Mo. Kg. and F as the inputs of kernel update head to dynamically update the kernels and refine the panoptic mask prediction. Because 'things' are already separated by instance masks in Mine, while M6TT contains the semantic masks of both 'things' and 'stuff', we select Mo', the masks of stuff categories from Mgem, and directly concatenate it with Mé to form the panoptic mask prediction Mo- Due to similar reason, we only select and concatenate the kernels of stuff classes in Koemm with Kinss to form the panoptic kernels Ko. To exploit the complementary semantic information in psem and localization information in pinn, we add them together to obtain F' as the input feature map of the kernel update head. With Mo. Ko. and F, the kernel update head fi can produce group-aware kernels K₁ and mask M1. Then kernels and masks are iteratively by S times and finally we can obtain the mask prediction Ms.
To produce the final panoptic segmentation results, we paste thing and stuff masks in mixed order following FaskFormer (12). We also find it necessary in K-Net to firstly sort the pasting order of masks based on their classification scores for further filtering out lower-confident mask predictions. Such a method empirically performs better (about 1 % PQ) than the previous strategy that pasting thing and stuff masks separately (28 34
Instance Segmentation. In the similar framework, we simply remove the concatenation process of kernels and masks to perform instance segmentation. We did not remove the semantic segmentation branch as the semantic information is still complementary for instance segmentation. Note that in this case, the semantic segmentation branch does not use extra annotations. The ground truth of semantic segmentation is built by converting instance masks to their corresponding class labels.
Semantic Segmentation. As K-Net does not rely on specific architectures of model representation, K-Net can perform semantic segmentation by simply appending its kernel update head to any existing semantic segmentation methods [8 40 57 67 that rely on semantic kernels.

---

Table 1: Comparisons with state-offthe-art panoptic segmentation methods COCO dataset
Framework
Backbone
Box-free NMS-fice
Epochs
PQ
PQ
PQ
val
Panoptic-Dece .ab I
Xception-71
1000
39.7
43.9
33.2
Punoptic HPN 28
R50-FPN
36
41.3
48.5
31.1
SOLOv2
15
R50-FPN
36
42.1
49.6
30.7
DETR
R50
300 25
43.4
48.2
36.3
Unifying 31
R50-FPN
222
43.4
48.6
35.5
Panoptic FCN
34
R50-FPN
36
43.6
49.3
35.0
K-Net
R50-FPN
36
47.1
51.7
40.3
RI01-FPN
36
49.6
55.1
41.4
K-Net
RI0I-FPN-DCN
36
48.3
54.0
39.7
Swin-L
39
36
54.6
60.2
46.0
teat-des
Sanoptic-Deeplee
Xception-7l
1000
41.4
45.1
35.9
Panoptic FPN
R101-FPN
36
43.5
50.8
Panoptic FCN
RI0I-FPN
36
45.5
51.4
36.4
DETR
R101
300 25
46.0
UPSNet
59
RI0I-FPN-DCH
36
46.6
53.2
36.7
Unifying
3T
R101-FPN-DCN
27
47.2
53.5
37.7
K-Net
R101-FPN
36
47.0
52.8
38.2
K-Net
R101-FPN-DCN
36
48.3
54.0
39.7
MaX-DocpLabbLL
52
Max-L
54
51.3
57.2
42.4
MaskFormer
12
Swin-L 39
300
53.3
59.1
44.5
Panoptic SegiFormer
PVTv2-BS
53
50
54.4
61.1
44.3
K-Net
Swin-L
36
55.2
61.2
46.2
Experiments
Dataset and Metrics. For panoptic and instance segmentation, perform experiments the challenging COCO dataset (38). All models are trained on the train2017 split and evaluated on the va12017 split. The panoptic segmentation results are evaluated by the PQ metric (29). We also report the performance of thing and stuff, noted as PQ PQ respectively, for thorough evaluation. The instance segmentation results are evaluated by mask AP (38]. The AP for small, medium and large objects are noted as AP., AP ms and AP;, respectively. The AP at mask IoU thresholds 0.5 and 0.75 are also reported as AP30 and AP75, respectively. For semantic segmentation, experiments on the challenging ADE20K dataset [70] and report mIoU to evaluate the segmentation quality. All models are trained on the train split and evaluated on the validation split.
conduct
Implementation Details. For panoptic and instance segmentation, we implement K-Net with MMDetection (6). In the ablation study, the model is trained with batch size of 16 for 12 epochs. The learning rate is 0.0001, I, and it is decreased by 0.1 after 8 and 11 epochs, respectively. We use AdamW 41 with a weight decay of 0.05. For data augmentation in training, augmentation with single scale. The long edge and short edge of images are resized to 1333 and 800, respectivlly without changing the aspect ratio. When comparing with other frameworks, we use multi-scale training with a longer schedule (36 epochs) for fair comparisons 66] The short edge of images randomly sampled from [640, 800] [21).
adopt horizontal flip
For semantic segmentation, we implement K-Net with MMSegmentation 13 and train it with 80,000 iterations. As AdamW [41) empirically works better than SGD, of 0.0005 by default on both the baselines and K-Net for fair comparison. The initial learning rate 0.0001, and is decayed by 0.1 after 60000 and 72000 iterations, respectively. More details provided in the appendix.
AdamW with weight decay
Model Hyperparameters. In the ablation study, adopt ResNet-50 23 backbone with FPN [36]. For panoptic and instance segmentation, we use Act= = 2 for Focal loss following previous meth- ods (22, and empirically find A 1, Ace 1, Adice 4 work best. For efficiency, the default number of instance kernels is 100. For semantic segmentation, N equals to the number of classes of the dataset, which is 150 in ADE20K and 133 in COCO dataset. The number of rounds of iterative kernel update is set to three by default for all segmentation tasks.
4.1 Benchmark Results
Panoptic Segmentation. We first benchmark K-Net with other panoptic segmentation frameworks T K-Net surpasses the previous state-of-the-an box-based method (3T) and box/NMS-free method 34 by 1.7 and 1.5 PQ val split, respectively. On the test-dev split, K-Net with ResNet-
7

---

Table 2: Comparisons with state-of-the-ar instance segmentation methods COCO dataset. 'P. (M) indicates the number parameters in model, and counting unit is million
Method
Backhone Box-freel NMS-free Epochs
APT
APso
AP
AP,
APm
AP:
FPST P. (M)
va12017
SOLO 54
R-S0-FPN
36
35.8
56.7
37.9
14.3
39.3
53.2
12.7
36.08
Mask R-CNN 22
R-50-FPN
36
37.1
58.5
39.7
18.7
39.6
53.9
17.5
44.17
SOLOv2 55
R-50-FPN
37.5
58.2
40.0
15.8
41.4
56.6
17.7
33,89
CondInst
R-50-FPN
37.5
58.5
18.7
41.0
53.3
14.0
46.37
Cuscude Mask R-CNN
R-50-FPN
36
38.5
59.7
41.8
19.3
41.1
55.6
10.3
77.10
K-Net
R-50-FPN
36
37.8
60.3
39.9
16.9
41.2
57.5
21.2
37.26
K-Net-N256
R-50-FPN
36
38.6
60.9
41.0
19.1
42.0
57.7
19.8
37.30
Lest-dev
SOLO
R-50-IPN
72
36.8
58.6
39.0
15.9
39.5
52.
12.7
36,08
Mask R-CNN
R-50-FPN
37.4
59.5
40.1
18.6
39.8
51.6
17.5
44.17
CondInst
R-50-FPN
37.8
59.2
18.2
40.3
52.7
14.0
46.37
SOLOv2
R-50-FPN
36
38.2
59.3
40.9
16.0
41.2
55.4
17.7
33.89
Cascade Mask R-CNN
R-50-FPN
36
38.8
60.4
42.0
19.4
40.9
53.9
10.3
77.10
K-Net
R-50-FPN
36
38.4
61.2
40.9
17.4
40.7
56.2
21.2
37.26
K-Net-N256
R-50-FPN
36
39.1
61.7
41.8
18.2
41.4
56.6
19.8
37.30
SOLO
R- R-101-PPN
72
37.8
59.5
40.4
16.4
40.6
54.2
10.7
55.07
Mask R-CNN
R-101-FPN
38.8
60.8
41.8
19.1
41.2
54.3
14.3
63.16
CondInst
R-101-FPN
38,9
60.6
18.8
41.8
S4.4
11.0
52.83
SOLOv2
R-101-PPN
36
39.5
60.8
42.6
16.7
43.0
S7.4
14.3
65.36
Cascude Musk R-CNN
R-101-FPN
36
39.9
61.6
43.3
19.8
42.1
55.7
9.5
96.09
K-Net
R-101-FPN
36
40.1
62.8
43.1
18.7
42.7
58.8
16,2
56.25
K-Net-N256
R-101-FPN
36
40.6
63.3
43.7
18.8
43.3 59.0
15.5
56.29
Table Results of K-Net on ADE20K semantic segmentation dataset
(a) Improvements of K-Net different architec-
(b) Comparisons with state-of-the-art methods. Re- sults marked by use larger image sizes
tures
Method
Backbone
Val mloU
Method
Backhone
Val mloU
FCN
[40]
R50
36.7
OCRNet
64
HRNet- W48
44.9
FCN
K-Net
R50
43.3 (46.6)
PSPNet
67
R101
45.4
PSPNNt 67
R50
42.6
PSANet
68
R101
45.4
PSPNet
K-Net
R50
43,9 (+1.3)
DNL 63
R101
45.8
DLab.v3
R50
43.5
DLab.v3
R101
46.7
DLab.v3 K-Net
RS0
44.6
DLab.v3-
S-101 66
47.3
UperNet 57
R50
42.4
SETR 69
VIT-L 17
48.6
UperNet K-Net
R50
(+1.))
UperNet
Swin-L -
53.5
UperNet
Swin-T.
50.6
UperNet K-Net
Swin-L
53.3
UperNet K-Net
Swin-L
52 (+1.4)
UperNet K-Net"
Swin-L 54.3
101-FPN backbone even obtains better results than that of UPSNet [59] which uses Deformable
K-Net equipped with DCN surpasses the previous method (31] by 1.1 PQ. Without bells and whistles, K-Net obtains new state-of-the-art single-model performance with Swin Transformer
Convolution Network (DCN) [16].
(39] serving as the backbone.
We also compare K-Net with concurrent work Max-DeepLee (52.. MaskFormer 12] and Panoptic SegFormer (35]. K-Net surpasses these methods with the least training epochs (36), taking only about 44 GPU days (roughly 2 days and 18 hours with 16 GPUs). Note that only 100 instance kernels and Swin Transformer with window size are used here for efficiency. K-Net could obtain a higher performance with more instance kernels (Sec. 4.2), Swin Transformer with window size 12 (used in MaskFormer [12), ), as well as an extended training schedule with aggressive data augmentation used in previous work
Instance Segmentation. We compare K-Net with other instance segmentation frameworks [10 22 49 in T More details are provided in the appendix. As the only box-free and NMS-free method, K-Net achieves better performance and faster inference speed than Mask R-CNN (22). SOLO (54), SOLOv2 (55] and CondInst [49], indicated by the higher AP and frames per second (FPS). We adopt
256 instance kernels (K- Ne-5 in the table) to compare with Cascade Mask R-CNN (3). The performance of K-Net-N256 is on par with Cascade Mask R-CNN 3 but enjoys 92.2 % faster inference speed (19.8 v.s 10.3).
On COCO test-dev split, K-Net with ResNet-101-FPN 101-FPN backbone obtains performance that is 0.9 AP better than Mask R-CNN (22). It also surpasses previous kernel-based approach CondInst (49 and SOLOv2 (55 by 1.2 AP and 0.6 AP, respectively. With ResNet- 1PI- backbone, K-Net surpasses Cascade Mask R-CNN with 100 and 256 instance kernels in both accuracy and speed by 0.2 AP and 6.7 FPS, and 0.7 AP and 6 FPS, respectively.
8

---

Table 4: Ablation studies of K-Net instance segmentation
(a) Adaptive Kemel Update (A. K. U.) and Kernel
(b) Positional Encoding (P. E.) and Coordinate Convo- lution (Coors.)
Interaction (K. I.)
K. K.
AP
APso
AP7s
Cours.
P.E.
AP
AP50
AP
10.0
18.2
9.6
30,9
51.7
31.6
22.6
37.3
23.5
34.0
55,4
35,6
31.2
52.0
32.4
34.1
55.3
35.7
34.1
55.3
35.7
34.0
55.1
35.8
(c) Number of rounds of kernel update
(d) Numbers of instance kernels
Stage Number
AP
AP50
AP7s
FPS
N
AP
AP50
APts
FPS
21.8
37.3
22.1
24.0
50
32.7
53.7
34.1
21.6
2
32.1
52.3
33.5
22.7
64
33,6
54.8
35.1
21.6
3
34.1
55.3
35.7
21.2
100
34.1
55.3
35.7
21.2
34.5
56.5
35.7
20.1
128
34.3
55.6
35.8
20.7
34.5
56.5
35.9
256
34.7
36.3
We also compare the number parameters of these models in Table 2 the least number of parameters, it is more lightweight than Cascade Mask R-CNN by approximately half number of the parameters (37.3 M vs. 77.1 M).
Though K-Net does not have
Semantic Segmentation, We apply K-Net to existing frameworks 8 40 57 67 that rely on static
semantic kernels in Table 3a K-Net consistently improves different frameworks. Notably, K- Net significantly improves FCN (6.6 mloU). This combination surpasses PSPNet and UperNet by 0.7 and 0.9 mIoU, respectively, and achieves performance comparable with DeepLab v3. Furthermore. the effectiveness of K-Net does not saturate with strong model representation, as it still brings significant improvement (1.4 mIoU) over UperNet with Swin Transformer 39 the versatility and effectiveness of K-Net for semantic segmentation.
The results suggest
In Table 3b further compare K-Net with other state-of- of-the-art methods 9 69 with test-time
augmentation on the validation set. With the input of 512x512, K-Net already achieves state-of- the- art performance. With larger input of 640x640 following previous method (39] during training and testing, K-Net with UperNet and Swin Transformer achieves new state-of-the-art single model performance, which is 0.8 mIoU higher than the previous one.
4.2 Ablation Study on Instance Segmentation
We conduct an ablation study on COCO instance segmentation dataset to evaluate the effectiveness of K-Net in discriminating instances. The conclusion is also applicable to other segmentation tasks since the design of K-Net is universal to all segmentation tasks.
Head Architecture. We verify the components in the kernel update head in Table 4a The results of without A. K. U. is obtained by updating kernels purely by K Fx K₃-1 followed by FC-LN-ReLU layer. The results indicates that both adaptive kernel update and kernel interaction are necessary for high performance.
Positional Information. We study the necessity of positional information in Table 4b The results show that positional information is beneficial, and positional encoding [1 works slightly better than coordinate convolution. The combination of the two components does not bring additional improvements. The results justify the use of just positional encoding in our framework.
Number Stages. We compare different kernel update rounds in Table 4c The results show that FPS decreases as the update rounds grow while the performance saturates beyond three stages. Such conclusion also holds for semantic segmentation shown in Table 5 The performance of FCN + K-Net on ADE20K dataset gradually increases the increase of iteration number but also saturates after four iterations.
Table 5: Numbers of semantic kernels
Stage Number mloU
2
36.7
42.7
43.0
43.3
43.1
42.6
Number of Kernels. We further study the number of kernels in K-Net. The results in Table reveal that 100 kernels are sufficient to achieve good performance. The observation is expected for COCO dataset because most of the images in the dataset do not contain many objects (7.7 objects per image in average (38). K-Net consistently achieves better performance given more instance kernels since they improve the models' capacity in coping with complicated images. However, larger N may lead

---

Before Lddat
round
3 round
(a) Average activation over 5000 images.
(b) Mask prediction hefore and after kernel update. Figure Visual analysis of kernels and their masks. Best viewed in color and by zooming in.
to small performance gains and then get saturated (when N 300, 512, 768, we all get 34.9% mAP). Therefore, we select N 100 other experiments for efficiency without further specification.
4.3 Visual Analysis
Overall Distribution of Kernels. We carefully analyze the properties of instance kernels learned in K-Net by analyzing the average of mask activations of the 100 instance kernels over the 5000 images in the val split. All the masks are resized to have similar resolution of 200 200 for the analysis. As shown Fig. 4a the learned kernels are meaningful. Different kernels specialize on different regions of the image and objects with different sizes, while each kernel attends to objects of similar sizes at close locations across images.
Masks Refined through Kernel Update. We further analyze how the mask predictions of kernels are refined through the kernel update in Fig. 4b) Here we take K-Net for panoptic segmentation to thoroughly analyze both semantic and instance masks. The masks produced by static kernels incomplete, e.g., the masks of river and building are missed. After kernel update, the contents are thoroughly covered by the segmentation masks, though the boundaries of masks still unsatisfactory. The boundaries are refined after more kernel update. The classification confidences of instances also increase after kernel update. More results given in the appendix.
5
Conclusion
This paper explores instance kernels that can learn to separate instances during segmentation. Thus, extra components that previously assist instance segmentation can be replaced by instance kernels, including bounding boxes, embedding generation, and hand-crafted post-processing like NMS, kernel fusion, and pixel grouping. Such an attempt, for the first time, allows different image segmentation tasks to be tackled through a unified framework. The framework, dubbed as K-Net, first partitions an image into different groups by learned static kernels, then iteratively refines these kernels and their partition of the image by the features assembled from their partitioned groups. K-Net obtains new of-te-of-the-arr single-mode performance on panoptic and semantic segmentation benchmarks and surpasses the well-developed Cascade Mask R-CNN with the fastest inference speed among the recent instance segmentation frameworks. We wish K-Net and the analysis to pave the way for future research unified image segmentation frameworks.
Acknowledgements. This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry psrtner(s). is also partially supported by the NTU NAP grant. Jiangmiao Pang and Kai Chen are also supported by the Shanghai Committee of Science and Technology, China (Grant No. 20DZ.1 100800). The authors would like to thank the valuable suggestions and comments by Jiaqi Wang, Rui Xu, and Xingxing Zou.
10

---

References
[1] Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In CVPR, 2017.
[2] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. YOLACT: Real-time instance segmentation. In ICCV, 2019.
[3] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In
CVPR, 2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020.
[5] Kai Chen, Jiangmiao Pang, Jiaqi Wang. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang. Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. In CVPR, 2019.
[6] Kai Chen, Jiqqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang. Dazhi Cheng, Chenchen Zhu, Tianheng Cheng. Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang. Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
arXiv:1906. 07155. 2019.
[7] Liang-Chiae Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.
TPAMI, 2018.
[8] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: /706v 05587, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encnder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[10] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár. Tensormask: A foundation for dense object segmentation. 2019.
[11] Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, and Liang- A simple, strong, and fast baseline for bottom-up panoptic segmentation.
Chich Chen.
In CVPR, 2020.
[12] Bowen Cheng. Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need
for semantic segmentation. CoRR, abs/2 107 006778 2021.
[13] MMSegmentation Contributors. MMSegmennation: Openmmlab semantic segmentation toolbox and benchmark.
com open - mml ab/ abb/mmmemmnttaiionn 2020.
[14] Marius Cordis, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson,
Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscupes dataset for semantic urhan scene understanding.
In CVPR, 2016.
[15] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. CVPR, 2016.
[16] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. 2021.
[18] Hang Gao, Xizhou Zhu, Stephen Lin, and Jifeng Dai. Deformable Kernels: Adapting effective receptive fields for object deformation. In /CLR, 2020.
[19] Stephen Gould, Tianshi Gao, and Daphne Koller. Region-baset segmentation and object detcction. In Yoshua Bengio, Dale Schuurmans, John D. Lafferty, Christopher K. Williams, and Aron Culotta, editors, NewrIPS, 2009.
[20] Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In ICCV, 2019.
[21] Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking ImageNet pre-training. In ICCV, 2019.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016.
[24] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask Scoring
R-CNN. In CVPR, 2019.
[25] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NeurlPS, 2015.
[26] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue Van Gool. Dynamic filter networks. In NeurIPS.
2016.
[27] Lei Ke, Yu-Wing Tai, and Chi- Keung Tang. Deep occlusion aware instance segmentation with overlapping bilayers. In CVPR, 2021.
11

---

[28]
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks.
In CVPR, 2019.
[29]
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation.
In CVPR. 2019.
[30]
Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savehynskyy, and Carsten Rother, Instance-
cut: From edges to instances with multicut. In CVPR, 2017.
[31]
Qizhu Li, Xiaojuan Qi, and Philip H. S. Torr. Unifying training and inference for panoptic segmentation.
In CVPR, 2020.
[32] Xia Li, Zhisheng Zhong. Jianlong Wu, Yiho Yang, Zhouchen Lin, and Hong Liu. Expectation maxxiizionn
attention networks for semantic segmentation. In ICCV, 2019.
[33] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic
segmentation. CVPR, 2017.
[34] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, 2021.
[35] Zhiqi Li, Wenhai Wang. Eaze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Tong Lu, and Ping Luo. Panoptic segformer. CoRR, ahs/2119.033814 2021.
[36] Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.
Feature pyramid networks for object detection. In CVPR, 2017.
[37] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[38] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. arviv preprint arXiv:2103.1403 2021.
[40] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[41] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[42] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks
for volumetric medical image segmentation. In 3DV, 2016.
[43] Davy Neven, Bert De Brabandere, Mare Proesmans, and Luc Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. In CVPR, 2019.
[44] Alejandro Newell, Zhiao Huang. and Jia Deng. Associative Embedding: End-to-end learning for joint
detection and grouping NeurIPS, 2017.
[45] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou. Deep snake for real-time instance segmentation. In CVPR, 2020.
[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[47] Russell Stewart, Mykhaylo Andriluka, and Andrew Y. Ng. End-to-end people detection crowded scenes.
In CVPR, 2016.
[48]
Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
[49]
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV,
2020.
[50] Zhuowen Tu, Xiangrong Chen, Alan L. Yuille, and Song Chun Zhu, Image parsing: Unifying segmentation, detection, and recognition. IJCV, 2005.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[52] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chich Chen. Max-deeplah: End-to-end panoptic segmentation with mask transformers. CoRR, abs/2012.00755 2020.
[53] Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVTv2: Improved baselines with pyramid vision transformer. CoRR, abs/2106.13797 2021.
[54] Xinlong Wang. Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. SOLO: Segmenting objects by locations. In ECCV, 2020.
[55] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOV2: Dynamic and fast instance segmentation. NeurlPS, 2020.
[56] Jialin Wu. Dai Li, Yu Yang, Chandrajit Bajaj. and Xiangyang Ji. Dynamic filtering with large sampling field for convnets. In E.CVV, 2018.
[57] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In ECCV, 2018.
[58] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. PolarMask: Single shot instance segmentation with polar representation. In CVPR, 2020.
12

---

[59]
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai. Ersin Yumer, and Raquel Urtasun. UPSNet: A unified panoptic segmentation network. In CVPR, 2019. Wenqiang Xu, Haiyang Wang. Fubo Qi, and Cewu Lu. Explicit shape encoding for real-time instance segmentation. In ICCV. 2019.
[60]
[61] Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu. Jyh-Jing Hwang. Ting Liu, Xiao Zhang, Vivienne Sze. George Papandreou, and Liang-Chiee Chen. DeeperLab: Single-shot image parser. CoRR, abs/ 19020099000 2019.
[62] Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the scene a whole: Joint object detection, scene classification and semantic segmentation. In CVPR, 2012.
[63] Minghao Yin, Zhuliang Yao. Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In ECCV, 2020. Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation.
[64]
In ECCV, 2020.
[65]
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit
Agrawal. Context encoding for semantic segmentation. In CVPR. June 2018.
[66] Hang Zhang. Chongruo Wu, Zhongyue Zhang. Zhu, Zhang, Haibin Lin, Yue Sun, Tong Hc, Jonas Muller. R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. arXiv preprint
arXiv:2004.0892 08955, 2020.
[67] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[68]
Hengshuang Zhao, Yi 7hang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wiss wise spatial attention network for scene parsing. In ECCV, 2018
[69]
Sixiao Zheng. Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, 7ekun Luo, Yahiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang. Philip H.S. Tomr and Li Zhang. Rethinking semantic segmentation from sequence-to- sequence perspective with transformers. CVPR, 2021.
[70] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, 2019.
[71] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable ConvNets V2: More deformable, hetter results. In CVPR, 2019.
[72] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and lifeng Dai. Deformable DETR: deformable transformers for end-to to to--t--dd object detection. CoRR, abs/20010044155 2020.
13


Reviews:
[{"review_id": "dX4o-6e1WlD", "reviewer": "Reviewer_PzxM", "summary": "This paper presents a unified approach for Semantic, instance, and panoptic segmentations tasks.  Proposed method segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class.\nThe model is training using bipartite matching, and the training and inference are naturally NMS-free and box-free. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The panoptic segmentation approaches usually use a separate method for prediction things and stuff. It’s nice that this paper proposes a unified approach for them.\n\nThe proposed method is evaluated on three tasks of coco panoptic segmentation, coco instance segmentation and ADE20k semantic segmentation.\nThey show that by adding k-net on top of different backbone and model architectures they can consistently improve the performance of the model.\nOn the coco panoptic segmentation task they could achieve a new state-of-the-art result.\nHowever, the best result on coco instance segmentation is much lower than state-of-the-art. Also the improvement on instance segmentation task seems to be less significant. I’m wondering why is that? Does k-net give more improvement on stuff compared to things?\nIt would be nice to provide per category improvements by adding k-net to one of the models for the panoptic segmentation task..\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion about broader impact look good to me.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any concern regarding ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Hz1Vf7ow_J", "reviewer": "Reviewer_ukKk", "summary": "This paper tries to unify semantic and instance segmentation with a group of learnable kernels that is conditioned on image features. Although the idea of using dynamic kernels (aka dynamic convolutions) has been already explored in instance segmentation [44, 47] and panoptic segmentation [30], this paper proposes to use sparse dynamic kernels instead of using kernels that are generated on dense grid in [30, 44, 37]. This paper further borrows the idea of bipartite matching loss in DETR [4] to make their model NMS free. The experiments show competitive results compared to state-of-the-art methods on multiple segmentation tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I have concerns to the writing:\n\n(1) In my optional, Table 4 (a) is a very important and also interesting experiments, however, there is not much description to this experiment. First, what is the baseline model without A.K.U. and K.I.? This is very important, because the improvements of these two components seem to be very significant. I want to make sure the author is not making comparison to some unreasonable baseline. Especially for without A.K.U., is kernel feature simply updated by $\\tilde K = F^K + K_{i-1}$? If this is the case, why gating makes so much improvement? Another question is, without A.K.U., why K.I. makes performance worse? This does not make much sense, adding self-attention is possible to not improve the performance, but I don't expect it to decrease the performance.\n\n(2) L295, positional encoding is mentioned here for the first time. It is unclear where positional encoding is added.\n\n(3) L42, the authors claim they make two \"novel\" designs here. I would suggest the authors to not make a such strong claim. Especially for the second design: bipartite matching loss is proposed by DETR [4] and the authors should not claim it as their novel design.\n\nSecond, I have some questions/concerns on Adaptive Kernel Update (A. K. U.): According to the ablation study in Table 4 (a), the proposed A.K.U. is the most important component in K-Net. However, the motivation of A.K.U. is not very clear in my opinion. In the draft, the authors claim since $M_i$ is not precise, $F^K$ may contain noises and the gating mechanism can reduce the adverse effect of the noise. My questions are:\n(1) Since there is no spatial resolution anymore, the only way to reduce the effect of noise is to reduce the amount of information from $F^K$ to $\\tilde K$. But why the multiplication between $F^K$ and $K_{i-1}$ can provide such information?\n\n(2) Why there is also a gate needed for $K_{i-1}$ as well?\n\n(3) The proposed gating seems very simple according to the text description, so I have tried such gating myself with a simplified K-Net architecture (without semantic features on instance segmentation). However, I found the gating mechanism does not work as well as that is reported in Table 4 (a). In fact, it always decreases performance. I'm wondering if the authors provide the exact description on how they implement the A.K.U. and is there any special design that is not mentioned in the paper?\n\nSome minor suggestions:\n\n(1) L182: $g_i$ is undefined?\n\n(2) L217: does it mean there is still conflict between instance and semantic segmentation so you need to paste instances first?\n\nIn general, this paper proposes an interesting architecture and performs enough ablation studies to validate the architecture. But the lack of motivation to the most important component (A.K.U.) and the difficulty to reproduce this important component lead to the weak accept score. If the authors can well address my concerns (especially on the A.K.U. part), I am happy to raise the score.\n\n\n\n\n----\n\\#### Post rebuttal comment ####\n\nThe authors have addressed my concern on the reproduction issue. The new ablation Table is much more reasonable than the original one in the submission. On the other hand, the new ablation Table shows one of the proposed components is less effective and may not even be necessary.\n\nAs I said in my original review, this paper proposes an interesting architecture and it is of value to the community. Since the authors address my concerns, I will raise my score to something between 6 and 7 (more towards 7).\n\nAs pointed out by multiple reviewers, the presentation of this paper is not satisfactory: it misses many important details. Thus, I would suggest the authors to check their writing more carefully, and provide all the necessary context and details.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "DTnmJAPmMSe", "reviewer": "Reviewer_c5jK", "summary": "The paper presents a convolutional neural network method for panoptic (multi-class, multi-instance) image segmentation. Semantic segmentation is addressed in the standard fashion with pre-trained model weights (convolutional kernels). Novelty is introduced in the instance segmentation component where the parameters of the convolutional kernels responsible for producing activation maps, one for each instance, are adapted at test-time to fit the given image. Overall this is a neat idea and the experimental results are compelling. However, I found the paper difficult to understand on first readings and have several suggestions for improvement of the presentation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found the idea of adaptive model components for instance segmentation appealing but the presentation of the paper lacked clarity. It was not immediately clear (although obvious after reading Section 3) that the kernels referred to in the paper are filter kernels for convolutional neural networks. The term \"kernel\" has multiple meanings in the machine learning literature and this should be clarified much earlier in the paper.\n\nMy biggest concern relates to details of the initialization and updating of the adaptive instance kernels, which is the core contribution of the paper. First, it is not clear how the number, N, of kernels is chosen and what happens (at test time) if this number is over- or under-estimated. Second, it seems at least when initialized that the instance kernels should be more sensitive to spatial location than appearance, and then adapt to appearance. Can the authors comment? Also, is there a need to treat \"thing\" and \"stuff\" instances differently?\n\nThird, it may be helpful to summarise the iterative updating of the kernels  in a formal algorithm. Eqn (2) gives the high-level overview although it is not clear why function $f$ is indexed by $i$. The spatial indices $(i,j)$ in Eqn (3) clash with the index $i$ for the iteration number and its not clear how the batches are indexed (presumably the operations are applied independently for each batch). Eqn (5) introduces $\\tilde{K}$ without providing an explicit relationship to $K_i$ (from the following paragraph appear to be a transformer followed by multi-layer perceptron?). All of this can be presented more rigorously.\n\nLast, it is not clear what effect the number of iterations applied to adapting the kernels has on the results. Table 3(c) suggests that accuracy saturates at four iterations for ADE20K and doesn't decrease thereafter but further data points should be given. Does the same finding hold for COCO? How many iterations (rounds) were used during training and does this affect the finding?\n\nSome other minor things to address:\n1. The image on page 1 is poorly placed and distracts the flow of the paper. I suggest moving to the top of the second page.\n2. The idea of joint semantic and instance segmentation appears in the literature well before the term panoptic segmentation was proposed (line 20). For example, Tu et al., IJCV 2005 propose the idea of image parsing, Gould et al., NeurIPS 2009 propose the idea of image decomposition, and Yao et al., CVPR 2012 develop the idea of holistic scene understanding.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I could not identify any societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0W9h_NTflCY", "reviewer": "Reviewer_LzXD", "summary": "This paper propose a unified framework named K-Net which segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. The authors propose a refined kernel update strategy to group categories efficiently and effectively. Extensive experiments on three different segmentation tasks prove that such simple design can achieve STOA results while running faster than many previous hand-crafted segmentation methods. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, from the motivation aspect, this is the most interesting paper in the topic of segmentation I have reviewed in this year. It brings a new way of modeling three segmentation tasks in one framework.  The new concept is to unify all the segmentation tasks which is also the goal of panoptic segmentation. This will be a fulture direction of segmentation task.\n\nAlthough MaxDeeplab[1] and DETR[2] solve the panoptic segmentation in a NMS free manner, the former requires specific backbone and attention layers and the latter needs much longer time to converge also along with the heavy mask head. \nK-Net can fairly compare with previous work using ResNet backbone (PanopticFCN, Panoptic FPN) and generalize well on large backbone (Swin-Transformer[3]). \n\n\nHere are the more details:\n\n1, Writing: The overall writing is good and easy to follow. \n\n2, Method: The proposed framework is simple yet effective which has novel components such as kernel update for interative refine. The kernel fusion is updated by the  learnable gate which is reasonable.  \nMoreover such design is also computationally efficient which can be a plug-in module for other tasks such as human couting and pose estimation.\n \n3, Experiment: The proposed K-Net surpasses all previous state-of-the-art single-model results of panoptic segmentation on MS COCO and semantic segmentation on ADE20K.  Most importantly, the comparison is fair and convincing. \nThe ablation studies are enough and good to prove the effectiveness of each part in method.  \n\n4, Potential: The framework is easy to extend into related tasks such as video segmentation tasks.\n\nThus I give my init rating as strong accept. \nHope the code can be available for the community soon. \n\n[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers CVPR-2021 \n\n[2] End-to-End Object Detection with Transformers ECCV-2020\n\n[3] swin transformer: hierarchical vision transformer using shifted windows.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No major weakness on both method and experiment parts.\n\nHere are several detailed suggestions and questions. \n\nSuggestions: \n\n1. Better cite and compare Max-Deeplab [1]. \n2. It seems that using kernel is a little confusing. Better to discuss the difference with object query in DETR [2] or Max-Deeplab [1].\n3. Better give the detailed network architectures. (You can put it into supplementary just like Max-Deeplab).\n4. What is g  in Line 182 (need the defination)?  I guess it is a fc layer. \n\n\n[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers CVPR-2021 \n[2] End-to-End Object Detection with Transformers ECCV-2020", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8.0"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission discusses how to unify methods for semantic, instance and panoptic segmentation. For this the authors propose to use a group of learnable kernels and a corresponding update strategy. All reviewers appreciated the contributions of this paper and recommend acceptance/leaning to accept after having read each others reviews and the rebuttal, which addressed some concerns about missing/incomplete ablations. However, all reviewers also pointed out that presentation and writing are not satisfactory and lack clarity. AC read the paper and concurs. E.g., "With the constructed M_0, K_0, and F, the kernel update head can start produce group-aware kernels K_s, s = {1, 2, ..., S} iteratively by S times and obtain the refined mask prediction M_S." (randomly picked sentence). Sentences like this are hard to parse and may prevent others from replicating the results. This being said, the authors promised to release code and models.
Decision: Accept (Poster)