Language-driven Semantic Segmentation
Authors: Boyi Li
Kilian Q Weinberger
Serge Belongie
Vladlen Koltun
Rene Ranftl
Venue: ICLR 2022 Poster (2022)
Keywords: language-driven
semantic segmentation
zero-shot
transformer

Abstract: We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., ``grass'' or ``building'') together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., ``cat'' and ``furry''). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.

Published as a conference paper at ICLR 2022
LANGUAGE-DRIVEN SEMANTIC SEGMENTATION
Boyi Li
Kilian Q. Weinberger Comell University
Serge Belongie
Cornell University, Cornell Tech
University of Copenhagen
Vladlen Koltun Apple
René Ranftl
Intel Labs
ABSTRACT
We present LSee, a novel model for language-driven semantic image segmenta- tion. LSeg uses text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") logether with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with contrastive objective to align pixel embeddings to the text embed- ding of the corresponding semantic class. The text embeddings provide flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows I. L.eeg to generalize previously unseen categories at test time, without retraining even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at http::/github.coo/sssoo/////aaaa
INTRODUCTION
Semantic segmentation is a core problem in computer vision, with the aim of partitioning an image into coherent regions with their respective semantic class labels. Most existing methods for semantic segmentation assume limited set of semantic class labels that can potentially be assigned to pixel. The number of class labels is dictated by the training dataset and typically ranges from tens (Everingham et al. 2015) to hundreds (Zhou et al. 2019 MMttaghi et al. (2014) of distinct categories. As the English language defines several hundred thousand nouns (Li et al. (020c),, is likely that the limited size of the label set severely hinders the potential recognition performance of existing semantic segmentation models.
The main reason for the restricted label sets in existing methods is the cost of annotating images to produce sufficient training data. To create training datasets, human annotators must associate every single pixel in thousands of images with semantic class label task that is extremely labor intensive and costly even with small label sets. The complexity of the annotation rises significantly as the number of labels increases since the human annotator has be aware of the ane-grained candidate labels. Additionally, inter-annotator consistency becomes an issue when objects are present in an image that could fit multiple different descriptions or are subject a hierarchy of labels.
Zero- and few-shot semantic segmentation methods have been proposed as a potential remedy for this problem. Few-shot approaches (Shaban al. [017]] Rakelly et al.) 2018 Siam et al. 2019 Wang
et al.
2019 Zhang et al. 2019: Nguyen& & Todorovic 2019: Liu et al. [222bb] Wang et al. 20200 Tian
et al. 2020 Boudiaf et al. 2021 Min et 2021) offer ways to learn to segment novel classes based
on only few labeled images. However, these approaches still require labeled data that includes the novel classes in order to facilitate transfer. Zero-shot methods, on the other hand, commonly leverage word embeddings discover generate related features between seen and unseen classes (Bucher et al.] [2119] Gu et al.] (202) without the need for additional annotations. Existing works in this space standard word embeddings (Mikolov et al., 2013) and focus on the image encoder.
In this work, we present a simple approach to leveraging modern language models to increase the flexibility and generality of semantic segmentation models. Our work is inspired by the CLIP model

---

Published as a conference paper at ICLR 2022
other deg
other, dag. tree
other, pet, vehicle, tree
dog
dag
pet
bicydl
bicydle
sky, grass
bicycle,
other
other
person
person
person
biryclie
sky
sky
mountain
grass
other
other, wall,
window, floor, celling
other. chair.
ather
other chair
other
thair
window
flcor
celling
Input Image
LSeg Output
LSeg Output
LSeg Output
Figure Example results. LSeg is able to handle unseen labels as well as label sets of arbitrary length and order. This enables flexible synthesis of zero-shot semantic segmentation models the fly. From left to right, labels that are removed between runs are underlined, whereas labels that are added are marked in bold red.
for image classification (Radford al. 2021), which pairs high-capacity image and encoders to produce robust zero-shot classifiers. We propose to use state-of-the-arr text encoders that have been co-trained on visual data, such as CLIP, to embed labels from the training set into an embedding space and to train a visual encoder to produce per-pixel embeddings from an input image that are close the corresponding label embeddings. Since text encoder trained to embed closely related concepts near one another (for example, "dog" is closer to "pet" than to "vehicle"), we can transfer the flexibility of the text encoder to the visual recognition module while only training on the restricted label sets that are provided by existing semantic segmentation datasets. An example is shown in Figure (top row), where the model can successfully label pixels belonging to the class "pet" although the training set did not contain this label.
Our approach enables the synthesis of zero-shot semantic segmentation models on the fly. That is, user can arbitrarily expand, shrink, reorder the label set for any image test time. We further introduce an output module that can spatially regularize the predictions while maintaining this flexibility. We demonstrate several examples of the flexibility of our model in Figure [ LSeg is able to output different segmentation maps based on the provided label set. For instance, in the last row, output (a) recognizes the chair and identifies all non-chair objects "other" since these are the only two labels provided to the model. When labels are added, as in (b) and (c), the model is able to successfully segment other objects with the expanded label set.
We conduct quantitative evaluation on variety of zero- and few-shot semantic segmentation tasks. Our approach outperforms existing methods in zero-shot settings and is competitive across multiple few-shot benchmarks. Unlike the state-of-the of-the-rrt baselines we compare to, our approach does not require additional training samples. Our experiments also show that introducing the text embeddings incurs only negligible loss in performance when compared to standard fixed-label segmentation methods.
RELATED WORK
Generalized semantic segmentation. The majority existing semantic segmentation models are restricted to fixed label set that is defined by the labels that are present in the training dataset (Minace) jet al.) 202T). Few-shot semantic segmentation methods aim to relax the restriction fixed label set when or few annotated examples of novel classes available test time. These approaches
2

---

Published as a conference paper at ICLR 2022
learn to find reliable visual correspondences between query image that is to be labeled and labeled
support images that may contain novel semantic classes (Shaban et al. 2017 Rakelly et al., 22018)
Siam et al.
2019 Wang al.] 2019; Zhang et al. 2019 Nguyen & Todorovic 2019
1 Liu al. 2020b
Wang et al.
2020 TTan et al. 2020 Wang et al. 2020 Tian et al. 2020; Boudiaf et al. [021] Min
al., 2021 While this strategy
significantly enhance the generality of the resulting model,
requires the availability of at least one labeled example image with the target label set, something
that is not always practical.
Zero-shot semantic segmentation approaches aim to segment unseen objects without any additional samples of novel classes. Text embeddings of class labels play a central role in these works. Bucher it al. (2019) and Gu et al. (2020) propose to leverage word embeddings together with generative model to generate visual features of unseen categories, while Xian et al. (2019) propose to project visual features into simple word embedding space and to correlate the resulting embeddings to
assign a label to pixel. [Hu et al. (2020) propose to use incertainty-awan learning to better handle noisy labels of seen classes, while Li et al. (2020b) introduce a structured learning approach to better exploit the relations between seen and unseen categories. While all of these leverage text embeddings, our paper is, the best of our knowledge, the first show that it is possible synthesize zero-shot semantic segmentation models that perform on par with fixed-labe and few-shot semantic segmentation methods.
variety of solutions have been proposed (Zhang et al. 2020h Liu et al. 2020a Perera et al. 2020
Zhou al., (02T) for open-set recognition (Scheirer et al.] 2012 Geng et al.] (020). These aim to provide a binary decision about whether or not a given sample falls outside the training distribution, but do aim to predict the labels of entirely new classes.
Finally, different line of work explores cross-domain adaptation methods for semantic segmentation by using feature alignment, self-training. and information propagation strategies (Yang al., 2021 Wang et al. (021). The target of these works is to enhance the transferability of models to novel visual domains, but they do not address the issue of a restricted label set. As such they are orthogonal to our work.
Language-driven recognition. Language-drivee driven recognition an active area of research. Common tasks in this space include visual question answering (Antol et al. (2013), image captioning (Vinyals) et al. 2014), and image-text retrieval (Li et al. 2020a). CLIP (Radford et al.) 202T) demonstrated that classic recognition tasks that are not commonly ly associated with language can strongly benefit from language assistance. CLIP uses contrastive learning together with high-capacity language models and visual feature encoders to synthesize extremely robust models for zero-shot image classification. Recent works have extended this basic paradigm to perform flexible object detection. ViLD (Gu et al. 202T) introduces an advanced zero-shot object detection method that leverages CLIP, whereas MDETR (Kamath et al.) 2021) proposes an end-to-end approach that modulates a transformer-basee baseline detector with text features that are obtained from state-of-the-art language model. Like CLIP, these works have shown that the robustness and generality of object detection models can be strongly improved by language assistance. Our work is inspired by these approaches and presents, to the best of our knowledge, the first approach to flexibly synthesize zero-shot semantic segmentation models by leveraging high-capacity language models.
LANGUAGE-DRIVEN SEMANTIC SEGMENTATION
Our approach, Language driven Semantic segmentation (LSSe) embeds text labels and image pixels into a common space, and assigns the closest label to each pixel. We illustrate the framework in Figure 2 and describe each part in detail below.
Text encoder. The text encoder embeds the set of N potential labels into continuous vector space RCC producing N vectors Ti...., Tn CRC as outputs (blue vectors in Figure 2}. Multiple network architectures are possible, and we use the pretrained Contrastive throughout (Radford et al. 202I). By design, the set of output vectors is invariant to the ordering of the input labels and allows their number, N, to vary freely.
Language-Immage Pre-training (CLIP)
Image encoder. Similar to the text encoder, the image encoder produces an embedding vector for every input pixel (after downsampling). We leverage dense prediction transformers (DPT) (Ranfil
3

---

Published as a conference paper at ICLR 2022
people, tennis racket, other Input Label Set
Text Encouder
F1o
Image Errodder coder
40
P5
130
other
Inpul Image
Outgut
Figure 2: Overview. A text encoder embeds labels into a vector space. An image encoder extracts per-pixel embeddings from the image and correlates the feature of each pixel to label embeddings. The image encoder is trained to maximize the correlation between the text embedding and the image pixel embedding of the ground-truth class of the pixel. A final spatial regularization block spatially regularizes and cleans up the predictions.
ct al., 2021) as the underlying architecture. Assume x W is the input image size and s is user-defined downsampling factor (s = 2 in our implementation). We define // w = _ The output is dense embedding 1 E TR. Hxx (green tensor in Figure 2 We refer the embedding of pixel ji as lij.
Word-pixel correlation tensor. After the image and the labels are embedded, we correlate them by the inner product, creating a tensor of size H W x N (orange tensor in Figure 2). defined as
fajk Lú Te.
We refer to the N N-diimnnsinnal vector of inner products between the embedding of pixel (i,j) and all N words as F.ù R.V, where Fij (fir1, .... fiyk )" During training, encourage the image encoder to provide pixel embeddings that are close to the text embedding of the corresponding ground- truth class. Specifically, given the text embeddings Tr Rº of N labels and the image embedding ly E I of pixel i, we aim to maximize the dot product of the entry Sijk that corresponds to the ground-truth label k = Yij of pixel i.j. We achieve this by defining pixelwise softmax objective over the whole image:
H.W
softmax,1. 91
where is a user-defined temperature parameter that we set to = 0.07 (Wu al. 2018 Radford et al., 2021). During training, minimize per-pixel softmax with cross-entropy loss (with temperature scaling) as is standard in semantic segmentation
Spatial regularization. Due to memory constraints, the image encoder predicts pixel embeddings lower resolution than the input image resolution. We use an additional post-processing module that spatially regularizes and upsamples the predictions to the original input resolution. During this process, we have to ensure that all operations stay equivariant with respect to the labels. In other words, there should be no interactions between the input channels, whose order is defined by the order of the words and can thus be arbitrary. We evaluate two functions that fulfill this property: simple cascade of depthwise convolutions (Chollet 2017) followed by non- con-linear activations (DepthwiseBlock), and another block that additionally augments the depthwise convolutions with the result of max-pooling operation over the set of labels (BottleneckBlock) (Li al.) (019). In final step we use bilinear interpolation to recover predictions at the original resolution. We refer to these functions "spatial regularization blocks" and illustrate them in Figure
In practice implement this using the standard nn. CrossEEnroopl..oss from Pytorch.
4

---

Published as a conference paper at ICLR 2022
Training details. We initialize the backbone of the im- age encoder with the official ImageNet pretrained weights from ViT (Dosovitskiy et al. 202I) or ResNet (He et al. 2016 and initialize the decoder of DPT randomly. Dur-
Depthwise Corv
Depthwise Corv
ing training freeze the text encoder and only update the
weights of the image encoder. We provide the full label the text encoder
Activatton
that defined by each training for each image.
Activation
DeptthwizeBlock
BottleneckBlock
Our model can be trained on any semantic segmentation dataset and supports flexible mixing of multiple datasets through the text encoder. Existing semantic segmentation models assign fixed channel in the output to represent the probability of pixel being the corresponding semantic class. In contrast, our approach can dy- namically handle label sets with varying length, content, and order. This property allows synthesizing arbitrary zero-shot semantic segmentation models by simply changing the labels that are fed to the text encoder.
Figure 3: Illustration of BottleneckBlock and DepthwiseBlock.
4 EXPERIMENTS
We designed LSeg primarily for the zero-shot setting, where labels that are used for inference have never been seen during training. However, due lack of standardized protocol and sufficient datasets and baselines for the zero-shot setting, we compare LSeg to zero- and few-shot semantic segmentation models on few-shot benchmarks. Note that few-shot methods have access to more information and are thus expected to yield higher accuracy. However, the need for labeled samples severely restricts their flcxibility compared to our approach.
4.1 EXPERIMENTAL SETUP
We follow the protocol of the recent tfateoof-the-art few-shot method HSNet (Min al. 2021) and evaluate on three widely-used few-shot semantic segmentation benchmarks: PASCAL (Ever ingham al. 2015), COCO-20' (Lin et 2014). and FSS-1000 (Li al. 2020c). Following standard protocol for few-shot segmentation, we use the mean intersection over union (mloU) and foreground-backgrouud intersection of union (FB-IoU) as the evaluation metrics. The mloU caleu- lates the average loU over all classes, FB-IoU computes mean value of foreground and background IoUs fold and ignores object classes.
When not stated otherwise we use an LSSeg model with the text encoder provided by CLIP-VIT-B/32 and leverage DPT with ViT-I/16 backbone as the image encoder. For datasets that provide background or unknown class, set the corresponding background label to "other". We use SGD with momentum 0.9 and a polynomial learning rate scheduler with decay rate 0.9. We train with a batch size of 6 six Quadro RTX 6000.
4.2 PASCAL-5' AND COCO-20*
PASCAL-5' and COCO-20* are few-shot segmentation datasets that have been created from PASCAL VOC 2012 [Everingham et al. 2015) and the COCO dataset (Lin et al. (204)), respectively. PASCAL- 5 is composed of 20 object classes with corresponding mask annotations and has been evenly divided into 4 folds of 5 classes each. We denote different folds by 5², where {0, 1, 3). Similarly, COCO-20* is composed of 4 folds of 20 classes each.
We compare LSeg various state-of-the-as art few-shot models: OSLSM (Shaban et al. 2017), Co- FCN (Rakelly et al. 2018). AMP-2 (Siam et al. 2019). PANet (Wang et al., 2019). PGNet (Zhang al. 2019), FWB Nguyen Todorovic 2019). PPNet (Liu et al. 20206), DAN (Wang al. 2020). PFENet (Tian et al.) 2020), RePRT (Boudiaf et al. 2021), and HSNet (Min et al.) 2021).
These few-shot methods propose strategies to segment unseen objects based on pretraining on seen categories and finetuning with few images from the target class. In addition, we also compare
We also evaluated on a model initialized with the CLIP image encoder with the same setup and hyperparam- eters, but observed worse performance than using the VIT initialization.
5

---

Published as a conference paper at ICLR 2022
Model
Backbone
Method
50
5
5²
5
mean
FB-IoU
OSLSM
1-shot
33.6
55.2
40.9
33.5
40.8
61.3
co-FCN
VGG16
1-shot
36.7
50.6
44.9
32.4
41.1
60.1
AMP-2
1-shot
41.9
50.2
46.7
34.7
43.4
61.9
PANet PGNet
1-shot
44.0
57.5
50.8
44.0
49.1
ResNet50
I-shot
56.0
66.9
50.6
50.4
56.0
69.9
FWB
I-shot
51.3
64.5
56.7
52.2
56.2
PPNet
1-shot
52.7
62.8
57.4
47.7
55.2
70.9
DAN
1-shot
54.7
68.6
57.8
51.6
58.2
71.9
01
PFENet
I-shot
60.5
69.4
54.4
55.9
60.1
72.9
RePRI
I-shot
59.6
68.6
62.2
47.2
59.4
HSNet
--shot
67.3
72.3
62.0
63.1
66.2
77.6
SPNet
zero-shot
23.8
17.0
14.1
18.3
18.3
44.3
0e
ZS3Net
zero-shot
40.8
39.4
39.3
33.6
38.3
57.7
LSeg LSeg
ResNet101 ViT-L/16
zero-shot
52.8
53.8
44.4
38.5
47.4
64.1
zero-shot
61.3
63.6
43.1
41.0
52.3
67.0
Table 1: Comparison of mloU and FB-IoU (higher is better) on PASCAL-5'.
Model
Backbone
Method
20
20
202
203
mean
FB-IoU
PPNet PMM
1 1-shot
28.1
30.8
29.5
27.7
29.0
1 1-shot
29.3
34.8
27.1
27.3
29.6
ResNet50
RPMM
I-shot
29.5
36.8
28.9
27.0
30.6
RePRI
1-shot
32.0
38.7
32.7
33.1
34.1
FWB DAN
1-shot
17.0
18.0
21.0
28.9
21.2
I I-shot
24.4
62.3
ResNet101
PFENet
1-shot
36.8
41.8
38.7
36.7
38.5
63.0
HSNet
1-shot
37.2
44.1
42.4
41.3
41.2
69.1
ZS3Net
ResNet101
zero-shot
18.8
20.1
24.8
20.5
21.1
55.1
LSeg
ResNet101
zero-shot
22.1
25.1
24.9
21.5
23.4
57.9
LSeg
ViT-L/16
zero-shot
28.1
27.5
30.0
23.2
27.2
59.9
Table 2: Comparison of mloU and FB-IoU (higher is better) on COCO-20
to the competitive zero-shot baseline ZS3Net (Bucher et al. 2019). which adopts the DeepLabv3+ framework and to Xian al. (2019) which leverages DeepLabv2. We follow their official code, training setting and training steps on the basis of their provided model pretrained on ImageNet (Deng al. 2009). We follow the common experimental setup (Nguyen Todorovic/ 2019) and conduct cross-validation over all folds. Assuming that is the number of classes in fold for each fold i, we use images of other folds for training and randomly sampled 1000 images of target fold for evaluation. We show PASCAL-5' and COCO-20" results in Tables and Our model (with the same ResNet101 backbone) outperforms the zero-shot baseline by considerable margin across folds and datasets and is even competitive with several few-shot methods. We also observe obvious edge of LSeg by using a larger backbone (ViT-L/16).
4.3 FSS-1000
PSS-1000 (Li et al., 2020ch is recent benchmark dataset for few-shot segmentation. It consists of 1000 object classes with pixelwise annotated segmentation masks. It contains significant number of unseen or unannotated objects comparison to previous datasets such as PASCAL and COCO. Following the standard protocol, split the 1000 classes into training, validation, and test classes, with 520, 240, and 240 classes, respectively. We use a base learning rate of 0.05 and train the model for 60 epochs.

---

Published as a conference paper at ICLR 2022
Table compares approach state-of-the art few-shot models. Notably, under the same ResNet101, LSeg could achieve comparative results of the state-of-the-art one-shot method. Also, I.Seg outperforms state-of-the-arr one-shot method: 87.8 mloU (ours) vs. 86.5 mloU (HSNet) with a larger backbone ViT-L/16, indicating that LSeg generalizes very well to unseen categories. Figure 4 shows examples of segmentation results on unseen categories.
Model
Backhone
Method
mloU
OSLSM
1-shet
70.3
GNet
1-shot
71.9
VGG16
FSS
1-shol
73.5
DoG-LSTM
1-shot
80.8
DAN HSNet
1-shol
85.2
ResNet101
1-shot
86.5
LSeg LSeg
ResNert01 ViT-L/16
zemo-shot
84.7
zeru-shot
87.8
Table 3: Comparison of mloU on FSS-1000.
ottwns
minicooper
cchoo
wooden aprin
maggie Jrrt
othera
others
others her
others
Image
L8ag Ouput
Input Image
LSig Ouput
Input Image
LSag Ouput
Figure 4: LSeg zero-shot semantic segmentation results unseen categories of FSS-1000 dataset.
EXPLORATION AND DISCUSSION
5.1 ABLATION STUDIES
We further empirically explore various properties of LSeg. We conduct experiments the ADE20K dataset (Zhou et al. 2019). which a standard semantic segmentation dataset that includes a diversity of images and provides pixelwise segmentation of 150 different categories. We set the base learning rate to 0.004 and train the model for 240 iterations. We use SGD with momentum 0,9 and polynomial learning rate scheduler with decay rate 0.9. We use LSeg with DPT and smaller ViT-B/32 backbone together with the CLIP ViT-B/32 text encoder unless stated otherwise.
Spatial regularization blocks. We first conduct an ablation study on the two variants of the spatial regularization blocks for cleaning up the output. We ablate the different types of blocks as well as stacking various numbers of blocks (N [0, 1, 2, 4]). The results are shown Table We notice that consistent improvement can be achieved by adding few regularization blocks. The strongest improvement is achieved by stacking two BottleneckBlocks, addition to the architecture that incurs little overhead.
Text encoders. LSeg supports arbitrary text encoders in principle. We show the influence of using different text encoders in Table 5 where we ablate various encoders that are provided by the CLIP
# depth
Block Type
Metric
0
2
4
Aix [%]
79.70
79.72
79.78
7.67
DepthwiseBlock
mloU [%]
37.83
39.19
39.45
0.18
pixAce [%]
79.70
79.64
79.70
79.68
BottleneckBlock
mloU [%]
37.83
39.16
39.79
38.78
Table Ablation study on the depth of BottleneckBlock and DepthwiseBlock before the last layer. For both Pixel Accuracy (pixAcc) and mloU, higher is better. For depth=1, we directly feed the output to reshape without activation.
7

---

Published as a conference paper at ICLR 2022
zero-shot image classification model (Radford et al. 2021). Note that all text encoders feature the same transformer-bus based architecture that purely operates on text. The main difference between the encoders is the image encoder that paired during CLIP pretraining (for example, the text encoder denoted by "ViT-B/32" was trained in conjunction with a ViT-B/32 image encoder) and the size of the embedding dimension.
We observe that using RN50 x 16 achieves the best performance among all text encoders and surpasses the weakest ViT-B/32 text encoder by 2.5%. We conjecture that this is because of the larger size of the embedding that is provided by this encoder.
Comparison fixed label set. Language assistance helps boost the recognition performance on unannotated or unseen classes. However, there might be concern that this flexibility hurts the performance tasks that have fixed label set. To test this, we train LSeg on ADE20K using the standard protocol on this dataset, where the training and test labels are fixed (that is, there are no unseen class labels test time). We compare the results to highly competitive standard semantic segmentation models, including OCNet (Yuan et al. 2020), ACNet (Fu al. 2019), Deeplab (Chen) al., 2017) 7 Zhang et al. 2020a), and DPT (Ranftl et al. 2021). The results are listed in Table We find that LSeg performs competitively when using the RN50 x 16 text encoder and incurs only negligible loss in performance when compared to the closest fixed-label segmentation method (DPT).
5.2
QUALITATIVE FINDINGS
We finally train LSeg on a mix different datasets (Lambert et al. 2020). including ADE20K (Zhou
et al. 2019). BDD Yu et al. 2020), Cityscapes (Cordts et al. 2016). COCO-Panoptic (Lin et al. Caesar et al. 2018), IDD (Varma et al. 2019). Mapillary Vistas (Neuhold et al. 2017). and SUN RGBD (Song al. 2015). Note that train model the original label sets that
2014
provided by these datasets without any preprocessing or relabeling. We follow the same training protocol on ADE20K and train LSeg with a ViT-L/16 backbone and ViT-B/32
encoder for 200 epochs with a base learning rate of 0.004, If there are multiple labels for one class, we only use the first label that provided during training. We select images from the web and show the results in Figure 3 illustrate the use of the resulting model.
Related but previously unseen labels. We illustrate some salient examples of the capabilities of LSeg to generalize to new classes in Figure In the first row, on the left we first start with the label set "sky", "road", "house". and "plant", and observe that the model capable of segmenting the image into the provided classes. We then change the label "house" to "building" and the label "plant" to "grreeney"". The model produces a similar segmentation as before on this different but semantically related label set. This is despite the fact that the label "greenery" or even "green" was not present in any of the training images. A similar effect shown in the second row, where LSeg successfully
Method
Backhone
Text Encoder (fixed)
embedding dimension
pixAcc [%]
mIoU [%]
LSeg
ViT-B/32
ViT-B/32
512
79.70
37.83
LSeg
VIT-B/32
VIT-B/16
512
79.77
38 69.69
LSeg
VIT-B/32
RN50 4
640
79.85
38.93
TSeg
ViT-B/32
RN50 16
768
80.26
40.36
Table 5: Ablation study on LSeg with fixed text encoders of different CLIP pretrained models.
Method
Backbone
Text Encoder
pixAce [%]
mloU [%]
OCNet
ResNet101
45.45
ACNet
ResNet101
81.96
45,90
DeeplabV3
ResNeSt101
82.07
46.91
DPT
ViT-L/16
82.70
47.63
LSeg
VIT-L/16
ViT-B/32
82.46
46.28
LSeg
VIT-L/16
RN50 x 16
82.78
47.25
Table 6: Comparison of semantic segmentation results on the ADE20K validation set. For LSeg, we conduct experiments with fixed text encoders of ViT-B/32 and RN50 x CLIP pretrained models.
8

---

Published as a conference paper at ICLR 2022
Eal SOEL plat.stora
turry plant, store
sky ood
roed building
part
litle.
labls
peiring plart, ofhers
famiuum peiring
sits
trane
Nox chters
partio
part
others
A
Fegg Ouput
Image
Cuput
Beg
(s) Related unseen labels.
(b) Hierarchical unseen labels.
Figure 5: LSeg examples with related but previously unseen labels, and hierarchical labels. Going from left right, labels that are removed between are underlined, whereas labels that are added are marked in bold red.
segments the image and correctly assigns the labels "cake" or "dessert" (again, the label "dessert" was not seen during training), while successfully suppressing the label "bread" which is both visually and semantically related.
Hierarchical unseen labels. Figure 5 b)b) demonstrates that LSeg can implicitly provide correct segmentation maps for hierarchies of labels. In the first row, the model is able to recognize the "cat", "plant" and "grass" segments of the image, as expected since these labels are present the training set. When replacing "cat" with the label "furry", we notice that the model is able to successfully recognize this parent category (that is, most cats are furry, but not all furry objects are cats). Similarly, when removing the label "grass", we notice that the original "grass" region is merged into "plant", again an indication of an implicit hierarchy that is afforded by the ficxibility of the text embeddings. The second row illustrates similar scenario, where LSeg recognizes the sofa and other objects. However, the small shelf on the left is segmented as the unknown category "other". When we change "sofa" to "furniture", LSeg successfully identifies both the sofa and the small shelf "furniture". Note that "furniture" never appeared in the training label set.
Failure cases. While LSeg in general achieves very promising results, we also observe some failure cases, illustrated in Figure The left image illustrates that LSeg is only trained with positive samples from class. When the test- time input labels do not contain any of the true labels for the corresponding pixel, the model assigns the highest probability to the closest la- bel the text embedding space. In this specific example, the model assigns the label "toy" since
toy, grass
house,
free, grass, aky.
toy grass
grass
sky
house
other
(a) Figure 6: Failure cases.
(b)
the visual features of the dog are apparently closer to "toy" than to "grass" in the embedding space and there is other label that can explain the visual features. A second failure case is shown the right, where the model focuses on a single most likely object when multiple explanations are consistent with the label set. In this specific example, the windows of the house are labeled "house" instead of window, even thought the label "window" is available as a choice. We hope that these failure cases can inform future work, which could involve augmenting training with negative samples or building fine-grained language-driven semantic segmentation models that can potentially assign multiple labels when multiple explanations fit the data well.
CONCLUSION
We introduced LSeg. novel method and architecture for training Lunguage-drienn semantic segmen- tation models. LSeg enables a flexible label representation that maps semantically similar labels to similar regions in an embedding space and learns correlate visual concepts in this space to produce semantic segmentations. Our formulation enables the synthesis of zero-shot semantic segmentation models with arbitrary label sets on the fly. Our empirical results show that the resulting models are strong baselines for zero-shot semantic segmentation and can even rival few-shot segmentation models while not sacrificing accuracy existing fixed label sets.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported in part by the Pioneer Centre for AI, DNRF grant number PI. KQW is supported by grants from the National Science Foundation NSF (IS--210716,, III-1526012, IIS- 1149882, and IIS-1724282), the Cornell Center for Materials Research with funding from the NSF MRSEC program (MMR--171875), and SAP America.
ETHICS STATEMENT
We proposed novel approach solve the generalized semantic segmentation problem. We use public computer vision datasets and leverage pretrained language models for our experiments. We do not believe that our code or method are inherently subject to concerns of discrimination bias / fairness, inappropriate potential applications, impact, privacy and security issues, legal compliance, research integrity or research practice issues. However, image datasets and language models may be subject to bias that may be inherited by models trained with our approach.
REPRODUCIBILITY
Our code is reproducible and can be implemented based on the method description Section Bs well as training details in Section 4 and We provide an interactive demo for people to try with input images of their choosing.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision, 2015.
Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed. Jose Dolz. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13979-13998 13988, 2021.
Maxime Bucher, Tuan Hung Vu, Matthicu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32:468 479, 2019.
Tuan-Hung
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 05587, 2017.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE
conference on computer vision and pattern recognition, 1251-1258, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe
Franke, Stefan Roth. and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 225.. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, image worth 16x words: Transformers for image recognition at seale. In International Conference on Learning Representations, 2021.
M. Everingham, S. M. A. Eslami, L. Van Gool, K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-136 January 2015.
Jun Fu, Jing Liu, Yuhang Wang. Yong Li, Yongjun Bao, Jinhui Tang. and Hanqing Lu. Adaptive context network for scene parsing. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 6748-6757, 2019.
10

---

Published as a conference paper at ICLR 2022
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
Xiuye Gu, Tsung- Yi Lin. Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. ACM International Conference on Multimedia, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ping Hu, Stan Sclaroff, and Kate Saenko. Uncertainty aware learning for zero-shot semantic segmentation. In Advonces Neural Information Processing Systems, 2020.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabricl Synnaeve, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2879-2888, 2020.
Boyi Li, Felix Wu, Kilian Q Weinherger, and Serge Belongie, Positional normalization, Advances Neural Information Processing Systems, pp. 1620-1632, 2019.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder vision and language by cross-modal pre-training, In AAAI, 2020a.
Peike Li. Yunchao Wei, and Yang. Consistent structural relation learning zero-shot segmentation. Advances in Neural Information Processing Systems, 33, 2020b.
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paternn Recognition, pp. 2869-2878, 2020c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740 755. Springer, 2014.
Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos. Few-shot open-set recognition using meta-Icarning In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8798-8807, 2020a.
Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference Computer Vision, pp. 142-158. Springer, 2020b.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, JCLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. U.RLthttt//axxii.. org wttp://ww...rraab/133........ 13011.b.
Juhong Min. Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. Priceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
Roozbeh Mottaghi, Xianjic Chen, Xiaobai Liu, Nam- Gyu Cho, Scong- Whan Lec, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pottern Recognition, 2014.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschicder. The mapillary vistas datasct for semantic understanding of street scenes. In Proceedings of the IEEE international conference computer vision, pp. 4990 4999, 2017.
Khoi Nguyen and Sinisa Todorovie. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEFF/CVP International Conference Computer Vision, pp. 622-631, 2019.
11

---

Published as a conference paper at ICLR 2022
Pramuditha Perera, Vlad I Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington. Vicente Ordonez, and Vishal M Patel. inperative-diserimnnatte feature representations for open-set recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11814-11823, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 7748-8763. 2021.
Kate Rakelly, Evan Shelhamer, Trevor Darrlll, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. In International Conference on Learning Representations Workshop, 2018.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In The IEEE International Conference on Computer Vision, 2021.
Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):1757-1772, 2012.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference, 2017.
Mennatullah Siam, Boris Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5249-525,,, 2019.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 567-576. 2015.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang. Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1-1, 2020.
Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawahar. Idd: A dataset for exploring problems of autonomous navigation in unconstrained env environments. In 2019 IEEE Winter Conference on Applications of Computer Vision, pp. 1743-1751. IEEE, 2019.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. CoRR, abs/1411.4555, 2014. URL [aLlatt::// http://axxvvvv.. wtg//bv.orrg/bb... 1411.411 4555
Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang. Xianbin Cao, and Xiantong Zhen. Few-shot se- 2020: 16th European
mantic segmentation with democratic attention networks. In Computer Conference, Glasgow, UK, August 23-288, 2020, Proceedings, Part XIII 16, pp. 730-746. Springer, 2020.
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9197-9206, 2019.
Yuxi Wang, Jian Liang, and Zhaoxiang Zhang. Give me your trained model: Domain adaptive semantic segmentation without source data, arXiv preprint arXiv:2106. 11653, 2021.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- non-parametrie instance-level discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 8256-8265, 2019.
Jinyu Yang, Chunyuan Li, Weizhi An, Hehuan Ma. Yuzhi Guo, Rong. Peilin Zhao, and Junzhou Huang. Exploring robustness of unsupervised domain adaptation in semantic segmentation. In Proceedings of the IECE/CVF International Conference Computer Vision, 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp.
2020.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In Computer VCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 173-190. Springer, 2020.
12

---

Published as a conference paper at ICLR 2022
Chi Zhang. Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based based one-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9587 9595, 2019.
Hang Zhang, Chongruo Wu. Zhongyue Zhang, Zhu, Zhi Zhang. Haibin Lin, Yue Sun, Tong He. Jonas Muller, Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020a.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pp. 117. Springer, 2020b.
Bolci Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3): 302-321 321, 2019.
Da- Wei Zhou, Han-Jia Ye, and De- Chuan Zhan. Learning placeholders for open-set recognition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2021.
13


Reviews:
[{"review_id": "tfd-F2TiAOG", "reviewer": "Reviewer_Cogy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- This paper proposes to use language knowledge (especially semantic similarities, I guess) to label the pixels of new classes for zero-shot semantic segmentation. The idea of distilling knowledge from pre-trained models is reasonable and becomes a new trend for many zero-shot settings.\n- The experiment results seem good, which outperforms the current SOTA on a zero-shot setting and is on par with some few-shot results. \n\nWeaknesses:\n\n- The method is not that novel. It just computes the semantic similarity between the language embedding from CLIP and visual embedding from DPT, and adopts cross-entropy as supervision. Such a simple design cannot address some critical issues in zero-shot semantic segmentation, as mentioned in Failure cases in Sec. 5.1. The authors notice the issues but leave them to the readers.\n\n- The result comparison in the experiment is not fair. This work uses a stronger backbone (ViT) in the image encoder than the previous works (RN-101). Is there any way to get a fair comparison?", "summary_of_the_paper": "This paper uses the pre-trained large model (CLIP) to transfer the language knowledge to the unseen labels for zero-shot semantic segmentation. The idea is reasonable and coherent with previous works that distill knowledge from pre-trained models. The experiments also show good results on several benchmarks in a zero-shot setting. However, the method is not novel and leads to many problems.", "summary_of_the_review": "I like the idea of utilizing pre-trained models to transfer the knowledge for zero-shot semantic segmentation, and I think it would be the main trend for many other zero-shot settings. However, this paper only adopts a naive method to solve the problem, leaving some critical issues to the readers. So my initial rating is borderline.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "epv3FGgBY_r", "reviewer": "Reviewer_3LsU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The biggest strength of the proposed approach is its simplicity and its flexibility since it can dynamically handle arbitrary label sets on the fly with varying length, content, and order. \n\nWeaknesses:\n1.\tWhile the performance on FSD-1000 and ADE-20k are impressive, it is compared against just 1-shot. Also, the gap is larger in Pascal-5i and COCO-20i with less mIoU and FB-IoU for LSeg.\n2.\tThe strongest baseline uses ResNet-101 which has much fewer parameters (~45M) than the backbone used by LSeg (307M). So, it does not seem like a fair comparison.\n\n", "summary_of_the_paper": "The paper proposes Language driven Semantic segmentation (LSeg) for semantic segmentation. Essentially, LSeg embeds text labels and image pixels into a common space, and assigns the closest label to each pixel. LSeg is flexible and can dynamically handle arbitrary label sets on the fly with varying length, content, and order. The paper demonstrates that LSeg achieves comparable performance as state of the art few-shot semantic segmentation networks on FSD-1000 even though it is used in zero-shot setting. When a fixed label set is used based on the data set (all labels in the training set), it also matches the accuracy of traditional segmentation algorithms on ADE-20k. LSeg uses text encoder from CLIP-ViT-B/32 which is frozen during training while the weights of the image decoder (DPT with a ViT-L/16) is updated to maximize the correlation between the text embedding and the image pixel embedding of the ground truth class of the pixel. Spatial regularization is applied at the end that also up samples the predictions to the original input resolution. ", "summary_of_the_review": "Experiments section is weak. Most of the tables contain much weaker and older models for comparison (even VGG16). The biggest concern is that ViT-L has 307M parameters while the largest backbone used in comparisons is ResNet-101 which has about 45M parameters. ViT-B has 86M which is still larger than ResNet-101 but will at least be a fairer comparison that using ViT-L for image decoder.  \n\nHowever, the flexibility of the approach is powerful. Lseg can dynamically handle arbitrary label sets on the fly with varying length, content, and order.\n\nHence the recommended rating is “marginally above” the acceptance threshold.\n\n=====\n\nThe reviewer thanks the authors for their response. Glad to see that LSeg still shows improvements when using ResNet101 as backbone. Updated the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_HfsjUADQq8", "reviewer": "Reviewer_u2M8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This method is similar to image classification of CLIP on ImageNet-1k, where the template weights of each class derived from language text, and also called zero-shot learning. As a comparison, this paper v.s. CLIP can be treated as FCN v.s. VGG. I have some questions about this paper\n1. In Figure 2, F_{H1} the orange cube is a typo? Should be F_{1W}? \n2. In Figure 2, the size is also not correct. The input image H*W*C, the feature should not be that size as stated in the paragraph of bottom of Page 3. \n3. In Figure 2, what does C stands for? Input image have C channel, I think C=3 for RGB image. Then Why Text embedding also C channel?\n4. In Figure 2, what is the real method to compute training loss? In Equation (2) it computes a global average of softmax_{y_{ij}}(F_{ij})? What does this expression mean? Besides I think an explicit notation of F_{ij} = {f_{ij1}, …f_{ijK}} is necessary. \n5. In Table 1, Table 2, Table 3, LSeg with ResNet101 is missing. In Table 6 it said Text Encoder is RN50x16. I think this kind of statement is very strange and an explanation in caption is absolutely necessary.\n", "summary_of_the_paper": "This paper is about semantic image segmentation, the template weights of each category is generated by language text.", "summary_of_the_review": "The clarity of presentation needs improvement. More ablation study necessary.\n\n\n==========================================\n\nI have read the revised version and still think it is not the final version. For example\n\n1 In Equation(2), what does y_ij means? An image has several labels therefore y_ij is a vector? In this formula it seems it is a scalar. \n\n2 In the revision paper, I still cannot find the training loss. softmax is not a loss objective. A objective function should be something like\n\n                                                                                         min f(x)\n\n3 In comparison, although they are both ResNet-101, but ResNet-101 from CLIP is abosolutely better than other ResNet-101.\n\n===================================\n\nI have read the response and now think the comparison is OK.\nFrom Figure 2 I think every pixel has more than one label. But in Equation (2) it seems y_ij is a scalar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to semantic segmentation based on text embedding of class labels. This enables zero-shot semantic segmentation with class labels that were not seen during training. I appreciate the new ablation against a ResNet-101 backbone. I don't find the similarity with CLIP substantial, and I recommend that the paper is accepted.
Decision: Accept (Poster)