SwinZS3: Zero-Shot Semantic Segmentation with a Swin Transformer
Authors: Tian YingJie
Wang YiQi
Venue: Submitted to ICLR 2023 (2023)
Keywords: zero shot semantic segmentation
deep learning
transformer

Abstract: Zero-shot semantic segmentation (ZS3) aims at learning to classify the never-seen classes with zero training samples. Convolutional neural networks (CNNs) have recently achieved great success in this task. However, their limited attention ability constraints existing network architectures to reason based on word embeddings. In this light of the recent successes achieved by Swin Transformers, we propose SwinZS3, a new framework exploiting the visual embeddings and semantic embeddings on joint embedding space. The SwinZS3 combines a transformer image encoder with a language encoder. The image encoder is trained by pixel-text score maps using the dense language-guided semantic prototypes which are computed by the language encoder. This allows the SwinZS3 could recognize the unseen classes at test time without retraining. We experiment with our method on the  ZS3 standard benchmarks (PASCAL VOC and PASCAL Context) and the results demonstrate the effectiveness of our method by showing the state-of-art performance.

Under review as a conference paper at ICLR 2023
SWINZS3: ZERO-SHOT SEMANTIC SEGMENTATION WITH A SWIN TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
Zero-shot semantic segmentation (ZS3) aims learning to classify the never-seen classes with zero training samples. Convolutional neural networks (CNNs) have recently achieved great success in this task. However, their limited attention abil- ity constraints existing network architectures to reason based word embed- dings. In this light of the recent successes achieved by Swin Transformers, we propose Swin7S3, new framework exploiting the visual embeddings and seman- tic embeddings on joint embedding space. The SwinZS3 combines a transformer image encoder with language encoder. The image encoder is trained by pixel- text score maps using the dense Linguage-guided semantic prototypes which are computed by the language encoder. This allows the SwinZS3 could recognize the unseen classes at test time without retraining. We experiment with our method the ZS3 standard benchmarks (PASCAL VOC and PASCAL Context) and the results demonstrate the effectiveness of our method by showing the state-of-art performance.
INTRODUCTION
Semantic segmentation is the foundation of several high-level computer vision applications such as autonomous driving, medical imaging, and so on. Recent deep learning has achieved great suc- cess in semantic segmentation Chen al. (2018); Long al. (2015); Ronneberger al. (2015); Zhao et al. (2017b). However, the fully-supervised semantic segmentation models usually require extensive collections of labeled images with pixel-level annotations. And it could only handle the pre-defined classes. Considering the high cost of collecting dense labels, recently weakly supervised semantic(WSSS) segmentation methods have been explored. WSSS are often based on easily ob- taining annotations, such as scribbles Sun et bounding boxes Dai et al. (2015). and image-level labels Hou et al. (2018). Among them, popular trend is based on network visualization techniques like classification activation map generating pseudo ground-truths Zeiler & Fergus (2014); Zhang et al. (2020). However, these methods also require the networks to have labeled images.
On the contrary, humans could recognize novel classes with only descriptions. With the inspira- tion of this, some recent methods seek zero-shot semantic segmentation (ZS3) Zhao et al. (2017a); Bucher et al. (2019); Gu al. (2020); Li et al. (2020) ZS3 benefits from semantic level supervi- sion from texts by exploiting the semantic relationships between the pixels and the associated texts, which makes it enjoy cheaper source of training data. The ZS3 methods could be categorized into generative and discriminative methods Baek et al. (2021). Both could predict unseen classes using only ianguage-guided semantic information of the corresponding classes. For the generative ZS3 methods Creswell et al. (2018); Kingma & Welling (2013), segmentation networks are first trained with only seen classes labeled data. Then, they freeze the feature extractor extract seen classes' visual features and train a semantic generator network to translate the language embedding to visual space. By doing this, the semantic generator could generate visual features conditioned on language embedding vectors. Finally, classifier is trained to classify the features combined with features produced by the feature extractor on seen classes and generated features produced by the semantic generator from language embeddings on unseen classes. With generative methods achieve impressive performance in zero-shot semantic segmentation tasks. The methods are limited by a multi-stage training strategy, and the visual features extracted from the feature extractor not sidering the language information during training. This will cause a seen bias problem towards the visual and generated features.

---

Under review as a conference paper at ICLR 2023
Dorand
Figure 1: Effects of transformer's global reason and the score map decision boundary for zero-shot semantic segmentation. The motorbike(blue) is the unseen class. The existing solutions Deeplabv3+ often yield inaccurate segmentation results with limited receptive field and attention ability, losing fine-grained details. Using the transformer extractor significantly improves the prediction accuracy of the unseen classes. But there is still seen bias problem, which classifies the unseen class pixels into seen classes. So, our SwinZS3 proposes a language- guided score map to reduce it
To overcome the limitation, introduce discriminative approach for ZS3 that exploits to train visual and language encoders on a joint embedding space. During the training time, we avoid the multi-stage training strategy. We alleviate the and unseen bias problem by minimizing the euclidean distances and using the pixel-text score maps between the semantic prototypes produced by the language encoder and the visual features of corresponding classes. At test time, use the learning discriminative language prototypes and combine the pixel-text score map with the euclidean distance as the decision boundary avoid retraining.
Improving the network backbone of zero-shot networks is another effective way to reduce the bias problem. As shown Figure.1 We argue that shared shortcoming previous ZS3 models falls in the reduced receptive field of CNNs and less using attention mechanisms for extracting the global relations of visual features conditioned with language semantic information. The local nature of convolutions leads CNNs to extract visual features missing long-range relationships across the same image. CNNs based frameworks sometimes fail to extract language guided activation fields for lack- ing the global perceiving attention mechanism. Recently, transformers Vaswani et al. (2017) have significant breakthroughs in both natural language processing(NLP) and computer vision(CV) field Xie et al. (2021a); Zheng et al. (2021); Arnab et al. (2021). Dosovitskiy et al. (2020) (VIT) is the first work to apply the transformer architecture to image classification. Moreover, the Liu et al. (2021) (swin-transformer) presents cially dense predicting. We argue that the self-attention mechanism benefits the zero-shot semantic segmentation tasks and the semantic information supervisor. The transformer-basee model could capture the global feature relations and the semantic information in visual features by multi-head self-atten auffaattention(HHA))
architecture for more general-purpose vision tasks, espe-
This paper takes this missing step and explores the swin-tre transformer for ZS3. It combines convolu- tional layers and transformer blocks to model the global information guided by pixel-text distances and score maps. We also improve the decision boundary by modifying the Nearest Neighbor(NN) Classifier with weighted euclidean distance by score map. We demonstrate the effectiveness of our approach standard zero-shot semantic segmentation benchmarks, achieving slate-of-the-art per- formance on PASCAL-VOC Everingham et al. (2010) and PASCAL-ContexI Mottaghi et al. (2014).
Some methods based on CLIP (Radford al. (2021)Xuu et al. (2022)Ding al. (2022)Amab et al. (2021) Xu al. (2021)) often claim to be zero-shot learning methods. However, those methods usu- ally use all classes of images and text labels during training, which will cause supervision leakage.
2 RELATED WORK
Semantic segmentation: Semantic segmentation has made great advancements due to the rise of deep learning. Most recent state-of-the-an models are based on fully convolutional neural networks Long al. (2015) and assume that all the training data have pixel-level annotations. Deepl.ab
2

---

Under review as a conference paper at ICLR 2023
Semanii Prokctrpo
Langaage lasade
Lew
Regension Loss
Sarr Maps
vosal Extrater
Traedeener
dosettrr
CE
Vinal Festere Srcheddings
Figure 2: The overall framework of our approach SwinZS3. SwinZS3 first extracts the image visual embeddings using a transformer-based feature extractor and K-class semantic prototypes using a language encoder. The prototypes will have regression loss with the visual features and their inter- relationship are transferred from the language embeddings (word2uec) using semantic consistency loss. And then SwinZS3 calculates pixel-text visual features and semantic prototypes. The score maps are supervised by the ground-truth labels. The visual features also fed into classifier for being supervised by ground-truth labels with cross-entropy loss.
maps in hyper-sphere space for the projected
exhibited distinct performance improvement the PASCAL VOC2012 Everingham al. (2010) and MS-COCO Lin al. (2014) by using the multiple scales Li et al. (2020); Zhang et al. (2018) and dilated convolution Chen al. (2014; 2017). Other methods like UNet Ronneberger et al. (2015) and SegNet Badrinarayanan et al. (2017) also achieve an impressing performance by different strategies. The VIT Dosovitskiy al. (2020) is the first work use transformer architecture for recognition task. And swin transformer extends transformer to the dense prediction task and achieved the state- of-the-art performance. But they all heavily rely on expensive pixel-level segmentation labels, and they assume that training data for all categories is available beforehand. Recently, lots of weakly supervised semantic segmentation (WSSS) methods which are based on easily obtained annotations such as bounding boxes Dai et al. (2015), scribbles Sun et al. and image-level labels Hou et al. (2018) have been explored. The key to prevailing pipeline WSSS is to generate the pseudo-labels, especially network visualization techniques like CAM. And some works use growing strategies to grow the CAM ground-truth regions to the entire objects. But is still difficult to get the pseudo- labels revealing entire object areas with accurate boundaries for the ill ill-posed procedure Singh & Lee (2017); Li et al. (2018).
Zero-shot semantic segmentation: The 7S3 networks could be categorized into discriminative and generative methods. For the discriminative approach, the work of Zhao et al. (2017a) focuses on hierarchical predicting unseen classes by adopting the discriminative approach. SPNet Xian et al. (2019) exploits a semantic embedding space by mapping visual features to fixed semantic ones. JoEm Back al. (2021) propose align visual and semantic featuers in joint embedding space. In contrast to discriminative methods, ZS3Net Bucher et al. (2019) synthesize visual features by generative moment matching network (GMMN). However, the ZS3Net training pipeline consists of three stages that will cause the bias problem. CSRL Li et al. (2020) exploit the relations of both seen and unseen classes to preserve them to synthesized visual features. CaGNet Gu al. (2020) proposes to use the channel-wise attention mechanism in dilated convolutional layers for extracting visual features.
Visual-language learning: Recently years, image-language pairs learning is rapidly growing field.
There are some representative works such as CLIP Radford et al. (2021) and ALIGN Jia et al. (2021) which are pretrained on hundreds of millions of image-language pairs. Yang et al. (2022) presents the unified contrastive learning method that can leverage both the image-lange image-language methods and image-label data. And our work further extends the method to pixel-level on 7S3.
3

---

Under review as a conference paper at ICLR 2023
Unseen Classes
Seen Classes
Seen Classes
Unseen
.ll
Distance
Boundary
Modified the Decision Roundary by Score Map
Figure 3: Compare the difference between the decision boundary without and with the score map. We visualize the visual features by circles and the semantic prototypes by triangles. Because of the bias problem in zero-shot learning, the visual features of seen classes are tightly, and the unseen classes' semantic prototypes and visual features the euclidean distance dl is smaller than d2. So, the unseen classes' pixels will be classified into seen classes. However, for score map distance show al bigger than a2, which inspires to the score map distance to modify the euclidean distance. After adjusting, we could get the (b) view. is crucial to improve the performance ZS3.
biased. (a) We show of the situations where
3 METHOD
3.1 MOTIVATIONS
Unlike supervised semantic segmentation methods, The unseen classes prototypes of discriminative zero-shot semantic segmentation rely joint optimization of the visual encoder and language coder. Thus, to achieve good performance, this formulation requires the network to perceive the language context structure. Current network Back al. (2021) adopt traditional convolutional lay- ers for aggregating language information. However, the intrinsic locality and weak attention of the convolution operator hardly model long-range and accurate visual-language joint features. So, we propose to use transformer-based blocks to address the limitation. Another limitation of ZS3 is the seen bias problem. Lacking the unseen classes labeled data, is difficult for the visual encoder to extract distinguishable features. As shown in Fig.3, modulating the decision boundary could also reduce the bias problem. We analyze the shortcomings of traditional NN classifiers and the new decision boundary to improve the performance. Driven by this, we design our SwinZS3 framework. As shown in Fig.2.
3.2 OVERVIEW
Following the common practice, we divide classes into seen classes and unseen classes U. Dur- ing the training time, we train model, which includes visual feature extractor and semantic prototype encoder with the seen classes set S only. Zero-shot semantic segmentation aims to allow the model recognize both seen classes S and unseen classes U in the test time. We use the visual extractor to extract visual features and input language embeddings (word2vec) to the language en- coder for obtaining semantic prototypes of corresponding classes. The visual features will be input into a classifier and supervised by ground-truth labels. The language prototypes will have a regres- sion loss with the visual features. And the prototypes' inter-relationship are transferred from the language embeddings like word2vec using the semantic-consistency loss. And then SwinZS3 cal- culates pixel-text score maps in syper-sphere space for the projected visual features and semantic prototypes. These score maps also supervised by the ground-truth labels. The visual features also fed into a classifier for being supervised by ground truth labels. the following, we describe framework in detail.

---

Under review as a conference paper at ICLR 2023
3.3 TRANSFORMER BACKBONE
Our framework uses the transformer Liu et al. (2021) as backbone that an input image is split into non-overlapping x w patches. The patches will be projected attention (MHSA) layer is used for the transformer block to capture the global features information. In MHSA, the patch tokens are projected to queries E Ruxxd keys Ruxdd, and values V E TR Axx, h and w are the shapes of feature maps. dę and d, denote the dimension of features. Based on Q, K, and V. outputs X are
w tokens. The multi-head self-
QKT
X softmar
)V
Vd
The MHSA the core operation of the transformer block. The transformer backbone's final output is produced by stacking multiple transformer blocks.
3.4 NETWORK TRAINING
As shown in 2ig.2. our framework consists of four loss terms: cross-entropy loss Lee Misra & Maaten (2020), pixel-wise regression loss Ly, pixel-text score map loss Lauzs semantic consistency loss Lyc- The overall loss is finally formulated as
== + L, + M + dzLaux
(2)
where A and X balance the contributions of different losses.
Cross-entropy loss: Given the final outputs of feature maps Rexuxe, where h, w, and c the height, width, and the number of channels. Then, v will be put into classifier head fo. For the zero-shot setting, the classifier could learn the seen classes. So, we apply the cross-entropy loss Murphy (2012) that is widely adopted in supervised semantic segmentation on the seen classes set as follows:
@meri(p)
Lee
ΣΣ log( CES PEN.
EeES No
Ejes (w)((())))
where No indicates set of locations labeled as the class in ground-truth.
Regression loss: Although the lee could train the model to a discriminative embedding space on seen classes S. However, the model is not adaptable classify the unseen classes U while the clas- sifier head does not learn the unseen classes prototypes. At test time, we want to use the language prototypes of both seen and unseen classes as the classifier recognize the dense visual features extracted by the transformer backbone. For this, the distances of visual features and corresponding language prototypes should be minimized in embedding space. To address it, we introduced regres- sion loss 4,. As the lee. we first get the final outputs visual feature maps v E hxxxx,, Then, we get semantic feature maps s hxxxxd where each pixel 8, of a word language embedding and the same class with corresponding visual feature pixel. Given the language embedding maps, we input them to semantic encoder fa as follows:
= Ja(s)
(4)
where E hxxxx We denote each pixel of je is a semantic prototype for a class c. So, the regression loss follows:
ΣV Σ d(v(s),
L Eces Re
CES SER.
where d() is the euclidean distance metric. Rs means the regions labeled with the same class in the ground truth. The I, give promise that the dense visual features and semantic prototypes will be projected to a join embedding space where the pixels of corresponding classes will be close. But, for ZS3, there similar limitations with l.o: The 4 deal with pixel-wise visual features and semantic
5

---

Under review as a conference paper at ICLR 2023
Picab
Prozls
K Clance
Visal eapoder
wordhooe
Prnrypas
Prolatpps
Aroaad
cranker
Iegrage cnadde
cons-entooy
pteeloee nap
Semantio-smiiaaeey
Figure 4: An illustrative similarity matrix comparisons between different losses N the number of pixel samples fro image features extracted by visual encoder. K is the number of classes. (a) The cross-entropy loss could be considered as pixel-label learning by assigning pixels to ground- truth labels, and the relationship between the pixels and labels is many-to-one.
(b) The pixel-text
score map loss focuses on the relationship for the semantic prototypes and the visual features, which means semantic prototype will be assigned to many pixels. (c) For semantic-consistency loss, it keeps the structure of language embedding like word2vec.
prototypes independently but ignore explicitly considering the other pixels' relationship between them. To address it, we proposed to use the contrastive loss.
Pixel-text score map: In our framework, we use the score map to reduce the seen bias problem in ZS3. As shown in 4,g.4, to get the discriminative joint embedding space, we compute the pixel- text score maps using the language prototypes μec. E RRxxe and the final outputs of feature maps E hxxuuxe by:
opc E Thhwxxk
where Åe and the la normalized version of v and Me along the channel dimension. By the way, the in the score map must use the seen classes prototypes only. Otherwise, it will make the unseen bias problem and uanuuaee-guided semantic prototypes matching, which is one of the most crucial parts of our SwinZS3. First, the score maps compute an auxiliary segmentation loss:
serious. The score maps characterize the results of visual features pixel
laur CrosssEnropyySSofttt (s/T).))
(7)
where T is a temperature coefficient which we set 0.07 and y is the ground truth label. The auxiliary segmentation loss can make the joint embedding space more discriminative which is beneficial to zero-shot semantic segmentation.
Semantic consistency loss: The semantic consistency loss lse could transfer the relationship of word2wee to the semantic prototypes embedding space. For adopting the pre-trained word em- bedding features, lee could keep the class contextual information. The Lac defines the relation of prototypes as follows:
(
where d() is the distance between two prototypes and Tp is temperature parameter. Then for word embedding space, the relationship could defined:
6 -,,d(()));)
Túj Ejes e Tod(nang)
(9)

---

Under review as a conference paper at ICLR 2023
So, the semantic consistency loss is defined follows:
r reylog
Lsc = -E
(10)
XES JES
The Lse could distill the word embedding contextual information the prototypes.
3.5 NETWORK INFERENCE
During the inference, use the semantic prototypes, which the outputs of the semantic encoder as a NN classifier Cover & Hart (1967). We compute the euclidean distances and score maps from individual visual features language prototypes, and classify each visual feature the nearest language prototypes as follows:
(()) argmin d(v(p), siignid(sss))))
(11)
cESUU
where d is the euclidean distance metric and s is the score map. For the unseen classes U still being biased towards the seen classes S, we adapted the work Baek et al. (2021), which proposed the Apollonius circle. The lop2 nearest language prototypes with individual visual features are di and d2. d1 is the euclidean and score distance with the language prototype μn and d2 is the distance with language prototype Al where denote e1 and 0g the class of μn and u2. The decision rule is defined with the Apollonius circle as follows:
i(p) = {(o)
(d))
01 C and 02 EU ci otherwise
(12)
where
= S 02 di
di d2
7
(13)
d2
We denote IT as an function whose value if the argument is true, and 0 otherwise. The y is an adjustable parameter which could modulate the decision boundary.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Training: For transformer backbone, we use the swin transformer (swin-tiny) proposed Singh & Lee (2017), which is a baseline for transformer-baed ZS3 task. To avoid supervision leakage from unseen classes Xian et al. (2018), the backbone parameters are initialized with self-supervised model MoBY Xie et al. (2021b) pre-trained on Imagenet. We use AdamW optimizer as the optimizer to train SwinZS3. For the backbone, we set the initial learning rate as 10
10-4, 44, and it uses the polynomial scheduler decay every iteration. The other parameters' learning rate is 10 times the backbone parameters' learning rate. The weight decay factor is set as 0.01. For data augmentation, keep the same setting with Back et al. (2021). For other parameters (A, 7). we set \ X2 to 0.1 and the is 0.6.
We perform experiments on PASCAL VOC and PASCAL Context. The PASCAL- VOC2012 dataset contains 1464 training images and 1449 validation images with a total of 21 cate- gories (20 object categories and background). The PASCAL Context dataset contains 4998 training and 5105 validation samples of 60 classes with 59 different categories and single background class. Following the common practice, we adopt the 10582 augmented training samples for PAS- CAL VOC. For the zero-shot semantic segmentation network, we divide Pascal- samples according to N-seen and 20-N unseen classes. For example, we choose cow and motorbike as unseen categories. Then we filter out those samples with cow and motorbike labels and train the segmentation network using the remaining samples. During the training time, the segmentation model should keep the mIOU of unseen classes O. We follow the experiment settings provided by ZS3Net which dividing the Pascal-VOC 2012 training samples 20 object classes into four splits (1) 18-2 classes (cow, motorbike), (2) 16-4 classes (cat, sofa), (3) 14-6 classes (boat, fence), and
Dataset split:
training

---

Under review as a conference paper at ICLR 2023
Ground trath
JoEm
Figure 5:
Qualitative results on PASCAL VOC. The unseen classes is "cow"","moobbik" "mottooiie"""""cca We compare the results of the other state-of-ar method and our SwinZS3.
Table 1: Ablation study the unseen- 6 split of PASCAL Context by comparing mloU scores using different loss terms.
method
Ice
4
Lsc
mIoU,
mIoU,
hIoU
Deeplabv3+
33.4
8.4
13.4
Deeplabv3+
/
36.2
23.2
28.3
Deeplabv3+
/
/
1
37.7
25.0
30.2
SwinZS3
7
25.8
12.0
16.4
SwinZS3
37.1
24.3
29.3
SwinZS3
39.3
26.2
31.4
(4) 12-8 classes tymontor)))))))). Each split contains previous unseen classes gradually. Then the model evaluated on the full 1449 validation images.
We use the mean intersection-over-niiie (mIoU') as evaluation metrics Long al. (2015). In detail, we separately count the metries of the seen classes and the unseen classes which are denoted by mloo', and mloU,.. We also adopt the harmonic mean (hloV) of mloU, and mloU, for the arithmetic mean might be dominated by mlou...
Evaluation metrics:
4.2 ABLATION EXPERIMENT AND RESULTS
present ablation analysis on aspects: (a) CNNs Trans- former (b) Whether to use the score map (laue) to modulate the decision boundary. For cross-entropy loss and regression loss are crucial to recogmize the unseen classes, we choose the Deeplaby3+ with lee: I, and Lap as our baseline. For the first row, we report the baseline ToU scores without lava- Then, we compare swin transformer (swin-tiny) backbone with Deeplaby3+ and transformer back- bone gives gain of hloU 1.0 over the baseline. The second and third rows show that the louz respec- tively give gain of mloU, 3.0 and 3.1 over the baseline and 1.9, 2.1 over the SwinZS3 baseline. This shows significant improvement for the ZS3 and gives demonstration of the effectiveness of the two methods. Finally, we combine the transformer and score maps, and report the best mIoU scores.
Ablation study: In the table
Comparison to state-of-the-Arts As showed in table 2, we compare our approach with other state-of-art methods PASCAL VOC and Context. We report the best scores of different split
8

---

Under review as a conference paper at ICLR 2023
Table 2: Quantitative results the PASCAL VOC and Context validation sets. The numbers in bold are the best performance.
VOC
Context
method mloU.
mloVu
hloU
mloUs
mloUs
hloU
DeSi
68.1
3.2
6.1
35.8
2.7
5.0
SPNet
71.8
34.7
46.8
38.2
16.7
23.2
ZS3Net
72.0
35.4
47.5
41.6
21.6
28.4
CSRL
73.4
45.7
56.3
41.9
27.8
33.4
JoEm
68.9
43.2
53.1
38.2
32.9
35.3
Ours
69.2
45.8(+2.6)
55.3
39.8
33.5(+(0.6))
36.3(+(1.0)
VeVi
64.3
2.9
5.5
33.4
2.5
4.7
SPNet
67.3
21.8
32.9
36.3
18.1
24.2
ZS3Net
66.4
23.2
34.4
37.2
24.9
29.8
CSRL
69.8
31.7
43.6
39.8
23.9
29.9
JoEm
67.0
33.4
44.6
36,9
30.7
33.5
Ours
68.9
34.4(+1.0)
45.7(+1.1)
38.7
33.5(+2.8)
35.1(+1.6)
DeViSE
39.8
2.7
5.1
31.9
2.1
3.9
SPNet
64.5
20.1
30.6
31.9
19.9
24.5
ZS3Net
47.3
24.2
32.0
32.1
20.7
25.2
CSRL
66.2
29.4
40.7
35.5
22.0
27.2
JoEm
63.2
30.5
41.1
36.2
23.2
28.3
Ours
62.6
31.6(+1.1)
42.0(+0.9)
39.3(+3.1)
26.2(+3.0)
31.4(+3.1)
DeViSE
35.7
2.0
3.8
22.0
1.7
3.2
SPNet
61.2
19.9
30.0
28.6
14.3
19.1
ZS3Net
29.2
22.9
25.7
20.9
16.0
18.1
CSRL
62.4
26.9
37.6
31.7
18.1
23.0
JoEm 58.5
29.0
38.8
32.4
20.2
24.9
Ours
60.2
29.6(+0.6)
39.9(+1.1)
35.0(+2.6)
21.4(+1.2)
26.6(+1.7)
settings. And the other reported mloU are from Back et al. (2021). For PASCAL Context dataset, the comparison shows that we outperform the second best method JoEm by large margins, for outperform 3.0 in mloU, and 3.1 in hloU. It is remarkably outperforming for the ZS3. We outperform the best generative 7S3 method GSRL Li et al. (2020) and give a mJoU, gain of 4.4 and hToU gain of 4.2, which shows the effectiveness and convenience of the discriminative method. And the CSRL has to be retrained when the novel unseen classes are added. However, our framework adopts one-stage training strategy We achieve state-of-the-an performance on almost all the sero-shot settings like umseen-2.4,66 splits on mIoU, and hToU. It confirms our approach could learn the discriminative representations. The PASCAL VOC dataset experiments emphasize the above analysis. The PASCAL VOC experiment shows competitive performance.
the 6-split setting,
Qualitative results. Fig.5 reports some qualitative examples from PASCAL VOC, with SwinZS3 modeling the unseen classes more accurately than its competitors. We notice that SwinZS3 can effectively reduce the false positive prediction. reports some qualitative examples from PASCAL VOC, with SwinZS3 modeling the unseen classes more accurately than its competitors.
5 CONCLUSION
We have proposed transformer-based framework that exploits the visual and language features on the joint embedding space for zero-shot semantic segmentation. We have proposed to use the language -uiieeggddddddddd the decision boundary to reduce the seen bias problem. Finally, we experiment with our approach state-of-the-art performance.
map to better learn the discriminative space while innovatively modifying
standard 7S3 benchmarks and achieve

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481-2495 2017.
Donghyeon Back, Youngmin Oh, and Bumsub Ham. Exploiting a joint embedding space for general- ized zero-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9536-9545, 2021.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmenta- tion, Advances in Neural Information Processing Systems, 32, 2019.
Liang-Chieed Chiih Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected erfs. arXiv preprint arXiv:1412.7062, 2014.
Liang- Chichhhhhhh Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.0558%) 2017.
Liang- Chigg Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions infor-
mation theory, 13(1):21-27, 1967.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35 (1):53-65, 2018.
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolu- tional networks for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1635-1643, 2015.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11583-11592, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 /929, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Zhangxuan Gu, Siyuan Zhou, Niu, Zihan Zhao, and Liqing Zhang. Context-awars feature gen- eration for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Con- ference on Multimedia, pp. 1921-1929, 2020.
Qibin Hou, PengTao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object attention. Advances in Neural Information Processing Systems, 31, 2018.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:13/2.611,20 2013.
Auto-encoding variational bayes.
arXiv preprint
Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9215-9223, 2018.
Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmen- tation. Advances in Neural Information Processing Systems, 33:10317-10327, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3431-3440, 2015.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6707-6717, 2020.
Roozbeh Mottagii, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing network to be meticulous for weakly-supervised object and action localization. In 2017 IEFE international conference on computer vision (ICCV), pp. 3544-3553. 2017.
J Sun, D Lin, J Dai, Jia, and K Scribblesup He. Scribble-supervised convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, volume 26.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zhot learninga com- prehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-2265 2018.
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic pro- jection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8256-8265, 2019.

---

Under review as a conference paper at ICLR 2023
Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-
supervised learning with swin transformers. arXiv preprint 1xXiv:2105.04535 2021b.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. sim- ple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arXiv preprint 1rXiv:211..14755 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. arXiv preprint arXiv:2204.03610, 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference on computer vision, pp. 518-833. Springer, 2014.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Stan-Sheng Hua, and Qianru Sun. Causal interven- tion for supervisd semantic segmentation. Advances in Neural Information Processing Systems, 33:655-666, 2020.
Zhenli Zhang, Xiangyu Zhang, Chao Peng, Xiangyang Xue, and Jian Sun. Exfuse:
Enhancing feature fusion for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp.
2018.
Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2002- 2010, 2017a.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017b.
Sixiao Zheng. Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVE confer- ence on computer vision and pattern recognition, pp. 688166881898 16890, 2021.
12


Reviews:
[{"review_id": "hfCsbeSxed", "reviewer": "Reviewer_sxBw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a transformer based approach for zero shot semantic segmentation. It makes the use of different loss functions like cross entropy loss for seen classes, regression loss between language and visual features to account for unseen classes, a pixel text score map to reduce the seen bias problem and a semantic consistency loss to transfer the relationship of word2vec features to the semantic prototypes of the embedding space. State of the art results are shown on Pascal VOC and Context datasets.", "summary_of_the_review": "The paper has many errors and has a few missing ablation experiments. The method does not seem very novel as it mixes a combination of loss functions (mostly known) with transformers to improve results marginally over baselines. Overall the contributions are borderline and near the acceptance threshold if some concerns are alleviated. \n\nUpdated rating after reading other reviews.", "strength_and_weaknesses": "The method obtains reasonable improvements over good baselines and state of the art results. \nThe use of pixel text score map is interesting. \n\n\nNo ablation experiments for Lsc and Lr, which are described as sub-sections of the technical contribution in the approach. \nThe paper is not written well and has several errors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gq-lGyVUipl", "reviewer": "Reviewer_nqEw", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to tackle the problem of zero-shot semantic segmentation (ZS3). To capture the global feature relations and semantic information, SWIN transformer is adopted as the backbone. To improve the decision boundary, pixel-text auxiliary segmentation loss is used. Good benchmark results are obtained in Pascal VOC and Context datasets. ", "summary_of_the_review": "Due to lack of novelty, unfair experimental comparisons, and poor paper writing, the reviewer suggests rejecting this paper in its current form. The paper needs major revision before resubmitting. ", "strength_and_weaknesses": "Pros:\n1. The topic of zero-shot semantic segmentation is interesting and important. \n2. It achieves good benchmark results on Pascal VOC and Context datasets. (however the comparisons might be unfair, which will be described in Cons.)\n\n\nCons:\n1. The novelty of this paper is limited. It seems that the paper claims its novelty as using Swin Transformer, however, Swin Transformer has been widely used in CV community and simply adopts it as the backbone is not novel. In addition, the paper describes the cross-entropy loss, regression loss, semantic consistency loss in details. However, these techniques are proposed by JoEm (Baek et al. ICCV'2021). This paper simply uses them without any modifications. For L_aux, many papers have used similar losses. for example, Dong, Xiaoyi, et al. \"MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining.\" arXiv preprint arXiv:2208.12262 (2022)..\n \n2. The comparisons of the experiments (Table. 2) might be unfair. It uses stronger backbone (Swin Transformer) network compared to other methods. Please add experiments of DeepLabV3+ for all K in Table. 2. \n\n3. Please also discuss the relationship with CLIP-based zero-shot segmentation methods in the related work section. \n\n4. The paper writing needs significant improvement and careful revision. (1) The use of symbols is inconsistent. Does s in Eq.4 and Eq.6 the same? Some times N is used for the number of seen categories, sometimes K is used. (2) The introduction of \"regression loss\" is mostly unclear. where does semantic feature maps (s) come from? (3) The use of citation is wrong throughout the whole paper. (4) There are many typos. for example, extra \"()\" in Table. 2; \"deeplabv3+\" and \"Deeplabv3+\" in Table. 1. \n\nQuestions:\nThe paper writing is not clear enough. \n1. In Table.1, Does the 1st row (deeplabv3+) mean JoEm? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GVQab3adX0", "reviewer": "Reviewer_xJ8c", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a discriminative zero-shot semantic segmentation approach that uses transformer image and language encoders to exploit the global relations of visual features and to reduce seen class bias. In this approach, the image encoder is trained with calculating pixel-text score maps which use dense language-guided semantic prototypes. The authors argue that CNN-based ZS3 models sometimes fail to extract language-guided activation fields because they work in reduced receptive fields and use fewer attention mechanisms to extract global relations. Experimental results on benchmark Pascal VOC and Pascal Context datasets show that this approach (i.e. SwinZS3) obtains state-of-the-art results.", "summary_of_the_review": "The arguments put forward by the authors are worth examining. (Limited) experiments also show that the method obtains successful results. However, the main and additional experiments are incomplete from my point of view. In addition, the study also needs serious improvements in writing, the overall manuscript quality does not meet the ICLR standards.", "strength_and_weaknesses": "Strengths:\n- The proposed transformer framework appears to be promising. The obtained limited experimental results (please see weaknesses) also support this situation.\n- Nice to see the impact of auxiliary segmentation loss on SwinZS3 and DeepLab methods.\nWeaknesses:\n- The text flow feels irregular. For example, CLIP-based meyhods are mentioned regardless of the context at the end of the Introduction section. Some figures are not referred from the text (eg, Fig.1 and Fig.4). Grammatical problems also disrupts the flow. In addition, the text contains too many typos; e.g. \"fro\", \"networkBaek\", \"generatining\", etc.\n- The authors follow the experimental settings provided by ZS3Net, and use 4 different splits for comparison with previous methods. However, there are 5 different splits according to the ZS3Net method and they share the results for 5 different splits in their paper. Hence, unseen-10 split results seem to be missing for both Pascal VOC and Pascal Context datasets.\n- When the results shared in Table 2 are examined, it is observed that the CSRL method generally obtains better results for the seen classes. However, there is no clear experimental result to support the central claim that the seen/unseen bias is alleviated via minimizing the euclidean distance and using the pixel-text score maps.\n\n*ZS3Net: Bucher, M., Vu, T. H., Cord, M., & Pérez, P. (2019). Zero-shot semantic segmentation. Advances in Neural Information Processing Systems", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "41SuloVWaIQ", "reviewer": "Reviewer_S84P", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The article proposed a new network SwinZS3 that applies swin transformer to the zero-shot semantic segmentation domain. Experimental results on PASCAL VOC and PASCAL Context show that the method proposed in this paper outperforms existing methods.", "summary_of_the_review": "The method proposed in the article is not novel enough. In addition, there are major problems with the writing and layout of the article. I deem that it's not a solid work. So I tend to give weak reject.", "strength_and_weaknesses": "Strength\n\n      -The proposed method achieved relatively good results on benchmarks.\n\n\nWeakness\n\n\n      -The presentation of some parts of the article is unclear. For example I'm confused about the structure of Language Encoder, but the article doesn't explain what it is throughout.\n      -There are many problems with the layout and writing of the article. For example, there is incorrect capitalization in the first line of section 3.1 and missing spaces in line 4. There is confusion about the case of symbols in section 3.4.\n     -Some recent related papers are not cited. Such as [1, 2, 3]\n    [1] https://arxiv.org/abs/2112.14757\n    [2] https://openaccess.thecvf.com/content/CVPR2022/papers/Ding_Decoupling_Zero-Shot_Semantic_Segmentation_CVPR_2022_paper.pdf\n    [3] https://arxiv.org/abs/2202.11094", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject