ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
Authors: Pengzhen Ren
Changlin Li
Hang Xu
Yi Zhu
Guangrun Wang
Jianzhuang Liu
Xiaojun Chang
Xiaodan Liang
Venue: ICLR 2023 poster (2023)
Keywords: Zero-shot semantic segmentation
Vision-Language Pretraining
Visual Self-Supervision
Consistent Semantics

Abstract: Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation. However, existing works focus on pixel grouping and cross-modal semantic alignment, while ignoring the correspondence among multiple augmented views of the same image. To overcome such limitation, we propose multi-View Consistent learning (ViewCo) for text-supervised semantic segmentation. Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image. Additionally, we propose cross-view segmentation consistency modeling to address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders.  The text-to-views consistency benefits dense assignment of the visual features by encouraging different crops to align with the same text, while the cross-view segmentation consistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks. Trained with large-scale image-text data, our model can directly segment objects of arbitrary categories in a zero-shot manner. Extensive experiments show that ViewCo outperforms state-of-the-art methods on average by up to 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.

Published as a conference paper at ICLR 2023
VIEWCO: DISCOVERING TEXT-SUPERVISER SEGMEN- TATION MASKS VIA MULTI-VIEW SEMANTIC CONSIS- TENCY
Pengzhen Ren', Changlin Li,, Hang Xu², Yi Zhu?, Guangrun Wang", Jianzhuang Liu',
Xiaojun Chang", Xiaodan Liang!.-
'Sun Yat-sen University Huawei Noah's Ark Lab
RRLERR, AAII, University of Technology Sydney
Univeessity of Oxford 'MBZUAI
pzhren@foxmai com, (xu.hang, zhuy136, 1iu. jianzzaang com, xiaojun. xiaoojun.canngeute changeuts edu.
{changlinii,c wanggrun, xdliiang328)ggmsill com
ABSTRACT
Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation. However, existing works focus on pixel grouping and cross-modal semantic align- ment, while ignoring the correspondence among multiple augmented views of the same image. To overcome such limitation, we propose multi View Consistent learning (ViewCo) for text-supervised semantic segmentation. Specifically, we first propose text-to-views consistency modeling to learn correspondence for mul- tiple views of the same input image. Additionally, we propose cross-view segmen- tation consistency modeling address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders. The text-to-views consistency benefits the dense assignment of the visual features by encouraging different crops to align with the same text, while the cross view segmentation con- sistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks. Trained with large-scale image-text data, our model can directly segment objects arbitrary categories in a zero-shot manner. Extensive experiments show that ViewCo outperforms state- of-the-art methods on average by up to 2.9%, 1.6%, and 2.4% mloU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.
INTRODUCTION
Recently, vision-language contrastive learning (Radford et al. (2021) Li et al. (2021a)
has attracted a lot attention because it can obtain more generalized feature representa- tion. And at the same time, it can also make use of abundant image-text pairs to avoid labor-intensive annotation costs. Vision- languale pre-training (VLP) models have exhibited impressive potential in various visual (Xu et al. (2022); Mu
et al. (2021; Radford et al. (202T)) and multimodal (Wang al. 2021bb; Kim et al. (2021) tasks, including text-
image
supervised semantic segmentation (Xu et al. (2022); Ghiasi
view
al. 2022] ; Xu al. (222); Zabari & Hoshen (2021): Zhou
text
et al. 2021a), which uses text instead of traditional dense la- bels for supervision achieve zero-shot semantic segmenta- tion. It provides feasible solution for learning segmentation masks without mask annotation.
?
Advocary Dsy, pociss
However, existing works with CLIP-based (Radford et al. 202T segmentation (Xu al. (2022) 2021); Zhou et al, 2021a)) mainly focus pixel grouping cross-modal mantic alignment. They have the following two obvious limi- tations: (i) the excessive strictness of image-text correspon- dence; and (ii) the ambiguity text description. First, in vanilla vision-language contrastive learning, each image-text
Figure 1: Illustration of text descrip- tion ambiguity. Text descriptions highly abstract and difficult mantically aligned with images. Cross- view semantic consistency modeling effectively alleviate the effect of the text description ambiguity issuc.
se-
"Corresponding author.
"ooee release: https:: http:///ggthbbb..cm///rr////iiir

---

Published as a conference paper at ICLR 2023
12
21
Input
(a) GroupVIT
(b) ViewCo
The consistent comparison of semantic segmentation results in multiple views of "horse". (a) GroupViTi VIT: semantic segmentations of different views are inconsistent. (b) ViewCo: the semantic segmen- tations of different views are much more consistent. Here, x, u, and v represent the segmentation results on the original image and views v, respectively.
Figure 2:
pair is regarded as a unique positive pair, while all the other combinations are regarded as negative ones. This image-text correspondence is actually too rigorous. In fact, one textual description may correspond to different images. The excessive strictness is not conducive to the model learning high- level cross-modal semantic correspondences. Therefore, more relaxed vision-language contrastive learning needs to be considered. Second, the ambiguity of textual descriptions is also key chal- lenge. Compared with the traditional semantic segmentation pipeline that uses dense annotations as supervision information (Touvron et al. (2021); Ren et al. (2022)) the CLIPP-based segmentation methods (Xu et al. (2022) 2021); /Zhou et al. (2021a)) use text as supervision, which is easier to access but more noisy and ambiguous.
This is mainly because compared with traditional segmentation annotations, text descriptions are often more abstract and do not contain location information. Moreover, the background in the image is usually ignored in the description. In some cases, the objects in the image do not even exist in the text description (see Figure []. Such ambiguity is common in the textual supervision in vision- language pre- pre-training. In the semantic segmentation task, the ambiguity of textual supervision makes the segmented object-label correspondence very fragile. Therefore, Fully mining the information carried by the dataset itself may need to be considered.
On the other hand, visual self-supervision (Caron al. (2021) p; He et al. (2022); Chen et al. 2020a); Zhou et al. (20215)) has been widely used for visual pre-training. It includes two categories: re- constructing masked images (He al. (2022); Zhou al. (202Tb)) and multicrop image contrast (Caron et al. (2021); Chen et al. (2020a)). For example, SLIP (Mu et al. (202I)) introduces con- trastive learning of multicrop visual consistency for VLP. MaskCLIP (Dong et al. (2022) introduces visual self- supervieed task of reconstructing masked images. They utilize visual sell-supervision to provide more useful information for VI.P models. However, the semantic consistency of multiple views of an image in segmentation and cross-modal contrast have not received enough attention and research.
Based on the above observations, in this paper, we explore the impact of multi-view semantic con- sistency on the task of text-supervised semantic segmentation through visual self-supervision. To this end, propose multi- View Consistency learning (ViewCo), which aims discovering text- supervised segmentation masks via multi-view semantic consistency. Specifically, we propose text- to-views consistency modeling to alleviate the excessive strietness of image-text correspondence in vanilla vision-language contrastive learning. It enables the model to benefit from the dense assign- ment of visual features by encouraging different crops to align with the same text. This relaxed one-to-many contrast mechanism also facilitates the learning of multi-view consistent semantics, enabling the model to acquire high-level cross-modal alignment capabilities. Moreover, as shown in Figure to alleviate the ambiguity issue of textual supervision, we propose cross-view segmenta- tion consistency modeling. It overcomes the limitation imposed by textual ambiguity by providing additional self-supervision to vision-language contrastive learning via cross-vien view segmentation con- sistency. ViewCo uses the proposed text-to-views consistency modeling for vision-language cross- modal contrastive learning and additionally enables cross-view segmentation consistency modeling by contrasting the segment features of Siamese visual encoders. As shown in Figure 2 with the help of the two consistency modeling schemes, ViewCo establishes solid semantic correspondence in different views, and the semantics in different views maintain a good consistency. The semantic consistency of Group ViT in different views is difficult to guarantee.
Overall, ViewCo's design is simple and effective. We train on large-scale image-text pair datasets CC12M (Changpinyo al. (202T) and YFCC (Thomee al. (2016)). In the inference stage,
2

---

Published as a conference paper at ICLR 2023
use the similarity scores between the segmentation embeddings generated by the teacher network and the label prompts to assign labels to the image masks for zero-shot semantic segmentation. Compared with the ofate-of-the-ar methods, ViewCo achieves average improvement of 2.9%, 1.6%, and 2.4% mloU on PASCAL VOC2012, PASCAL Context, and COCO, respectively. Our contributions can be summarized as follows:
We propose a novel one-to-many text-to-views consistency modeling that improves the model's ability of high-level cross-modal semantic alignment by encouraging different crops of an image to align with the same text.
To alleviate the problem of supervision failure that may arise from text ambiguity, we pro- pose cross-view segmentation consistency modeling to provide additional self-supervision for the vision branch and encourage the model to generate consistent segmentation masks for different views.
ViewCo consistently outperforms the statc-of-the-art methods on PASCAL VOC2012, PASCAL Context, and MS-COCO when pre-trained on CC12M or CC12M+YFCC.
2 RELATED WORK
Vision- Language Pretraining. In recent years, vision-language pre-training models (Chen et al.
2 Desai & Johnson (2021); Li et al. (2020a) 202 2020bb) have developed rapidly with the help large-scale image-text pair data available
the Internet. Recently, VLP models such as CLIP (Radford et al. (2021)) ALIGN (Li et al. 2 and SLIP (Mu et al. (2021)) have made
great progress in visual representation learning by using contrastive learning. And they have been successfully transferred to various downstream tasks, such as visual question answering (Antol et al.
(2015); Zhou et al. (2020) and visual reasoning (Zellers et al. (2019)) In particular, CLIP (Radford jet al. (2221) uses the image-text matching relationship for contrastive learning, and the learned model can be directly transferred to ImageNet classification (Deng et al. (2009)) in a zero-shot manner without any fine-tuning. This success is also found on zero-shot semantic segmentation (Xu et al. (022)) ). However, the one-to-one contrastive learning mechanism between image and text in the vanilla VLP pipeline is strict, which is not conducive to the model learning high-level cross- modal semantic alignment. Based on the above observations, this paper proposes one-to-many text- to-views consistency modeling. It relaxes the original one-to-one correspondence by encouraging different crops of an image to match the same text, allowing the model to benefit from the dense assignment of the visual features.
Visual Self-Supervision. This framework relies on the information carried by the image it- self for self-supervision without any additional annotation information. Visual self-supervision is mainly divided into generative (He et al. (2022); Bao et al.] (202T)) and contrastive (He generative model allows the model to learn the feature representation of the image by reconstructing the masked image. Con- trastive models focus more on learning-centric global representations. Since semantic seg- mentation requires dense prediction of images, generative models may not help much because they destroy the original structure and information of im- ages. On the other hand, the contrastive visual self- supervised model can provide the required multi-view features for ViewCo's text-to-views consistency model- ing. Moreover, this visual contrastive learning can pro- vide additional visual self-supervision information for the VLP model to alleviate the risk of supervision failure caused by text ambiguity. Therefore, this paper focuses on the help of visual contrastive learning for semantic segmentation consistency.
al. (2020a): Caron et al. (202T); Chen et al. (2020a)).
view
pull together positive peirs push away negative peirs
text
-
distance between different objects
Consistent Semantics. Capturing consistent semantics one of the main challenges shared by many tasks such cross-modality and visual understanding. Vision- language contrastive learning (Radford et al. (202T) is essential to encode different modal data into the same fea-
Comparison of single-view text-to-image contrastive learning (left) and multi-view text-to-views contrastive learn- ing (right).
Figure 3:
ture space and enforces the features sharing the same semantics to get closer, and the features with different semantics to be pushed away. Similarly, multicrop image-level semantic consistency is
3

---

Published as a conference paper at ICLR 2023
Cross: View Segmontation Consistancy Module
Text: Views Consistency Module
segment tokm view features
DVG
MLP
fz
2
Patroot Nouns Prompts
EMA
(sor)
A woran using tablet while it's snowing.
ML.
photo of
ft
Fy
A photo of sablet.
Figure 4: Framework of ViewCo, is mainly composed of cross-view segmentation consistency module and a text-to-views consistency module. The visual branch adopts a visual self-supervised model, which consists of teacher fo and student fa networks with the same structure. ft and fa are the bottom-up up segmentation backhone that outputs segment features of the image.
also the core idea of visual self-supervised contrastive learning (Caron et al. 202 ) Chen et al. 2020aa;; He et al. (2020a)). For example, DenseCL (Wang et al. (2021a)) performs pixel-level dense contrastive learning on dense output vectors from multiple views, which is not helpful to the learning of high-level global semantic information. Further, GroupViT (Xu et al. (2022) uses text as supervision and achieves pixel grouping by capturing the contextual consistent semantics of images. However, the text-supervised semantic segmentation task, the ambiguous properties of text relative to dense annotations result in that the semantic consistency of images sharing the same semantics cannot be sufficiently guaranteed the embedding space. Furthermore, the strict one-to- one correspondence between image and text in the vanilla VLP model is also not conducive to the true alignment of high-level cross modal semantics. Figure (left) illustrates the above observation: although one of the views of image (e.g., the solid circle) is already close to the corresponding text embedding, other views (e.g., the dashed circles) may still be far away. Previous VLP methods generally only focus the alignment of single view with text. In contrast, shown in Figure 3 (right), ViewCo focuses on text-to-views consistency modeling, doing one-to-many matching in cross-modal contrastive learning.
3
MULTI- CONSISTENT LEARNING
As shown in Figure 4 our ViewCo is mainly composed of cross-view segmentation consistency module and a text-to-views consistency module. We describe these two modules Sections 3.1 and respectively, and summarize the final loss function in Section 3.3
3.2
3.1
VIEW SEGMENTATION CONSISTENCY MODULE
As shown in Figure 4 (left), given a batch of image-text pairs <<</,</)){ two random aug- mentations are performed on the input image z/. generating two warped views u and v. We use Group VIT (Xu et al.) (2022)) as the bottom-up segmentation backbone of ViewCo, where each view segmented into segment tokens. For each of the views (e.g., u), this process is expressed as: 2% (Zu. h 1,... K} [u( E Rxxd where Zee, E Rdd is the k-th segment feature from far and d is the dimensionality of the segment feature. Similarly, we have Zwe, and the seg- ment features Zu: and z from the teacher network, ft. We update the parameters of fe using the exponential moving average (EMA) He et al. (2020b) of the parameters of Ss. For example, let 0 and 0; be the parameters of f. and f: at training step i, respectively, and then 0, is updated as: 0, -0001 + (1 a),,, where or is a hyper-parameter for smoothing the update. In addition, the standard contrastive loss function, called InfoNCE (Oord al. (2018)) is considered in this paper, for an encoded query q and a set of encoded samples k [ko, K1, ..."" N that are the keys of dictionary, we have:
explg k+/T)
LNcelg,k) k) = log
(I)
Σn exp(q k2/T)
where 7 a learnable temperature parameter. And and kG are positive pairs, and the other (N-1) -1) pairs negative.
4

---

Published as a conference paper at ICLR 2023
positive pairs
teacher
u/v
views
negative pairs
Ris.
Alee.
Alie.
Alie.
x
A2
+
student
Pin.
Flg..
23
(a) [sse
(b) Liuest-Vees Views
Figure 5: Illustration the contrastive loss (a) cross-view segmentation consistency modeling and (h) text- to-views consistency modeling- Z is the k-th semantic feature of the i-th image (i.e., view u v). Z and 2 are the embeddings the views v u of the i-th image, respectively.
Intuitively, the segment features obtained from different crops of the same image should be roughly the same, i.e., cross-view segmentation consistency. To this end, for the semantic segmentation task, we replace the image-level contrastive learning in previous methods (Caron et al. (2021); Zhou et al. (2021b) with cross-view segmentation consistency learning within images. Therefore, we define the minimization training objective of the cross-view segmentation consistency module in ViewCo as:
Li-
(Seg]
It bi-directional contrast loss between the segment features from the teacher fa and the student fs- [seg] considers two pairs of views (i.e., (u,, v.) and (1t, U.)) outputted by f. and fa- The segment features of (u,, v2) from the same image are multiplied (2 -2v. [ after l2 normalization. In the image branch ViewCo, we use the EMA policy for parameter updates, so the learnable grouping tokens the corresponding position IDs of different views of the same image highly correlated, and they have the same semantics. Therefore, the semantic pairs {(Z Z ), 1,..., K} on the diagonal are regarded as positive, and the other K K(K-1) pairs {(2 2 ) 1,..., K * j) are regarded as negative. Therefore, the contrastive loss [|see] of the teacher-to- student segment features is defined as [Seg] Lu, -,,, + Lu, -,,, more specifically:
C Sogl
(CNCE
LNc 1Z 1K) LNCE (Za { 1K Ik=1)).
(3)
KB
Similarly, the contrastive loss Lisegi of the student-to-teache segment features is defined as Lisog)
bu, + far more specifically:
Seg)
(ENCE(Zug, (Z (Km) M + ENCE(Z"ag (Z }KK }k=)))
(4)
KB
Figure 5a shows the positive and negative pairs for vies segmentation consisteney learning in the vision branch.
3.2 TEXT-TO-VIEW CONSISTENCY MODULE
Previous methods Radford al. (2021) Xu al. (022)) build visual-linguistic semantic corre- spondences by performing a contrastive loss on image-text pairs. In this paper, we consider the contrastive learning between multiple views and text, using one-to-many text-to-views consistency modeling instead of one-to-one text-to-image contrastive learning. The model learns to capture intra-modal and inter-modal semantic consistency through the alignment of multi-view images and text.
Specifically, for a given image-text pair (a/, af)), by performing two different augmentations to the input image, we have triplet (u, 02 o;,,;") As shown in Figure 4 (right), in the training phase, we take the output (Z 2 of the view pair (u,,z) through the student network f. and the output Z of the text encoder ET to calculate the contrastive loss respectively. The visual embeddings (Z,", Z and text embedding 2 are mapped to the same feature space through two ML.S respectively, before performing the final la regularization. This procedure is represented
5

---

Published as a conference paper at ICLR 2023
- MLP(AvgPPol (Syg )), Z Tu; Z -MLP(AvgPool [
fv.)..
The multi-view feature 2 = - " Z:") and text embedding z constitute positive pairs, and the
other ( pairs are negative pairs. The contrastive loss of text-tovviews consistency modeling is defined as follows:
[ii = (i -T + -r-
(5)
where the contrastive loss of views ((u,))-00 -to-tett-extteee defined as:
B
K
1
LI(
( (2, ,
LNcE(Z,', (2,1 1)).
(6)
KB
and the contrastive loss of text-to-views [(N,o) is defined as:
B
CT-I(
1
L
E (CNcE(Z) ( 12 + LNCE (Zf 1
KB
Additionally, in order further enhance the association between mieti-view semantics and text semantics, we also compute the multi-label image-text contrastive loss (Xu et al. (2022)) of multi- view and "prompted text" pairs {(Z," .{Z?" J 1, 1
Jm= 1 "
J.M
where {-,- I.M J==1
are the embeddings of the additional M text prompts {T LM generated by the i-th text at ac- ie., the Im=1 are positive pairs, and the other combinations are negative pairs. Therefore, similar to (5. the multi-label M
cording to the "prompt engineering" mechanism (Radford et al. (021 (Z ) )m==),
embedding of the i-th image view u and the generated M text embeddings { 1M
contrastive loss of multi-view [[u,o} and multi-prompt (7 [ m==1 is defined as:
L 1 (TT
(T" + f -(
(8)
First, the views-to- pooproptts loss is the average of the losses of the two views. Considering a single view, e.g. u, the contrastive loss of to all the prompts is defined as:
1 B
MM 1== exp( Z." 2, /T)
L (I =
log
(9)
EME
Σ exp(Z 2/-0 2zn (T)
Second, the contrastive loss of multi- propptto-viiisss is defined as:
M
LirmYM M ( +I
Σ
(C/ce(Z)" 12 pam) + LNCE(Z (2/")E)))
(10)
2MB
In particular, similar work to our text-to-views consistency module is DeCLIP Li et al. 202 bb) believes that the text description may only be a small part of the image, so in addition to the global view in CLIP (Radford et al.) (2021)), DeCLIP also adds a local view for image self- seff-supervision, which may cause information leakage. In addition, DeCLIP uses EDA (Wei Zou (219)) as a text augmentation strategy. The augmented text still contains multiple semantics, which is not helpful to the alignment of local semantics in segmentation tasks. In contrast, ViewCo uses self-supervision of two local views to ensure the difficulty of the task, while using "prompt engineering" mechanism to obtain an augmented text with a single semantic. Combining one-to-man many alignment can help ViewCo to better mine consistent segmentation semanties in images.
3.3 OVERALL Loss FUNCTION
Finally, the total loss of VoewCo is the sum of the cross-view segmentation consistency contrastive loss and the two cross-modal contrastive losses:
C =
cttt + LI + L, ((u.e) (T (Sog]
(11)
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS Architecture. In the cross-view segmentation consistency module, f. and fa have the same network structure. The parameters of f, are updated using the exponential moving average of the parameters

---

Published as a conference paper at ICLR 2023
Pro-training
Transfer (mloU (66)
PASCAL vOC
PASCAL
Arch
Model
Dataset
Supervision
Zero-Shod
COCO
Context
De'T
ImageNet
class
X
53.0
35.9
DINO
ImageNet
solf
X
39.1
20.4
VIT
DINO
CC12M-YFCC
self
X
37.6
22.8
MoCo
ImageNet
seir
X
34.3
21.3
MoCo
CC12M-YKCC
self
X
36.1
23.0
CLIP
LAION-20M
text
13.5
8.2
GroupVIT
CC12M
text
41.1
18.2
18.4
GroupVIT'
CC12M-YPCC
TXXT
51.2
22.3
20.9
SLIP
LAION-20M
text & self
12.3
8.8
CLIP
CLIP-MAE
LAION-2UM
text & sell
16.8
11.8
MuskCLIP
20M
text & sell
17.7
11.8
ViewCo (ours)
CC12M
texi & sell
45.7
20.8
20.6
ViewCo (ours)
CCI MM-YCCC
text & self
52.4
23.0
23.5
Comparisons with recent methods. Zero-shot means that the model is directly transferred to the semantic segmentation task without any fine-tuning the target dataset.
Table 2:
of fa. We use GroupVil ViT (Xu et al. (2022)) with two stages the backbone for semantic feature extraction ViewCo's visual branch. is built on ViT-S (Dosovitskiy et al. (2020); Touvron et al. (202T) with 12 Transformer layers. The input image size is 224 x 224, the patch size is 16 x 16, and the hidden dimensionality is 384. The 2-stage GroupViT finally outputs segment tokens, i.e., K = 8). Following Radford al. (2021), ViewCo's text encoder E, consists of 12 Transformer layers with a hidden feature dimensionality of 256.
Training and Inference. In the training phase, filtered YFCC (Thomee et al. (2016)) as training datasets, which contain 12M and 14M image-text pairs, respectively. See A. of the supplementary material for more training details. the inference phase, following (Xu et al. (2022); Radford et al. (2021), the image is segmented by associating the image patches with the K segment tokens outputted by the teacher network ft- The semantics in the images are further classified by computing the similarity of the K visual-semantic embeddings to the text embeddings generated from the class labels of the test dataset.
CC12M (Changpinyo al. (2021) and the
Zero-Shot Transfer to Semantic Segmentation. We evaluate ViewCo on the task of zero-shot transfer semantic segmentation on the validation sets PASCAL VOC 2012 (Everingham et al. 2210)), PASCAL. Context (Mottaghi et al. (2014)) and COCO Stuff (Lin et al. (2014)) datasets. The three datasets contain 20, 59, and 80 foreground classes and an additional background class, respectively. During inference, following Group VIT (Xu et al. (202))), ViewCo predicts only the foreground classes by thresholding the the outputted image segments and the text segmentation labels. The thresholds on PASCAL VOC segeeneeeeee
normmliized-siillat
between the embedding of
2012, PASCAL Context, and COCO are set to 0.95, 0.35, and 0.95, respectively. We resize each input image have shorter side of 448.
4.2 COMPARISONS WITH RECENT METHODS
We first compare the performance of ViewCo with some ViT-S-based zero-shot baselines. Then, to further evaluate the performance of ViewCo the zero-shot semantic segmentation task, compare ViewCo with some fully supervised transfer and CLIP-based models.
Comparison with Zero-Shot Baselines. Table I shows the performance comparison of ViewCo and zero-shot baselines on PASCAL VOC 2012. Among them, the four ViT-based baselines train vision and text encoders through the image-text contrastive loss defined in CLIP (Rafford et al.] (202T)). They adopt four different pixel grouping methods: pixel-wise, K-means, Mean-shift (Comaniciu & Meer (2002). and Spectral clustering (Shi & Malik (1977) respectively. And GiT (Xu) al. (2022)) uses the bottom-up patch grouping mecha- nism. As shown in Table [] ViewCo significantly outperforms the CLIP-trained VIT and Group ViT (52.4% vs. 51.2%) baselines. It is worth noting that ViewCo and Group ViT adopt the same segmen- tation backbone, indicating that ViewCo can effectively improve the model's I's ability of segmentation and cross-modal semantic alignment with the help of the consistent semantic modelings.
Mask
Method
mIoU
VIT
pixel-wise
20.1
VIT
K-means
25.0
VIT
Spectral clustering
19.7
Group
$1.2
ViewCo (ours)
52.4
Table 1: Comparison with zero-shot base- lines on PASCAL VOC 2012.
7

---

Published as a conference paper at ICLR 2023
Comparison with Other SoTA Methods. These methods include fully supervised baseline DeiT (Touvron et al. (202I): D. two visual self-supervised baselines DINO (Caron et al. (2021) and
MoCo (He al. (2020ab), two vision-language contrastive learning baselines CLIP (Radford et al.
(20TT) and ViT (Xu et al. (,022), ), two vision-language contrast and visual self-suuervised
learning combined baselines SLIP (Mu et al. (202T) and CLIP-MAE (Dong et al. (2022). )) and one
vision-language contrast and self-distillation combined baseline MaskCLIP (Dong et al. (2022) )
Table[[shows the mioU performance comparison between ViewCo and the SoTA methods on PAS- CAL VOC 2012, PASCAL Context, and COCO validation sets. ViewCo significantly outperforms them on all three datasets. Compared Group ViT. when pre-trained on CC12M, ViewCo achieves a 4.6% mIoU improvement on PASCAL VOC. Similarly, when pre-trained on CC12M+ ViewCo achieves a 2.6% mloU improvement on COCO compared to GroupViT. Similar to ViewCo, SLIP, MaskCLIP, and CLIP-MAE all use additional supervision information in the vision branch of the VLP models. Compared with them, ViewCo still has clear advantages in PASCAL Context and COCO. In addition, ViewCo obtains segmentation performance close to the fully supervised DeiT on PASCAL VOC, which again demonstrates the effectiveness of ViewCo for zero-shot semantic segmentation.
4.3 ANALYSIS
In this section, for the convenience of comparison, we use CC12M as the pre-training dataset by default for the ablation of ViewCo components, qualitative analysis, and image classification perfor- mance comparison. Group VIT (Xu et al. (2022) is used as the baseline for ViewCo.
Image-Level Contrast vs. Semantic-Level Contrast. To ab- late the role of the cross-view segmentation consistency mod- ule in the vision branch, we add an image-level contrastive module to ViT in the visual branch, where we first cal- culate the average of the K segment tokens outputted by the teacher and student networks, and then perform contrastive learning. For ViewCo, we remove the text-to-views consis- tency module and directly average pool the multi-view fea- tures outputted by the student network. To be consistent with Gr,upViT, we the pooled visual features for contrastive learning with text embeddings. As shown in Table adding a visual self-supervised module for vision-language contrastive learning can improve the performance of the model on semantic segmentation by improving the quality of visual feature learning. Furthermore, the improved performance (i.e., 19.1 vs. 18.6) of semantic-level learning relative to image-level contrastive learning suggests that the cross-view seg- mentation consistency module can further improve the performance by capturing the consistency of cross-view semantic segmentation.
COCO
Visual branch
mloU (%)
GropViT GrouPVT++ ViewCo
18.4
image-level
18.6
semantic-level
(9.1
Table 3: Image-level contrast vs.
semantic-level contrast. "_" indicates that visual self-supervision module used. GroupViT+ represents modi- fying the corresponding component in GroupViT.
Vision-Language Contrast: Text-to-Image vs. Text-to- We further ablate the text-to-views consistency module in ViewCo. In single-view vision-language contrastive learning, use the average embedding of multi-view features out- putted by the student network and the text embed- ding for contrastive learning during training. As shown in Table 4 text-to-views consistency mod- eling significantly improves the performances of the models compared single-view text-to-image (i.e., 1.1% and 1.5%). This indicates that text-to- views consistency modeling has better high-level semantic alignment capabilities than text-to-image single-view modeling. This is exactly what pre- vious methods of single-view vision-language contrastive learning do not have.
Vision
COCO mloU (%)
Visual branch
language
contrast
GroupWi++
image-level
single
18.6
GroupiT+
image level
multiple
19.7 (1.1f)
ViewCo
semantic-level
single
19.1
ViewCo
semantic-level
maltiple
20,6 (1.5t)
Table 4: Vision-language contrast: single-view
ys. multi-view. "single" and "multiple" denote the number of image views used in lanion-language contrastive learning.
Qualitative Analysis. Figure 2 shows some visualization results of multi-view semantic seg- mentation consistency for ViewCo and GrTupViT. As shown in Figure aa), GrT,, the semantic segmentations of different views from the same image are inconsistent. For exam-
'The latest version of Group reports the results of training on the CCAM+CC12M-YFCC dataset.
8

---

Published as a conference paper at ICLR 2023
dog
Camuppyr
bicycle
cal
bottle
VieNCo (tuns)
(a) segmentation and prediction
(b) segmentation
Figure 6: Comparison of semantic segmentation of images in scenes. (a) Image segmentation and seman- tie prediction. (h) Image segmentation. ViewCo can better learn high-level cross-modal semantic alignment with the help two consistency modeling schemes.
plc, in image classified as "horse". There also the problem of inconsistent semantic segmentations be- tween views and
"umbrella" is misclassified as "cow", and in view "umbrella" is mis-
As shown in Figure 2bb), the semantic segmentation in differ- ent views ViewCo is completely consistent. This shows that our cross-view segmenta- tion consistency modeling and text-to-views consistency modeling in ViewCo are effective.
To evaluate ViewCo's s ability to perform seman- tic segmentation through semantic understand- ing in rare scenes, we show the more visual comparison in Figure 6 The images of scenes are selected from the Internet. In Fig- ure (().) we use the class labels of the PASCAL VOC 2012 dataset as the label set for the im- ages. ViewCo's segmentation and prediction re- sults in rare scenes are significantly better than GroupViT's. This indicates that ViewCo can better understand high-level semantics in images through consistent semantic learning. In Figure we only focus on the model's ''s ability to segment images in rare scenes. Compared to GroupViT, ViewCo handles the details of image segmentation much better.
Pre-training
Zero-shot
dataset
Acc@1 (%6
Acc@ (%)
GroupViT
CCT2M
37.3
65.5
ViewCo
CC12M
39.5 (2.0f)
68.4 (2.91)
VIT
CC12M+YPCC
42,4
GrupVVT
CCI2M+YFCC
42.9
71.7
ViewCo
YCCM
46.3 (3.49)
74.0 (2.3 1)
Table 5: Zero-shot performance on ImageNet.
More visual comparison results are shown in Figure 7 Af. of the supplementary material. In addi- tion, we also visually compare the segmentation consistency of ViewCo and ViT on different Finally, we present an analysis of ViewCo's cross-view segmentation consistency in
views in A.3
A.4
Image Classification. We also evaluate the classification performance of ViewCo. As shown in Table 5 ViewCo significantly outperforms ViT (i.e., 46.3% vs. 42.4%) and GroupVil VIT (i.e., 46.3% vs. 42.9%) showing that ViewCo achieves better cross-modal semantic alignment through text-to- views consistency modeling.
5 CONCLUSION
We propose a novel and simple multi-view consistency learning (ViewCo) for text-supervised se- mantic segmentation. To deal with the problems of excessively strict image-text correspondence and ambiguous text supervision in the VLP model, ViewCo models the text-to-views consistency and cross- view segmentation consistency. ViewCo can generate consistent segmentations and better capture high-level cross cross-modal semantic alignment. We expect that this exploration of multi-view consistent learning also applicable to other VIP tasks.
ACKNOWLEDDMENT
This work was supported in part by National Key R&D Program of China under Grant No. 20.200AAAA00097000, National Natral Science Foundation of China (NSFC) under Grant No.61976233, Guangdong Outstanding Youth Fund (Grant No.2021B151 15121166)),,,) Shenzhen Fundamental Research Program (Project No.RCYX202007.44 1400 110200071411442083.. 14642083, No.JCY100190007 1080715522 1365). We thank MindSpore and CAAl-Huawei MindSpore Open Fund for the partial support of this work, which a new deep learning computing framwork?
hhtps:/wwww.middsoo..o

---

Published as a conference paper at ICLR 2023
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2425-2433, 2015.
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint vrXiv:2/06.08254, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3566 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pp. 104-120. Springer, 2020b.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell., 24:603-619, 2002.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Icee, 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11162-1119 1173, 2021.
Xiaoyi Dong. Yinglin Zheng, Jianmin Bao, Ting Zhang. Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked self-distillation advances con- trastive language-image pretraining. arXiv preprint ar0iv: 12262. 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88 (2):303-338, 2010.
Golnaz Ghiasi, Xiuye Gu Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint 1rXiv:2112.121444 2021.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020a.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020b.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2023
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision Vision-and-a and-language transformer without convo- lution or region supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11336-11344 2020a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafig Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-9700 2021a.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive anggaaee-mmge pre- training paradigm. arXiv preprint arXiv:2110.05200 2021b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Ilya Loshehilov and Frank Hutter. Sgdr: Stochastic gradient descent with restarts. Learning, 10:3, 2016.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Scong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 891 991, 2014.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112.12750, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037444 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Pengzhen Ren, Changlin Li, Guangrun Wang, Yun Xiao, Qing Du, Xiaodan Liang, and Xiaojun Chang. Beyond fixation: Dynamic window visual transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11987-11997, 2022.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 731-737, 1997.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec The new data multimedia research. Communications of the ACM, 59(2):64-7,, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supervipe visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033 2021a.

---

Published as a conference paper at ICLR 2023
Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei. Distilled dual- encoder model for vision-language understanding. arXiv preprint arxiv:2112.08723, 2021b.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text 2019.
classification tasks, arXiv preprint
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134-18144 2022.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. sim- ple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arXiv preprint 12 14757, 2021.
Nir Zabari and Yedid Hoshen. Semantic segmentation in in-the-wild without seeing any segmentation examples. arXiv preprint arXiv:2112.03183. 2021.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6720-6731, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. arXiv preprint arXiv:2112.01071, 2021a.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2021b.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 34, pp. 13041-13049, 2020.
12


Reviews:
[{"review_id": "xteU9GCARjj", "reviewer": "Reviewer_kMrx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve text-supervised segmentation model i.e., GroupVIT by enforcing multi-view semantic consistency. Specifically, a text-to-views consistency module is included to encourage different views to be aligned with the same text input. In addition, there is a cross-view consistency module which enforces features from the different views of the same input image to be closer. The authors conduct extensive experiments on the semantic segmentation benchmarks and show SOTA results.", "summary_of_the_review": "Overall, I think this is a good paper with sufficient novelty and good results. I would recommend to accept it.", "strength_and_weaknesses": "Strength\n\n-The idea of multi-view consistency is new in visual-language learning. This paper proposes a new method to implement the idea. \n\n-This paper outperforms GroupVIT consistently on multiple benchmarks.  \n\nWeakness\n\n-A potential related work is missing. The DenseCL [A] paper studies multi-view consistency for self-supervised contrastive learning. The author should discuss the difference.\n\n[A] Dense Contrastive Learning for Self-Supervised Visual Pre-Training. Want et al., CVPR'21", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IA1VYKg434", "reviewer": "Reviewer_wv76", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies text-supervised semantic segmentation. The authors propose multi-View Consistent learning (ViewCo) to introduce the correspondence among multiple augmented views of the same image. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, it is a solid paper and gives a reasonable idea to improve text-supervised semantic segmentation.", "strength_and_weaknesses": "Strength\n+ Compared to the baselines and other methods, the proposed methods could bring some improvements in accuracy.\n+ Some ablation experiments are introduced to prove the effectiveness of the proposed method.\n\nWeaknesses\n-  The definition of positive pairs and negative pairs in the cross-view consistency module is missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8Ov6dQDBIp", "reviewer": "Reviewer_Dedj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on text-supervised semantic segmentation by studying a multi-view consistency learning framework. Specifically, a text-to-views consistency and a cross-view segmentation consistency training strategy are proposed and added upon existing work Group VIT. The proposed method tries to construct more training constrains based on correspondence among multiple augmented views to obtain more improvements. Experiments are conducted on multiple datasets for zero-shot segmentation, e.g. PASCAL and COCO., showing improvements upon some existing baselines. \n", "summary_of_the_review": "My current rating is based on concerns for limited technical contributions, unreasonable positive/negative pair selection for cross views, quite limited improvments upon baseline GroupViT. Please find more details in the Strength And Weaknesses section.", "strength_and_weaknesses": "\nStrength:\n\n+ The motivation of introducing multi-view consistency constrain to text-supervised semantic segmentation is reasonable.\n\n+ Experimental reuslts on multiple datasets show some benifits compared with existing methods. \n\n\nConcerns: \n\n- The general idea of introducing multi-view consistency constrains to vision-language pretraining models is not new. Some existing works, e.g. DeCLIP [a] have explored adding such multi-view self-supervision constrains and validated its effectiveness when learning vision-language models. The differences here are mainly about applying multi-view constrains on segment level rather than whole image when building upon the framework of Group VIT.\n\n[a] Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, ICLR 2022\n\n-  Cross-view constrains are applied on segments of two augmented views. Since there is no guarantee for location (segments) correspondence between these two augmented views, the proposed strategy in Figure 5 of constructing positive and negative pairs for the corss-view segments seems not reasonable. \n\n- The performance of the most important baseline GroupViT seems not consistent with the original paper. For example, in Table 1 Comparison with zero-shot baselines on PASCAL VOC 2012, it reports 51.2 mIoU for GroupViT and 52.4 mIoU for the proposed method ViewCo, while in the original paper [b] GroupViT obtains 52.3 mIoU, which is much higher than 51.2 reported in this submission and is only 0.1 lower than the proposed method. Besdies, on PASCAL Context, GroupViT reports 22.4 mIoU, the improvements from ViewCo seems also marginal (23.0 mIoU v.s. 22.4 mIoU). \n\n[b] GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0kwRZ6hZ41", "reviewer": "Reviewer_cqK1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves Group VIT, a text-supervised segmentation model, by introducing region-wise cross-view consistent regularization. The author benchmarked for zero-shot segmentation capability across multiple datasets and showed improvements over several baselines. \n\nMy primary concern with this work is selecting the positive segmentation across views, where the author assumes the segments with the same id form positive pairs. I think this assumption is wrong, and I'll expand the discussion to the Weaknesses section. \n\nThis work's contribution is also limited since most of its success is built upon the Group VIT model. The quantitative improvements from the newly introduced regularization are not substantial, around 1-2% over Group VIT.   \n\n\n\n\n\n", "summary_of_the_review": "Given my concerns over the assumption of the objectives and the limited improvement over the baseline, I tend to reject this submission. \n\nI'll reconsider my rate if the author can justify why using segment id for cross-view consistency makes sense or provide experimental results to show that segment ids indicate semantic categories rather than spatial bases. ", "strength_and_weaknesses": "### Strength:\n\n(1) This work improves Group VIT by enforcing view consistency\n\n(2) Author conducts substantial experiments across multiple datasets and shows consistent quantitative improvement over the baseline. \n\n### Weaknesses:\n\n(1)  I think the assumption of cross-view segment consistency by segment id is wrong. As observed in other VIT frameworks, which use learnable tokens for instance segmentation or objective detection, the tokens usually have spatial rather than semantic biases. Based on that, the segment id primarily indicates the segmentation's relative spatial location to the image, and matching group ids does not make sense (we can simply horizontally and vertically flip the images to break the assignment).  \n\nInstead of directly matching segment ids, I think reasonable alternatives are: (a) Compute bipartite matching based on group features similarity and (b) Spatially warp one view to another and compute the segmentation overlapping to match the segments. Then ignore the segment with zero overlapped pixels.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster