CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Authors: Zicheng Zhang
Yi Zhu
Jianzhuang Liu
Xiaodan Liang
Wei Ke
Venue: NeurIPS 2022 Accept (2022)
Keywords: Referring Image Segmentation
Vision Language Modeling
Cross-model Alignment

Abstract: Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the same object, leading to incomplete masks and localization errors in predictions. To tackle this problem, we propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word-pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of linguistic and pixel-level features in intermediate layers of the vision and language encoders. Based on the word-pixel aligned embedding, a set of mask proposals are generated to hypothesize possible objects. Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e.g., RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of-the-art methods, e.g., about 2% oIoU increase on the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CoupAlign.

CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Zicheng Zhang!*
Yi Zhu 2+ Jianzhuang Liu
Xiaodan Liang?
Ke Wei
Xi'an an Jiaotong University
Noah's 's Ark Lab, Huawei Technologies
Sun Yat-sen University
77c1999 stu.xjtu.edu.cn
1yhuyi36,,hujjanzzuuag huawei.com xdliang328 @mail.cmm foi.ke@xjt.edu..
Abstract
Referring image segmentation aims at localizing all pixels the visual objects described by natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the object, leading to incomplete masks and localization errors in predictions. To tackle this problem, propose CoupAlign, simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word- pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Aligmment (WPA) module performs early fusion of linguistic and pixel-level features intermediate layers of the vision and language encoders. Based on the word wideli aligned embedding, set of mask proposals are generated to hypothesize possible objects. Alignment (SMA) module, the masks are weighted
Then in the Sentence-Man by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e.g.. RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of -tate-of-the-art methods, e.g., about 2% oloU increase the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class. Code will be available at https:/ //gg/giteeee/ com/mim///////oc com/minnsspore models tree/naster/ master/ research/ wwtpcc/cccccceeeeccoeeeeeeeeeeet cv/CoupAlinn.
Introduction
Taking an image and a natural language sentence as input, a referring image segmentation (RIS) model is required predict mask for the object described by the sentence. Beyond semantic segmentation which assigns each pixel in an image with a label from a fixed word set, referring image segmentation needs to localize the object pixels indicated by the language expression, which is of arbitrary length and involves an open-world vocabulary, such as object names, attributes, positions,
Equual contribution. Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

X
X
Segmentation Head
Segmentation Head
Segmentation Head
Alignment
Cross-Modal Alignment
Sentence- Mask Alignment
0
Image Encoder
Text Encoder
Image Encoder
Text Encoder
Text Encoder
Word pixe alignment
Image Encoder
chocolate dessert directly front of us in the center"
'choolltte dessert directly in front us in the center"
"chocolate dessert
directly front of in the center"
(a) Late Fusion
(b) Early Fusion
CoupAlign
Figure I: Illustration of the cross-modal alignment (Late/Early Fusion) in existing RIS methods and our CoupAlign that enables hierarchical alignment word-pixel and sentence-men mask levels.
etc. Therefore, this task is the foundation of many applications in the real world, such as image editing [26, 34], autonomous driving [4, 38), and language-basee basedd human-robot interaction [35, 2].
The main challenge of RIS is how to align the given language expression with visual pixels for highlighting the target. Early works [16, typically follow the paradigm that first extracts visual and language features from the corresponding encoder and then fuses them for language-pixx aligmment. Based on the aligned feature, a decoder employed to predict mask. Recently, Transformer-basse vision-language models [20, 24, 22] have shown great potential in aligning visual and language features from fine coarse levels, c.g., pixel, concept, region and image from visual view, and e.g.. word, phrase, sentence, paragraph from language perspective. Inspired by this, extensive works [17, 7, 41, 11, 44] have been investigated to take the advantage of the remarkable cross-modal alignment ability of Transformer architectures for aligning language with pixels more effectively. Some works [7, 41] seek to improve the cross-modal alignment of the encoded vision and language features, as shown in Fig. (a)) Their cross-modal alignment mainly occurs at the sentence-pise entenl level. Furthermore, other works [11, 1, 44] fuse the language information into the pixel embeddings at early stages of the image encoder, as shown in Fig. (bb) The alignment happens more earlier than before, starting from the word-pixel level which is more fine- mne-grained.
Though effective, in Fig. ((a) and (b), we can also observe many undesirable pixel fragments and localization errors in their failure cases. The reason lies behind these issues may be that the pixel-level alignment (e.g., word-pixe and sentence-pixx -ixel level) connects each pixel embedding to language independently, and ignores the visual and semantic consistency among pixels in an object. Therefore, the predicted masks may miss some target pixels or spill over other objects. To deal with this problem, we propose to regard the pixels of an object as whole and explicitly introduce object mask constraints for the cross-modal alignment, at both the fine-grained and the coarse-graines levels.
Specifically, in this paper, we propose CoupAlign that couples entence-mask alignment with word- pixel alignment to enforce the object mask constraint for improving the localization and segmentation performance of RIS. The Word-Pixel Alignment (WPA) module enables cross-modal interaction at the intermediate layers of the vision and language encoders. Then a set of mask proposals are calculated based on the word- word-pixel aligned features to indicate the possible objects in an image. The Sentence-Mash Alignment (SMA) module learns to weight the masks using the sentence embedding localize the referred object, and the weighted masks are projected back to provide mask signals for effectively aggregating the pixels of the target. To further enhance the word-pixel and sentence-mas alignments, we design an auxiliary loss that forces the pixels of the target get close and the pixels not belonging the target to get away. Our CoupAlign captures the pixel coherence at both visual

---

and semantic levels, and bierarchically aligns the pixels and masks with language information. By doing this, CoupAlign generates more accurate mask predictions for RIS.
To validate the effectiveness of our method, we conduct extensive experiments on popular datasets (RefCOCO [18] and G-ref [32]). is shown that CoupAlign outperforms state-of-the-art RIS methods by significant margins, e.g, about 2% oloU increase on the validation and testing sets of RefCOCO. Besides, Coip shows remarkable ability in distinguishing the target from multiple (even greater than twenty) objects of the same class, for example, localizing one person in a team photo using a language query like "middle row second kid from right".
The main contributions of this paper are summarized as follows:
We tackle the challenging problem of RIS via integrating the mask-aware constraint into the pixel-level semantic alignment for capturing the pixel coherence within objects.
We propose CoupAlign, simple yet effective alignment framework, to couple sentence- mask alignment with word-pixel alignment to hierarchically align pixels and masks with linguistic information.
We conduct comprehensive experiments to demonstrate that our CoupAlign predicts more accurate object locations and masks and significantly outperforms state-of-the-art RIS methods on popular benchmarks.
2
Related Work
Referring Image Segmentation aims to segment the objects referred by natural language expressions. RIS Methods can be summarized into two categories. One focuses on cross-model alignment and representation learning [16, 25, 11]. Another views this problem as a language-conditionn reasoning [43, 14], which utilizes a dependency parsing tree to iteratively select the proposal regions.
visual
Due to the breakthrough of Transformer [8] in the computer vision field and its remarkable fusion power for multi-modality, many works [17, 7, 44] begin to explore the cross- cross-model alignment based on Transformer. MDETR [17] shows that simply concatenating vision and language features and then feeding them into the Transformer encoder and decoder can achieve surprising results in many lasoun-an downstream tasks. VLT [7] the first work to introduce the Transformer architecture to referring segmentation. It designs query generation module to utilize the contextual information of images to enrich language expressions and improve robustness. EFN [11] and LAVT [44] insert language-aware attention modules into the image encoding process to achieve early fusion of cross- modal features. Mail. [25] abandons the dual-stream encoders and adopts an unified cross-model Transformer to achieve word-pixe alignment. CRIS [41] focuses how to transfer CLIP's [36] strong image-text aligmment ability to benefit sentence- pixel alignment required by RIS. CRIS uses a Transformer decoder with word queries and sentence-pixe contrastive loss to achieve this goal.
The above mentioned RIS methods mainly focus on aligning each pixel to word or sentence indepen- dently while ignoring the pixel coherence of the target object, thus resulting in inaccurate masks and localization, contrast, our method couples centence-mask alignment with word-pixel aligmment to hierarchically align pixels and masks with linguistic information, and improving the segmentation and localization accuracy.
Vision Language Modeling targets at seeking an unified representation for the two modalities. From the aspect of architecture, vision language models can be divided into single-stream and dual-stream architecture structures. Single-stream models such as VisualBERT [22], Uniter [3], Uniencoder- vi [21] and Oscar [24]. utilizes an unified encoder based on self-attention to fuse the vision and language embeddings. For dual-stream models such as CLIP [36], ALIGN [15], FILIP [45] and GLIP [23], they adopt two unimodality encoders to encode vision and language embeddings and utilizes the dot product to align the outputs. Others like ViLBERT [29] and LXMERT [37] uses two self-attention-bant based encoders to model intra-modality interactions and employ cross-attention to model modas-modality interactions. Inspired by GLIP [23], we adopt a hybrid encoder design, where dual encoders are used to extract vision and language features and the word-pixel alignment realized at the intermediate stages of the encoders.
3

---

CoupAlign Encoder
CoupAlign Decoder
Image Encoder
Seg Loss
Cross- Cross-Atentn Attention
Segfteu
%
WPA
WPA
queries
Ow SMA
Qa
The closest girl on right
Generator
Aux Loss
Language Encoder
Word- Pixel Alignment
Sentence-Mask Alignment
Auxiliary
G1 Mask
Negative
Positive
cee
pull
pull
Average
Average
The closest gel
right
The diosest girl right
Negative
Figure 2: Architecture of our CoupAlign. Given an image and language sentence, the image features V. can be extracted from the image encoder, and the word-level features Lo and sentence feature Lg are extracted from the language encoder. The Word-Pixel Alignment (WPA) module enables cross-modal interactions at each encoder stage. Based on the word-pixel aligned features S., the mask generator produces N mask embeddings Q.. The Sentence-Mase Alignment (SMA) weights Qo using by and projects the mask signals back to YN, and finally generates the prediction M.
3 Methodology
3.1
Overview
The overall architecture of our CoupAlign is shown in Fig. 2. is designed following the encoder- decoder paradigm. The CoupAlign encoder consists of an image encoder fo and language encoder fi for extracting vision and language embeddings, Word-Pis -ixeii Alignment (WPA) module that enables the cross-modal interaction at the intermediate layers the visual and language encoders, and cross attention module for modal fusion after encoders. The CoupAlign decoder includes a mask generator module that produces segment proposals, a segmentation head (SegHead) that upsamples the pixel-level embeddings. and a Sentence-Mast Alignment module (SMA) to weight the masks via the sentence and then project back to the pixel-level embeddings to enforce object mask constraint. Together with the segmentation loss, auxiliary loss that compares foreground pixels with background ones to enhance the hierarchical aligning at word-pixel and sentence-mask levels.
3.2 CoupAlign Encoder
Image Encoder. Given an image / IR Hx333 we adopt Swin Transformer [27] as the image encoder backbone to extract multi-level visual features. We utilize the features from the 1st-4th stages of Swin Transformer, denoted {V, IR [.HxW. 14 Jim1 where C, H4, W, are the number of channels, the height, and the width of the feature map at the i-th stage, respectively.
language encoder backbone. Given a referring expression (w; where T is the length of the expression, we first use the BERT tokenizer based on WordPiece embeddings [42] to get the word embeddings E E TxD where D is the hidden feature dimensionality of BERT. Then the word embeddings E are fed into the hidden layers BERT to extract language features. order to make word-pixel alignment with vision features from Swin Transformer, can have the same stage number of language features as the vision features, which are denoted as {L, Tx P)4 Ji=1. We also utilize the output [CLS] token as the global semantic representation of the referring expression, which is denoted 4, 1
Language Encoder. We BERT [6] with 12 layers
aggregate every 3 hidden layers into stage, that

---

Word-Pixel Alignment (WPA). For segmentation task, we need to obtain fine tieeggriiedd segmentation mask, which means that low-level features like words in sentences and edges and shapes of objects in images are crucial for finding correct segments. The modsl alignment at early encoder stage proved to be effective for referring image segmentation [7, 44]. Their early alignments are in single-direction of language to vision. In contrast, our WPA module is implemented with bidirectional cross attention to achieve both language-to-vision and vision-to-language alignment, as:
Vi = BiAttn(V, (,, L.). i (1,...,,
Vi Gate(V/), L Gate(L)),,
(2)
V4+1 SwinStage, + L.), 1 BERTStage, V/);
where SwinStage,() represents the Stage i defined in Swin Transformer. (ERTStage;;) represents the Stage of BERT. The cross-modality interaction is achieved by the bidirectional cross attention (BiAttn), which computes the context vectors of one modality by attending to the other modality:
V. = V.W,, L, = L.W. Attn; = V.L. /vd,
(4)
L Noftmax(Attm,))LI)))
V softmax(Attm) Vat V.W""
(5)
where d is the dimension of the joint embedding space, and W." E Roxd W E Rxd, W
paxC, Wy E Dd are all projection matrices.
In order to prevent the fused features V/ and LA from overwhelming the original signals VÚ and L; respectively. We design Gate network [44] to control the fused information flow:
(F)) = MLP(FF) @ F.,
(6)
where F; is the fused feature from BiAttn, @ indicates clement-wise multiplication, and MLP is a two-layer perceptron with the first layer being linear layer followed by the ReLU activation [33] and the second layer being a linear layer followed by a hyperbolic tangent activation.
Cross Attention. To better capture high-level semantics and generate the pixel-Ievel fused features, we use a plain multi-head attention layer [40] to fuse the high-level vision and language features from the image and language encoders. As illustrated in Fig. 2, first, we obtain the high-level vision features Vo E H. x W. XC. from the output of the image encoder and the high-level language features L. E TX from the language encoder. Then we map the vision and language features to R. After that we concatenate them to produce F, E [.H.W. WTT) which is fed into the cross attention layer. We add learnable positional emdeddings Ep E H.H.....X. to the vision features before the concatenation. Finally the output of the cross attention layer is added with the original input vision features to produce a resolution feature map S, E R @.x.W.x....
as:
V/ = VoW + c L'= = W.!
(7)
5. = CrossAttn (F.) + Va, F. = VV' L'),
(8)
where W E RCxDD and W DDDD are projection matrices and : denotes concatenation.
3.3 CoupAlign Decoder
Mask Generator. To provide object mask constraints for the word-pixel alignment, we first design a mask generator that produces N segment proposals based on the output feature of the CoupAlign encoder. The generator is built upon a 6-layer transformer decoder. It takes S, E R H. XW. xC. and
N learnable queries Q Nxd, as input, where d, is the hidden dimensionality of the transformer decoder, and outputs the hidden feature Q. E NRd, of the N segment tokens, as:
Q. MaskGenerator((,.5 S.)).
Segmetation Head (SegHead). To upsample the pixel-level features for generating final segmentation maps, we construct a segmentation head that takes the CoupAlign encoder output feature S, and the multi-level visual features {V input, and obtains output features as:
Ys So,
(10)
Y. = Up(p(Y(+1) + 7(V), 4,3, 2,
5

---

where p is a two-layer convolution network where each layer performs a 3 x3 convolution followed by ReLU and batch normalization, Up denotes 2x bilinear upsampling, Y is 1 xx 1 convolution that maps the vision features to RR We take Y1 E * xd, as the output of SegHead.
Sentence-Mask Alignment (SMA). After generating mask proposals and obtaining upsampled pixel-level embeddings, the sentence- mask alignment assigns each mask a score according to the language query and then project the weighted masks back to the pixel-level feature maps to introduce object mask constraint for word-pixel alignment. At first, We project Qo and YI to Qo E R and V1 R x D via learnable weights W RA,x and W RR.x We the mask embeddings Q. and the sentence representation Tg generated from the language encoder compute attention weight Q E RRXX, Higher weight in Qu indicate that the corresponding mask is most likely contain the object referred by the language. Then, Q. and Yi are multiplied to get N mask predictions YN E R x 4 Finally, YN multiplied by Qu to generate the final mask prediction M E R x which further upsampled to the input image size. This procedure is represented as:
Qu w voftmax(siml) (Ly 2o)), M = YN 8 Qwi YN = 8
(11)
(12)
where sim(-) is cosine similarity and @ denotes the broadcast tensor multiplication.
3.4 Loss Functions
Segmentation Loss. In the context of referring image segmentation, each pixel in the image / is discriminated into background or foreground. Thus this task can be viewed as a pixel-wise binary classification problem. Specifically, we use bilinear interpolation to upsample M to the original image size, obtaining M'. Let m and m; be the values of pixel i from M' and M (ground truth mask) respectively. Then the segmentation loss are implemented via a binary cross entropy loss as:
HxW
Seg
Σ log(o((m)) + m.) log(1 - a(m))),
(13)
H W
where a denotes sigmoid function, and j is the image index in a training batch.
Auxiliary Loss. The word-pixel and sentence-mask alignment collaborate in hierarchically learning manner to enforce object mask constraint for improving referring image segmentation. We adopt pixel-level contrastive loss as auxiliary to further enhance the learning of the two alignments and enable the aligning consistency at both the fine-graine fine-grained and searse-grained level. This loss forces the pixels within the ground truth masks to get closer, and the rest pixels to get away. Therefore the foreground pixels are aggregated together to generate more accurate predictions. Specifically, we resize the ground truth mask M to the same shape as the segmentation map Y,. Let ys be the vector y, in pixel i. Then we use the resized mask to divide positive and negative samples each image. For pixel i, if m, is O, then y/ belongs to the negative set N and is denoted as y, otherwise y: belongs to the positive set P and is denoted as v,* Then we average the negative vectors and the positive vectors to get the negative prototype ğ and the positive prototype gt. Afterward we adopt InfoNCE [39] to compute the contrastive loss, as:
1 P
exply, 5 /T)
L =
(14)
exp(yi 8+ /T) + En; EN exp(yt Vy VE/TT)
Nr MtEP
exply, 1 /T)
L L2P
Σ
(15)
W
exply; -0 /T) Σn; EP exply; Vt /T)
v7 ETEN
I L
L A + N2P.
(16)
where 7 is the temperature parameter. Finally, the total loss for the training is that:
B Σ
1 + ALGUN),
(17)
j=1
where B is the batch size, A is parameer.

---

RefCOCO
RefCOCO+
G-Ref
ReferIt
Backbone
val
test A
test B
val
testA
TestB
val (U)
test (U)
test
MCN [3]]
Darknet-53
62.44
64.20
59.71
50,62
54.99
44.69
49.22
49.40
BRINet [12]
ResNet-101
60.98
62.99
59.21
48.17
52.32
42.11
63.46
CMPC [13]
ResNet-101
61.36
64.53
59.64
49.56
53.44
43.23
65.53
LSCM [14]
ResNet-101
61.47
64.99
59.55
49.34
53.12
43,50
66.57
CGAN [30]
ResNet-101
64.86
68.04
62.07
51.03
55,51
44.06
51,01
$1.69
BUSNet [43]
ResNet-101
63.27
66.41
61.39
51.76
56.87
44.13
EFN [1]
ResNet-101
62.76
65.69
59.67
51.50
55.24
43.01
66.70
LTS [16]
DarkNet-53
65.43
67.76
63.08
54.21
58.32
48.02
54.40
54.25
[7T
DarkNet-53
65.65
68.29
62.73
55.50
59.20
49.36
52,99
56.65
ReSTR [19]
ViT-B-16
67.22
69.30
64.45
55.78
60.44
48.27
54.48
70.18
CRIS [41]
ResNet-101
70.47
73.18
66.
62.27
68.08
53.68
59.87
60.36
LAVT [44]
Swin-B
72.73
75.82
68.79
14
68.38
55.10
61.24
62.09
CoupAlign (ours)
Swin-B
74.70
77.76
70.58
62,92
68.34
56.69
62.84
62.22
73.28
Table 1: Comparison with SOTA methods on the oloU metric on RIS datasets.
4 Experiments
4.1
Settings
Dataset. We evaluate our method on four widely-used datasets: RefCOCO [46], RefCOCO+[46),
G-Ref [32] and Referlt [18]. RefCOCO consists of 142,209 referring expressions for 50,000 objects in 19,994 images and RefCOCO+ has 141,564 referring expressions for 49, 49,856 objects in 19,992 images. G-Ref was collected Amazon Mechanical Turk in non-interactive setting which consists of 85,474 referring expressions for 54,822 objects in 26,711 1 images. ReferIt has 130,525 referring expressions 19,894 images which are collected from IAPR TC-12 [9]. RefCOCO+ disallowed from using location words in referring expressions. The languages used in RefCOCO and RefCOCO+ tend to be more concise and less flowery than the languages used in the G-Ref. The expressions in G-Ref tend to be longer with an average of 8.43 words, which makes it more challenging compared with RefCOCO and RefCOCO+. Expressions Referlt are usually shorter than the other datasets.
Metrics. We adopt three common used metrics in referring image segmentation: the overall intersection-over-umn
(oIoU), the mean intersection-over-uiii measures as the ratio between the total intersection area and the total union area of all test samples. By default, we use this metric to compare CoupAlign with other works. The mloU the average of the IoU over all test samples. prec@ X measures the percentage of test images with an IoU score higher than threshold e, where E {0.5, 0.7, 0.9).
(mloU) and prec @ X. The oloU
Implementation Details. We implement our model using the MindSpore Lite tool [1]. We choose BERT BASE [6] with 12 layers and hidden size 768 as language encoder and use the official pre- trained weights. The image encoder layers are initialized with the classification weights pre-trained on ImageNet-22K [5]. The number of queries of the mask generator N is 100. The rest of the weights in our model are randomly initialized. We adopt Adam [28] optimizer with weight decay 0.01. We adopt the polynomial learning rate decay schedule and set the initial learning rate to 3e-5, the end learning rate to 1.5e-5 and the max decay epoch to 25. Our model is trained for 50 epochs with batch size 16. The images are resized to 448 x 448 without specific data augmentation and the maximum sentence length is set to 30. The weight of the auxiliary loss À is set to 0.1.
4.2 Comparison with State-of-the-Arts
Results on RefcOco. In Tab. 1, we compare CoupAlign with state-of-the-art (SOTA) RIS methods by oloU metric. The language expressions provided by RefCOCO contain many words of positional relationships, e.g.. "The closet girl on the right", which requires the RIS models to not only capture the pixel coherence within objects. but also the spatial relationship among multiple objects of the same class. Compared with the SOTA method LAVT [44], CoupAlign achieves better performance with absolute oloU increases of 1.97%, 1.94% and 1.79% the validation, testA, and testB sets of RefCOCO, respectively.
Results on RefCOCO+. As shown in the Tab. 1, the performance of CoupAlign on RefCOCO+ comparable to or better than LAVT which the most recent state-of-the-ar method (published in CVPR2022). The improvements on RefCOCO+ are less significant than those on G-Ref and
7

---

CoupAlign
Ground Truth
CoupAliga
Ground Truth
CoupAlign
Ground Truth
(a) right
(b) "left baby elephant face towards us"
"white hear right lower one"
(d) "bear nexl to empply spol on ocnler scal"
(c) "glass close to sposn right"
(1) "Iup carrol in bag" F Failure
Success
Figure 3: Visualization examples of the success cases and failure cases of CoupAlign.
Method
JoU Range
[0.0.1)
(0.1.0.2)
[0.2,0.3)
(0.3.0.4)
(0.4...5)
Num of mask
20
mloU
74.77
pree@00.5
85,46
proc
76.22
prec@00.9
30.91
LAVT (44]
330
43
64
118
50
74.98
85.67
76.20
31.88
CoupAliga (ours)
271
47
44
100
75.49
86.40
77.59
32.40
Table 2: Analysis of localization ability.
Table Ablation of different mask numbers.
RefCOCO, This is because the language captions RefCOCO+ hardly contain the descriptions of the relative absolute spatial locations (e.g., "closest" or "right bottom") of the target object in images, and the effectiveness of our method mainly lies the ability of localizing objects in such challenging scenarios, e.g., "middle second kid from right" in Fig. 3.
Results on G-Ref. The language annotations of G-Ref have more complex language structure with more words, e.g., "chocolate dessert directly in front of us in the center", which requires more fine-detailed word-pixel alignment, more discriminative sentence-mas alignment, and more comprehensive understanding of the sentence. As shown in Tab. 1 on G-Ref (UMD partition), our CoupAlign outperforms LAVT by 1.6% and 0313% on the validation and test sets, respectively. The consistent improvements on all the evaluated datasets demonstrate the capability of our CoupAlign in modeling hierarchical cross-modal alignment at pixel and mask levels.
Analysis of Localization Ability. If the IoU between a mask prediction and the Ground Truth mask smaller than 0.5, we consider the prediction as failure case for localization. In Tab. 2. we can see that our CoupAlign has fewer failure cases than LAVT in most IoU ranges. Especially, in the IoU range of [0,0.1), the number of failure cases of CoupAlign decreases the most, which further validates the remarkable ability of CoupAlign.
Mask Number N. From Tab. 3, we find that the mask number N in mask generator is correlated to the model performance. To balance the computational cost and performance, N is set to 100.
4.3 Visualization
Success cases and failure cases. In Fig. 3, we show some representative visualization examples of success predictions and failure cases of CoupAlign. The examples are from the RefCOCO val. set. We can see that CoupAlign works well in the scenes where crowded objects have similar color and context and stay close each other. However, if the boundary of the object is blurred, or there exists occlusion, the segmentation prediction is roughly accurate, but not precise enough.
Word-Pixel and Sentence-Mask Alignment. To validate whether WPA and SMA modules can perform accurate and consistent alignments, we visualize the intermediate attention maps of the aligmment modules in Fig. It can be observed that our WPA module accurately highlights the most relevant pixels corresponding to the word. Note that the vocabulary of RIS language annotations is very large compared the small fixed label set of semantic segmentation. Surprisingly, our CoupAlign can not only capture the semantic meaning of words in various types, e.g., nouns, positions, and attributes, but also distinguish synonyms, e.g., child, man, and lady, which are roughly defined as "person" in many semantic segmentation datasets like PASCAL VOC [10]. As for the SMA module, we show three masks with their attention weights ranked in descending order. The mask with the highest weight is the most accurate one and the masks with lower weights are less accurate. Fig. 4(right) demonstrates that our SMA module provides the effective mask constraint for benefiting the hierarchical aligning at pixel and mask levels.

---

Word- pisel alignment
Sentence mask alignment
Input Image
child blue shirt
man on right
man on right
man rightt
high
stzention wedg
low
lady white dress
lady white dress
Iody white drress
high
attention weight
Figure 4: Visualization examples our word-pixel alignment and sentence- alignment.
Bi-WPA Uni-WPA
SMA
Aux Loss
oloU
mIoU
prec@0.5
prec@0.7
prec@0,9
70.43
69.61
80.03
68.91
27.47
72.70
73.44
84.07
72.60
29.02
73.02
73.85
84.86
74.62
29.25
73.70
74.21
85.32
75.31
30.14
74.70
75.49
86.40
77.59
32.40
Table 4: Ablation studies of our WPA module, SMA module, and auxiliary loss.
WPA's position
stage 1-4
stage 1,2
stage 3,4
stage 4
stage 3
stage 2
stage
oloU
74.70
72.74
73.93
73.61
72.53
72.63
72.59
mloU
75.49
73.87
74.88
74.68
73.47
73.48
73.97
Table 5: Ablation study of WPA modules' number and position.
4.4 Ablation Study
Effect of WPA Types. Our WPA module enables cross-modal interaction at both low-level and high- level encoding stages. Tab. 4, Bi-WPA indicates our WPA which is implemented via bidirectional cross attention, Uni-WPA represent unidirectional attention only from language image features, as in LAVT [44]. The performance dramatically drops by 4.3% (74.70% vs. 70.43%) on oloU when WPA is removed. And also when replace our Bi-WPA with Uni WPA, the oloU result drops by 2% (74.70% vs. 72.70%). These results demonstrate the effectiveness our WPA module.
Effect of WPA's Number and Position. In our experiment, we used four WPA modules, two of which are in the early encoding stage and the other two are in the late encoding stage. To study the effects of the numbers of WPA modules, we first conduct two baseline models that alternatively remove two WPA at early or late encoding stages. As shown the Tab. 5, when we remove the last two WPA modules the performance drops about 2%, and when remove the first two WPA modules the performance drops about 0.8%. These results validate the effectiveness of WPA modules at both early and late stages and indicate that the latter WPA modules play more important role in our modules. Then, we only use one WPA module, which is inserted at different encoding stages. The WPA module the 4-th stage is more effective than those inserted at other stages.
Effect of SMA. Tab. 4, when SMA removed, the oloU result decreases by 1.7% (74.70% vs. 73.02%) %), since the moos alignment becomes less accurate without mask constraints.
Effect of Auxiliary Loss. From the last columns of Tab. 4, we observe a performance drop of 1 5 174.0%% vs. 73.70%) on oloU when the auxiliary loss is disabled, which verifies the ability of the loss in enhancing the word-pixel and sentence-mast mask alignments.

---

Conclusion
To tackle the practical yet challenging problem of referring image segmentation (RIS), we pro- pose CoupAlign which is simple yet effective cross-modal alignment framework that couples sentence-man alignment with word-pixel alignment to enforce mask-aware constraint for referring segmentation. The word-pixel alignment module enables fine-grained cross-modal interactions at the early encoding stage. The sentence-mask alignment module uses the sentence features to weight the mask proposals generated based the word-pixel aligned features, and then projects the mask constraints back to guide the word-pixel alignment. The auxiliary contrastive loss that compares among pixel pairs to further enhance the hierarchical alignments word-pixel and sentence- mask levels. Benefiting from the collaboration of the alignment learning at different levels, CoupAlign captures both visual and semantic coherence of pixels within the referred object, and significantly outperforms maae-off-he-an RIS methods. Especially, CoupAlign has surprising ability localizing the target from crowded objects of the same class, showing great potential in segmenting natural language referred objects in real-world scenarios.
Acknowledgment
This work was supported in part by National Key Research and Development Project of China under Grant No. 2020AAA0105600, National Natural Science Foundation of China under Grant No. 62006182, and the Fundamental Research Funds for the Central Universities under Grant No. xzy012020017, Guangdong Province Basic and Applied Basic Research (Regional Joint Bund-Key) Grant No. 2019B15151 12003, Guangdong Outstanding Youth Fund (Grant No. 0021B151502006 We gratefully acknowleds Keknoweedge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
10

---

References
[1] Mindspore. https://ww. hwtp://wwwwdwww..ee//// cn/,
[2] H. Ahn, S. Choi, N. Kim, G. Cha, and S. Oh. Interactive text2pickup networks for natural language-base Numan-robo collaboration. IEEE Robotics and Automation Letters, 3(4):3308-333 -3115, 2018. 2018.
[3] Y.-C. Chen, L. Li, L. Yu, Kholy, F. Ahmed, Z. Gan, Y. Cheng. and Liu. Uniter: Learning universal image-text representations. 2019.
[4] F. Codevilla, E. Santana, A. M. López, A. Gaidon. Exploring the limitations behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13299-933, 2019.
[5] Deng, W. Dong, Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[6] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[7] H. Ding, C. Liu, Wang. and X. Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16321-16330, 2021.
[8] A. Dosovitskiy, Beyer, A. Kolesnikov, Weissenbomn, X. Zhai, T. Unterthiner. M. Dehghani, Min- derer, G. Heigold, Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[9] J. Escalante, A. Hernández. A. Gonzalez, López-López. M. Montes, Morales. Sucar, L. Villasenor, and M. Grubinger. The segmented and annotated iapr tc-12 benchmark. Computer vision and image understanding. (14(4):19-442) 2010.
[10] Everingham, (voc) challenge. International Journal of Computer Vision, 88(2):303-338 2010.
Gool, C. K. Williams, J. Winn, 7isserman. pascal visual object classes
[11] G. Feng, Z. Hu, L. Zhang, and H. Encoder fusion network with co-attention embedding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 15506-15515, 2021.
[12] Z. Hu, G. Feng, J. Sun, L. Zhang, and Lu. Bi-directional relationship inferring network referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4424 4433, 2020.
[13] S. Huang, T. Hui, S. Liu, G. Li, Y. Wei, Han, L. Liu, and B. Li. Referring image segmentation via cross-modal progressive comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 10488-10497, 2020.
[14] T. Hui, Liu, S. Huang, Li, S. Yu, Zhang, and J. Han. Linguistic structure guided context modeling for referring image segmentation. In European Conference on Computer Vision, pages 59-75. Springer, 2020.
[15] C. Jia, Y. Yang. Y. Xia, Y-TT. Chen, Z. Parekh, H. Pham, Q. Le, Y.H. Sung. Z. Li, and T. Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMIR, 2021.
[16] Y. Jing, T. Kong, W. Wang. L. Wang. L. Li, and T. Tan. Locate then segment: A strong pipeline for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9858 9867, 2021.
[17] A. Kamath, M. Singh. Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790. 2021.
[18] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), pages 787-798. 2014.
[19] N. Kim, D. Kim, C. Lan, W. Zeng, and S. Kwak. Restr: Convolution-fre referring image segmentation using transformers. arXiv preprint arXiv:2203.16766 2022.

---

[20] W. Kim, B. Son, and I. Kim. Vilt: Vision-and- lsioo-andd-annguae transformer without convolution region supervision. In International Conference Machine Learning, pages 5583-5594 PMLR, 2021.
[21] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang. Unicodet-vi: vt: A universal encoder for vision and language by cross- mross-modal pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336-11 1344, 2020.
[22] L. Li, M. Yatskar, D. Yin. C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 0355,7 2019.
[23] L. H. Li, P. Zhang. H. Zhang, J. Yang. C. Li, Y. Zhong, L. Wang. L. Yuan, L. Zhang, J.-N. Hwang, et al. Grounded language sageggg--gagge pre- pre-training. arXiv preprint arXiv:2112.08857. 2021.
[24] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong. F. Wei, ct al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.
[25] Z. Li, M. Wang, J. Mei, and Y. Liu. Mail: A unified mask-image language trimodal network for referring image segmentation. arXiv preprint arXiv:2111.10477 2021.
[26] H. Ling, K. Kreis, D. Li, W. Kim, A. Torralba, and S. Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34, 2021.
[27] 7. Liu, Y. Lin, Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[28] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:111.05101, 2017.
[29] J. Lu, D. Batra, D. Parikh, and S. Lec. Vilbert: Pretraining task-agnostic visiolinguistic representations for ssioo-and-language tasks. Advances in Neural Information Processing Systems, 32, 2019.
[30] G. Luo, Y. Zhou, R. Ji, X. Sun, J. Su, C.-W. Lin, and Q. Tian. Cascade grouped attention network referring expression segmentation. Proceedings of the 28th ACM International Conference on Multimedia, pages 1274-1288 1282, 2020.
[31] G. Luo, Y. Zhou, X. Sun, Cao, C. Wu, C. Deng, and R. Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 10034-10043, 2020.
[32] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11-20, 2016.
[33] V. Nair and G. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
[34] O. Patashnik, Z. Wu, E. Shechtman, Cohen-Or, and Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[35] S. Pate, W. Xu, Z. Yang, M. Love, S. Ganguri, and L. L. Wong. Natural language for human-robot collaboration: Problems beyond language grounding. arXiv preprint arXiv:2110.0444) 2021.
[36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[37] H. Tan and Bansal. Lxmert: Learning cross pross-modality encoder representations from transformers. arXiv preprint Xiv: 199800079999 2019.
[38] Toromanoff, E. Wirbel, and F. Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Pruceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[39] A. Van den Oord, Y. Li. and O. Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages ar87 2018
12

---

[40] A. Vaswani, N. Shazeer. N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. Advances in Neural Information Processing Systems, 30, 2017.
[41] Z. Wang, Y. Lu, Li, X. Tao, Y. Guo, M. Gong, and T. Liu. Cris: Clip-driven referring image segmentation. arXiv preprint arXiv:2111. 2021.
[42] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey. M. Krikun, Cao. Q. Gao, K. Macherey. et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.081444 2016.
[43] S. Yang, M. Xia, G. Li, H.-Y. Zhou, and Y. Yu. Bottom-up shift and reasoning for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages I 11266-11 1275, 2021.
[44] Z. Yang, Wang, Y. Tang, K. Chen, H. Zhao, and Torr. Lavt: Language-aware vision transformer for referring image segmentation. arXiv preprint ar Xi:iv:11 2021.
[45] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang. Z. Li, X. Jiang, and C. Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2 2021.
[46] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In European Conference Computer Vision, pages 69 85. Springer, 2016.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Please see the Abstract and Introduction sections.
(b) Did you describe the limitations of your work? [Yes] Please the Conclusion section.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see the Conclusion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Our code is enclosed in the supplementary material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Section ?? for more experimental details.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] The experiments have the same configuration as previous works.
conducted under fixed seed, and
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments are conducted on an NVIDIA 3090 GPU.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see the Experiment and Reference sections.
(b) Did you mention the license of the assets? [No] The data and models used in our work publicly released.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The data and models used in our work are publicly released.
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The data used in our work does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "SufrqIMEU-", "reviewer": "Reviewer_gpWt", "summary": "This paper proposes CoupAlign for referring image segmentation, which couples sentence-mask alignment with word-pixel alignment to enforce the mask-aware constraint. Experiments validate the effectiveness of the method.", "questions": "- The precision@0.5/0.7/0.9 values of the 3-rd and 4-th row are the same. Is this a typo error? Or it is just a coincidence.\n- The attention weights on different words are not analyzed in the WPA module. It is better to show if the model correctly attends to the referring motion or appearance words.\n- In table 4, I think there is one experiment missing, i.e., w/ Bi-WPA, w/o Uni-WPA, w/o SMA, w/o Aux Loss.\n- From Figure 2 and Table 4, I think the baseline without SMA is that directly summarizes all mask query embeddings without weights generated from the sentence features. What if evaluating the importance of each mask embedding from themselves? That is, generate the weights from the mask embeddings (e.g., using MLP like the equation (3) in [3]) without the sentence embeddings.\n- Could you please show the diversity of masks generate from mask queries by statistics?\n- The computational cost or runtime should be discussed.\n- Experiments on RefCOCO+ [4] are missing. Why?\n- Why the improvement in the test split of G-Ref is so small (i.e., 0.13%) but significant on Val split (i.e., 1.6%). Does it mean the proposed methods overfit the train and Val splits?\n- The improvements of WPA when inserting it into different stages of the encoders should be discussed. And what if only using the words features from the last stage of the language encoder instead of from the intermediate layers.\n\n[3] SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation, Arxiv, 2021.\n[4] Modeling context in referring expressions, ECCV, 2016.\n", "limitations": "The limitations and potential negative societal impact are discussed in the supplementary files.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is mostly clear and easy to follow.\n- The writing of this paper is good and the figures are appealing.\n\nWeaknesses\n\n- Novelty. The basic idea of the proposed CoupAlign framework is to couple sentence-mask alignment with word-pixel alignment for consistent and accurate segmentation results. However, the proposed Word-Pixel Alignment (WPA) module is just the same as the Language-Aware Deep Fusion proposed in GLIP [1]. The Auxiliary Loss can be regarded as a simple supervised pixel-level contrastive learning loss, which has been demonstrated by recent works [2]. The proposed Sentence-Mask Alignment (SMA) module can be a contribution, but I think it is just a cross-modal version weighted-summation of queries from the DETR-decoder-like mask generator. In conclusion, the novelty of this paper is limited.\n\n[1] Grounded Language-Image Pre-training, CVPR, 2022.\n\n[2] Exploring Cross-Image Pixel Contrast for Semantic Segmentation, ICCV, 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Qmz6SSE-XkK", "reviewer": "Reviewer_Vuxv", "summary": "This paper proposes a new model for referring image segmentation (RIS). The model takes in an image and a natural language description as input, and outputs the segmentation mask corresponding to the language input. The proposed approach features 1) a hierarchical fusion architecture in its encoder to gradually fuse information from the visual and language inputs (dubbed as Word-Pixel Alignment or WPA), and 2) a module to assemble segmentation masks from a set of candidate masks and the language representation (dubbed as Sentence-Mask Alignment or SMA). In terms of experiments, the authors train and test their models for RIS on two datasets -- RefCOCO and G-Ref, and achieved competitive results compared to previous SOTA methods. The authors also carried out ablation studies on various design choices and provided some qualitative visualizations to better understand the model's behavior.  \n", "questions": "- In Eq 5, V'_i is \"assembled\" from the language feature while L'_i is assembled from the visual feature? Should these two be swapped? \n- In Eq 14, is y^-_k sampled from the same image or from some other images? \n- L287, \"when SMA is removed\", how do you remove SMA while still generating the segmentation conditioned on the language input? This requires more elaboration. \n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The task of segmenting images with arbitrary language input is interesting and could be a prominent task in near future. \n+ For presentation, 1) the illustration in Fig. 2 is comprehensive and helpful and 2) the visualization in Fig. 4 is very helpful for readers to better understand the model behavior. \n+ Overall, the proposed model is intuitive and achieves strong performance when compared to other SOTA methods on popular datasets.\n\nWeaknesses\n- Notations are confusing. For example, there are d_k, d_s, d_q, are they equal to each other? In Eq 11, L_g is with shape 1 x D while Q_o is with N x d_q, does this imply D == d_q? Also, in Eq 12, Y_1 is with shape H/4 x W/4 x d_s while Q_o is with size N x d_q, how can you multiply these two (unless d_q == d_s, which I did not see stated anywhere)?   \n- Since both WPA and Cross Attention are fusing the visual and language information, why using different attention computation for these two? The design seems arbitrary and I did not see any justification for the difference. \n- Fig 3, are these examples representative or through cherry picking? Since I did not see why the proposed method would have a qualitatively different behavior compared to previous models (e.g. LAVT) by adding more intermediate level fusion. If this is not representative, I think it's better to not include these as they can be misleading (i.e. to make readers think the proposed model is qualitatively better/different). \n- L281-6, since this is where the main technical novelty lie, it would be better to have more detailed ablations on questions like how many layers of fusion is needed? Or which fusion layer contributes the most? \n\n############# POST REBUTTAL NOTE #############\nThe authors have properly addressed my comments above with the added experiments and paper revisions. With this, I will raise up my rating to accept the paper. \n\n\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PoGEGBimkU", "reviewer": "Reviewer_gikD", "summary": "This paper proposed a cross-modal alignment model (CoupAlign) for referring image segmentation.  The authors propose to use coupled sentence-mask alignments with word-pixel alignment to enforce the model learned more accurate and consistent segmentation masks. Word-pixel alignment serves as an early fusion module that fuses the features between each block of swing transformer feature and BERT feature. The Sentence mask alignment learns to weight the mask using the sentence embedding to localize the referred object. The authors benchmark the proposed method on 2 refer segmentation benchmarks and achieve state-of-the-art performance. ", "questions": "- Is it possible to extend the current framework to incorporate weakly supervised data such as image caption pairs? \n\n- What is the role of WPA module and how many do we need it. What is the good balance between fusion and computing? \n\n- For cross attention, why not use multi-head attention? is there any specific reason? \n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n\n- WPA module serves as an early fusion module that can balance the fusion and compute. \n\n-  The sentence mask alignment (SMA) module is a general module that can scale up to weakly supervised learning such as GLID. \n\n- The proposed model achieves state-of-the-art performance on refer segmentation benchmark.  \n\n- The visualization of word-pixel alignment shows the effectiveness of the module and aux loss. \n\n[Weakness]\n\n- It would be good to conduct an ablation study on how many WPA module is needed in the model. Does the early fusion (first few blocks of the model) need the WPA module or the more WAP module the better?  \n\n- It seems the WPA module use cross-attention similar to [1], so it would be good to compare the difference between different cross-attention operations. Why not use multi-head attention on the WAP module?\n\n[1] Lu, J., Yang, J., Batra, D. and Parikh, D., 2016. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FmnonUmP3hh", "reviewer": "Reviewer_hX8h", "summary": "In order to employ the semantic consistency of pixels within the same object, this paper proposes a CoupAlign mechanism to couple a work-pixel alignment (WPA) and a sentence-mask alignment (SMA). WPA fuses the linguistic and pixel-level features within the intermediate layers of the feature encoders. SMA weights the generated masks to localize the referred object. In addition, the authors provide an auxiliary contrastive loss for facilitating the segmentation accuracy. The experiments show that the proposed CoupAlign mechanism achieves state-of-the-art referring image segmentation performance.", "questions": "Though the proposed CoupAlign mechanism coupling WPA and SMA is interesting, the manuscript lacks some clear terminology definitions, as mentioned in [Weaknesses], and the experiments are not supported by sufficient standard evaluating datasets. I would like to see more information in the authors’ response.", "limitations": " The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths] \n+ The idea of implementing the CoupAlign mechanism is interesting.\n+ The manuscript is well organized, and most paragraphs are easy to follow.\n+ The references are adequate.\n\n[Weaknesses]\n- The experiment misses evaluating standard datasets, ReferIt and RefCOCO+, for comparison with other RIS methods. Such an incomplete comparison makes the evaluating experiments weak.\n- Missing some function descriptions harms the readers to reimplement the proposed model. For example, what are the functions of SwinStage and BERTStage in (3)? It is unclear why these functions relate to two different stages. Also, what is the function of CrossAttn in (8) and MaskGenerator in (9)? Please explain these functions' structures mentioned above, including the inputs and outputs, to clarify the model design.\n- The shapes of the projection matrices mentioned in section 3.2 are not correct.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept