Referring Image Segmentation via Joint Mask Contextual Embedding Learning and Progressive Alignment Network
Authors: ZILING HUANG
Shin'ichi Satoh
Venue: EMNLP 2023 Main (2023)
Keywords: segmentation
multi-modality

Abstract: Referring image segmentation is a task that aims to predict pixel-wise masks corresponding to objects in an image described by natural language expressions. Previous methods for referring image segmentation employ a cascade framework to break down complex problems into multiple stages. However, its defects also obvious: existing methods within the cascade framework may encounter challenges in both maintaining a strong focus on the most relevant information during specific stages of the referring image segmentation process and rectifying errors propagated from early stages, which can ultimately result in sub-optimal performance. To address these limitations, we propose the Joint Mask Contextual Embedding Learning Network (JMCELN). JMCELN is designed to enhance the Cascade Framework by incorporating a Learnable Contextual Embedding and a Progressive Alignment Network (PAN). The Learnable Contextual Embedding module dynamically stores and utilizes reasoning information based on the current mask prediction results, enabling the network to adaptively capture and refine pertinent information for improved mask prediction accuracy. Furthermore, the Progressive Alignment Network (PAN) is introduced as an integral part of JMCELN. PAN leverages the output from the previous layer as a filter for the current output, effectively reducing inconsistencies between predictions from different stages. By iteratively aligning the predictions, PAN guides the Learnable Contextual Embedding to incorporate more discriminative information for reasoning, leading to enhanced prediction quality and a reduction in error propagation. With these methods, we achieved state-of-the-art results on three commonly used benchmarks, especially in more intricate datasets. The code will be released.

Referring Image Segmentation via Joint Mask Contextual Embedding Learning and Progressive Alignment Network
Ziling Huang
Shin'i iihii Satoh
The University of Tokyo huangzilingénii. ac.
National Institute of Informatics, Japan satoh@nii. ac.
Abstract
"the giraffe oatlng grass from a
koU-30
Referring image segmentation is task that aims to predict pixel-wise masks correspond- ing to objects in an image described by natu- ral language expressions. Previous methods for referring image segmentation employ a cas- cade framework to break down complex prob- lems into multiple stages. However, its lim- itations are also apparent: existing methods within the cascade framework may encounter challenges in both maintaining strong focus on the most relevant information during spe- cific stages of the referring image segmenta- tion process and rectifying errors propagated from early stages, which can ultimately result in sub-optimal performance. To address these limitations, we propose the Joint Mask Contex- tual Embedding Learning Network (JMCELN). JMCELN is designed to enhance the Cascade Framework by incorporating Learnable Con- textual Embedding and Progressive Align- ment Network (PAN), The Learnable Contex- tual Embedding module dynamically stores and utilizes reasoning information based the cur- rent mask prediction results, enabling the net- work to adaptively capture and refine pertinent information for improved mask prediction accu- racy. Furthermore, the Progressive Alignment Network (PAN) is introduced an integral part of JMCELN, PAN leverages the output from the previous layer as a filter for current output, effectively reducing inconsistencies be- tween predictions from different stages. By iteratively aligning the predictions, PAN guides the Learnable Contextual Eimbedding to incor- porate more discriminative information for rea- soning, leading to enhanced prediction qual- ity and a reduction in error propagation. With these methods, we achieved state-of-the-an re- sults on three commonly used benchmarks, es- pecially in more intricate datasets.
sens
"the gu-aftt eating grass
ISU+00
embeddies
"Me girgff not cating
pou
graus flom di pole"
1oU-0777
50PN
Figure I: Comparison of our work and previous work. (a) Representative late fusion cascade networks (i.c. CGAN(Luo et al., 2020a) and CRIS(CRI, 2022)). The results shown here are from CRIS. (b) Cascade networks with early fusion (i.e. LAVT((Yng et al., 2022) and EFN(Feng et al., 2021)). The results are from LAVT. (c) The proposed network. CEL: Contextual Embedding Learning Learning. MP: Mask Predicting. The Yel- low Arrows show the Contextual Embedding Learning, while The Brown Arrows show the Mask Predicting.
a natural language expression. It is one of the most basic tasks in human-robot InteractionDDautnnahnh 2007; Goodrich et al., 2008). In particular, it helps robots understand commands and localize corre- sponding objects. Unlike semantic/instance seg- mentation(Strudel et al., 2021; Chen al., 2018), which focuses on the image only, referring image segmentation involves two different modalities: im- age and text. Thus, in referring image segmenta- tion, how to achieve good alignment between the image and the most beneficial information in text is essential for the mask prediction.
The prevailing methods in the field of referring image segmentation rely on the cascade framework, where static-text are merged with feature maps pro- duced by the current stage to generate the output
1 Introduction
Referring image segmentation is a task that aims to predict a pixel-wise mask for objects referred to in

---

of the subsequent stage, As illustrated in Fig. ((a) and (b), the late and early fusion frameworks are
improved mask prediction. Additionally, intro- duce the Progressive Alignment Network (PAN) as part of JMCELN, PAN operates by using the
the two commonly used cascade frameworks. The late fusion framework, as adopted in methods like CGAN and CRIS(Luo et al., 2020a; CRI, 2022), ex- tracts text and image features separately, and then integrates them to form multi-modality features for reasoning. On the other hand, the early fusion framework, employed in methods such LAVT and EFN((Yag et al., 2022; Feng al., 2021), ex- tracts language features and conducts the entire fusion operation in the image encoding backbone. The Cascade framework a methodology that in- volves breaking down complex problems into mul- tiple stages or steps, with each step building upon the previous ones. This approach can enhance the salience prominence of referent objects in multi- stage reasoning processes. Despite their effective- ness, its limitations are also apparent: (a) If the static text used in each stage of the cascade frame- work is overly complex or does not appropriately focus on the most relevant information for that par- ticular stage, it can hinder the effectiveness of the approach. For instance, consider the expression in Fig. 1. "The giraffe not cating grass from pole." A human observer would inspect the "giraffe" in the image and utilize additional attributes local- ize the target. (b) The mistakes made the early stages of the Cascade Framework can propagate and potentially affect the later stages. Since the rea- soning process in the Cascade Framework is built upon previous stages, any errors or incorrect infor- mation introduced early on can impact subsequent stages and potentially lead to incorrect conclusions results.
output from the previous layer as a filter for the cur- rent output. It addresses inconsistencies between predictions from different stages by reducing the response of the current output and guiding the Con- textual Embedding to include more discriminative information for reasoning. This iterative alignment process enhances the quality of predictions and reduces the propagation of mistakes. By combin- ing the Learnable Contextual Embedding and PAN, JMCELN achieves dynamic reasoning and refine- ment of mask predictions. This integrated approach overcomes the limitations of static text and offers powerful solution for enhancing the salience and accuracy of referent objects within multi-stage rea- soning tasks.
In summary, cascade network presents two key challenges: the repetition of static text accumulates noise throughout reasoning and misunderstanding reasoning at earlier stages propagate, causing inac- curate predictions in subsequent stages. To avoid introduce too much noise, introduce Learnable Contextual Embedding. Learnable Contextual E-m- bedding aims to provide each stage dynamic and the most pertinent information, while PAN combats inconsistencies between stages to avoid error prop- agation. Learnable Contextual Embedding and Pro- gressive Alignment Network ensure the successful operation of our error correction mechanism within the cascade network.
To evaluate the effectiveness of the proposed methods, we conduct extensive on three commonly used benchmarks and we achieved state-of-the-are results especially in in more intricate datasets. In conclusion, our main contributions are fol- lows:
To address the aforementioned issue, we pro- pose the Joint Mask Contextual Embedding Learn- ing Network (JMCELN). JMCELN is designed enhance the Cascade Framework by incorpo- rating a Learnable Contextual Embedding and Progressive Alignment Network (PAN). The pri- mary objective to improve the accuracy of mask prediction by dynamically storing and utilizing rea- soning information for each subsequent step. The Learnable Contextual Embedding in JMCELN al- lows for the dynamic storage and utilization of reasoning information. It is updated based on the current mask prediction results, rather than rely- ing on static natural language expressions. This adaptability enables the network to capture and re- fine pertinent information effectively, leading to
We designed a Joint Mask and Contextual Embedding Learning Network (JMCELN) in which replace static-text with Learnable Con- textual Embedding for stage reasoning, which endows the network with the ability capture pertinent information for mask predicting every stage.
To obtain more accurate mask, we propose the Progressive Alignment Network (PAN). If the predictions of the two stages are inconsis- tent, the PAN will reduce response of the later output and then force the Contextual Embed-

---

ding generate more discriminative informa- tion.
features output from the previous layer. CRIS(CRI. 2022) utilizes similar framework as CGAN(Luo et al., 2020a), while it benefits from pre-trained CLIP(Radford et al., 2021a) model. EFN(Feng et al., 2021) and LAVT(Yang et al., 2022) show that better cross-modal alignments can be achieved through early fusion of linguistic and visual fea- tures. VLT(Ding et al., 2021) generates multiple sets of word attention weights to represent different understandings. All of these methods employ static text fuse with multi-modality features, while the information required in every stage is different. Dynamic Query based Segmentation Inspired by DETR(Carion et al.. 2020), some studies have proposed to zero/random initial vector hook helpful information from the image fea- ture map semaanticnsttanccpaapptii segmen- tation. MaskFormer et al., 2021) and Mask2FormerrChenn et al., 2022) treat segmen- tation as a mask classification task and use a ran- dom initial query to generate information from the image feature map for final segmentation. K- Net(Zhang et al., 2021) employs semantic kernels and instance kernels dynamic kerness to gen- erate helpful information stage-by-stage from the image encoding backbone (i.e. the CNN and Trans- former) for the final mask prediction. Panoptic SegFormer(Li et al., 2022) proposes to use things queries and stuff queries to store different types of information. All these studies verify that learn- able embedding can catch demanding information based on the task requirements. Our study is the first to introduce learnable embedding which be updated adaptively in accordance with the re- quirements of each stage for stage reasoning.
Our methods achieved state-of-the-an perfor- mance on three commonly used datasets: Re- feoco(Yu et al., 2016), Refcoco+(Yu et al., 2016) and Refeocog(Mao et al., 2016), es- pecially in in more intricate datasets, Ref- coco+(Yu Yu et al., 2016) and Refeocog(Mao et al., 2016). Extensive ablation studies demonstrated the validity of each of the pro- posed components.".
2 Related Work
Referring Image Segmentation The task of refer- ring image segmentation was first proposed by Hu et al. (Hu al., 2016). All work in this area can be categorized into two types. The first type includes one-stage image and language fusing of different layer feature maps in the image encoding backbone followed by multi-layer feature exchange for bet- ter results. For example, RMI([Lu et al., 2017) and DMM(Margffoy-Thaa) Tuay et al., 2018) utilize mem- ory unit(Hochreiter and Schmidhuber, 1997; Lei al., 2018) to fuse individual words with visual features. CMPC(Huang et al., 2020) first perceives all entities in the image and then suppresses unre- lated ones by graph reasoning. CMSA(Ye et al., 2019) has a cross-modal self-attention module to capture long-range dependencies between image and language. BRINet(Hu et al., 2020) utilizes vision-guided linguistic attention to get useful lan- guage information and then language-guides visual attention get the final outputs. BUSNet Yang al., 2021) uses bottom-up shift that progres- sively locates the referent. All these methods work separately on feature maps from different layers in the image encoding backbone. Then, they fuse the feature maps produced by different layers by us- ing memory unit(Liu et al.. 2017: Margffoy-Tuay et al., 2018; Huang al., 2020) multi-level fea- ture exchange(Ye Ye al., 2019). Not much work implements methods like those above anymore be- cause a one-stage interaction between visual and language information is not enough to predict accu- rate pixel-wise masks. The more recent second type of method aims at achieving better language and visual alignment. In particular, CGAN(Luo et al., 2020a) has cascade framework in which language features can fuse many times with multi-modality segmentation.
Methods
The overall framework is presented below. Then, we detail the image and text encoder, LSRAN, and JMCELN. Finally, the upsampling module and loss is introduced.
3.1 Overall Framework
First, we feed an image and a natural language expression into the image encoder and language encoder generate image features and language features. Fig.2 (a) shows the overall framework. Then, the Neck fuses these two modalities into multi-modality features Fn E TR x xC where и and W are the height and width of the input image and C' is the channel dimension. Then, the
Code:
http://gghua.commoyyommtttttttte

---

Contextual Embedding Learning (CEL)
Mult-Med Attemer PacE
Mull-Houd Attenten & Ado
Mulie Attembon Mead
Contextual Embedding
Terget
Enooder
*for left guy"
Text
Aitesson Hdd
Mus-Hod
Antensonn & Add
Malc-toar Hoad
Allansion
Muli-ttad Head
Upsample
NN
PAN
Image
Prediction
Mask Prediction [MP)
(n)
Long Shart Relation Aware Network (LSRAN)
Progressive Alignment Network (PAN)
Element-wisn multiply
Add
Pm
Fint
Muliemudallty
3,4m13,4) Tank
F2
Commr SOTTT
Query
Wi
ConsC5.4) Tank
|I
Concateration
IRA 96
Conv(5.8) 7ath
Fret
Fet
W4 Window Astention
18,, um224
Cone(5.12) Tank
SW Swischorm
(c)
Figure 2: (a) Overall framework of our methods. It consists of a text encoder for the natural language expressions and an image encoder for the input images. Language features and image features are fused by the neck. The Long-Short Relation-Awar Aware Network (LSRAN) captures long and short relations between objects in the image. We employ cascade framework. is updated in each stage. (b) Details of LSRAN. Conu(k, d) means convolutions with kernel size and dilated rate of d. us is window size for window attention. (c) Details of the Progressive Alignment Network (PAN).
The maps represent Fo, E # Fuz E R TH x W. XCs and F E R R XC's where C2 de- notes the channel dimension of of the 3rd-5th lay- ers. The Transformer( Vaswani al., 2017) is used as the language encoder to extract the global textual representation Fs E RIxC and textual representa- tion for every word Fi E RTCC
spatial relations between objects one image are estimated by using the LSRAN based on Fn. The output is Fimt I x xC Unlike previous meth- ods which utilize static-text to interface with the multi-modality output of each stage, use Learn- able Contextual Embedding dº E TRO where Q the dimensionality of contextual embedding to control the description ability. Each contextual embedding is randomly initialized by different pa- rameters which means the information taken by different contextual embedding is not exactly same. The information carried by g' is updated to g in accordance with Fm and Fmn- gf+l serves K, V to update Fú, into F(++1) F(++1) is fed into the Progressive Alignment Network (PAN) to ob- tain the output p(i+1) There are n stages in total. Finally, we upsample the refined Fm as the final output.
We obtain multi multi-modality feature from the multi-level image features Fo, Fog. Fey and global textual feature F, in a top-down manner. We re- shape and repeat the global textual feature Fa. x W x C and multiply the result with the highest 32 level image feature Foz to form modti fea- tures (Eq. 1). Then, the lower level feature F,y and Fv, are blended into the multi-modal feature by using Eq.2, where [1, 2]. Here, af-) means batch normalization(Ioftt and Szegedy, 2015) with LeakyReLu, Up(-) denotes 2x upsampling, [ means concatenation, while Ws is a linear em- bedding and W., and Wm,, are convolution lay- ers for channel projection. The final output is F Fmo ER* Fmy - - of(F.. (1)
mt
3.2 Image & Text Encoder and Neck
The referring image segmentation task has two in- puts: image с kllx x3 and natural language expression L E IRT",, where T' is the number of words in the natural language expression.
The image encoder uses the third-, fourth-, and fifth-layer feature maps of Res et al., 2016).
Fm = = (UU (.,,), (2)

---

3.3 Long-Short Relation-Aware Network There are likely to be various relations in one im- age. relation between two distant objects is called long relation, whereas relation between two nearby objects is called short relation. To better represent long relations and short relations, we propose the Long-Short Relation-Aware Net- work (LSRAN), depicted in Fig.2 (b). Differ- ent from previous work, LSRAN aims to estimate object-wise relationships, instead of traditional pixel-wise relationship analysis (e.g., ASPP(Chen et al., 2017)), on feature maps for correctly iden- tifying target objects. Considering that, we use a window attention(Liu et al., 2021a) to transfer in- formation to every pixel inside window to identify semantic meaning. Then, the LSRAN employs di- lated convolutions to capture the different types of relations between objects. We concatenate the four different relations and mix their channels by using a x 1 convolution. To obtain boundary information more precisely, we feed Fm into a 3 convolution and multiply the results with the relation features obtained. The final output is Fmt-
has useful information. Multiplication with Fm leds to a reasonable correlation (attention map) be- tween contextual embedding and original text and visual features. This attention map is applied to mi to reduce distinct responses and discard un- necessary information.
q gi+l softmax o F ) ml
+ +9"
(3)
ddg
3.4.2 Mask Predicting
Multi-Head Attention After obtaining the updated gi+l, we refresh Fin to Pitt g²-l already con- tains the information required for generating F+ so set F the and g as the K and V in the multi celli-headdee attention to obtain p++1 as shown in Eq.4.
Fr Pinelgttly 1g
F'mt (4) Progressive Alignment Network Inspired by Hor- Net(Rao al.. 2022), we propose progressive alignment network for a two-stage feature align- ment and update. Our progressive alignment net- work contains three layers, as shown in Fig.2 (c). First, Fi+1 is projected by a x 1 convo- yut
Fi-l
ma = softmax(
)a
vdn
3.4 Joint Mask Contextual Embedding Learning Network
lution into G + 2 C) in channel dimension. After a 7 x 7 depthwise convolution, we split pi+1 into three parts whose dimensions are [F1 E IR x F2 E ER F, R x Here, the output of the previous stage Fin is also projected by
The total framework for Joint Mask and Contextual Embedding Learning Network is shown in Fig.2 (a). is composed of two parts: contextual embedding learning and mask predicting with the Progressive Alignment Network.
convolution into C 4 in channel dimension. The output is multiplied by Fi and the result fed into 3 3 convolution with C in 2 channel dimension to form a new filter for F2. This process repeated three times to reduce inconsis- tent responses in ml progressively. The output produced by filtering is fed into SwitchNorm(Lng et al., 2019) and then F is added to form the final output Fill In the PAN, the output from the pre- vious layer acts as filters reduce the responses of inconsistent pixels predicted by the current layer progressively, which in turn forces the query generate more discriminative information for the next stage.
3.4.1 Contextual Embedding Learning
Different from the previous work(Luo et al., 2020a; CRI, 2022; Yang et al., 2021; Ding et al., 2021) which uses a static text for reasoning, we intro- duce learnable embedding gd that can be updated adaptively based on the output of the current stage. Here, we
denote the total number of stages and i represent the order of stage. The contextual embedding learning is based on multi-head atten- tion( Vaswani et al., 2017). The (i + 1)-th stage Contextual Embedding is updated from the i-th stage Contextual Embedding, multi-modality fea- tures Fm, and the output of the i-th features Fiu- Fm provides the initial multi multi-modality features and Fy. provides reasoning results in the i-th stage. Eq.3 is the (i 1)-th query generation process. The target for contextual embedding learning is to hook the most expedient information based on the cur- rent reasoning output. Because initial Contextual Embedding is random randomly initialized, so
3.5 Training Loss
After the multi-stage reasoning decoder, we already have reasonable results with accurate locations. We progressive upsample reasoning results into origi- nal size. The output feature map is F BRx and the target mask M E HxW Here, we use the eross-entropy loss to regress the final segmen-

---

tation mask.
train our model on its training set, choose the model's parameters with the best mloU on the val split, and evaluate on test set. Evaluation metric details are show in Supplymentary Material.
4 Experiments
Datasets To verify the effectiveness of our ap-
proach, we conducted experiments on three com- monly used datasets: Refcoco( Yu et al., 2016), Ref- coco+(Yu et al., 2016), Refcocog(Mao et al., 2016). The Refcoco dataset targets 50,000 different ob- jects in 19,994 images with 142,209 unique natural language expressions, which means each image contains more than one instance and each instance is described by more than one natural language expressions. Different from the Refcoco dataset, Refcoco+ has no location words in their referring expressions, which means can locate instances by their attributes only. The length of sentence in Refcoco+ is longer than in Refcoco. Refcoco+ consists of 141,564 expressions, 49856 objects and 19,992 images. The natural language expressions for these two datasets are generated by a two-player GamecKazemzadeh et al., 2014). Refcocog is the most challenging one among the three. Refcocog utilizes Mechanical Turk to generate natural lan- guage expressions. The dataset contains richer and longer descriptions. There are 104,560 expressions that describe 54,822 objects in 25,711 images. The average length of the referring expressions is 8.3 words. We used the UNC partition on the Refcocog dataset. The images of the three datasets were col- lected from MS-COCO(Lin et al., 2014).
4,1 Main results
Tab.1 compares the mloUs of our method with those of other state-of-the-ar methods. For a fair comparison, we report mloU scores for all meth- ods by standardized the data preprocessing method and trained previous methods using their released source code, strictly adhering to the training pro- cess outlined in their papers?. Our method outper- formed the previous methods on almost all datasets, On the Refcoco dataset, our method surpassed the state-of-the-an LAVT(Yang et al., 2022) by 0.7% and 2.02% on val and testA splits, respectively, while all methods obtained similar mloUs on the testB split. Moreover, it outperformed LAVT by 2.77%, 4.38%, and 0.52% on val, testA, and testB of the Refcoco+ dataset, respectively. On the most challenging dataset, Refcocog, which has signifi- cantly longer sentences, our model also achieved a noticeable improvement of 1.84% and 2.1 %% on the val and test splits. It is evident that our method demonstrates superior performance on more chal- lenging datasets, particularly Refcoco+ and Ref- cocog, which are characterized by longer sentence lengths and more attribute descriptions.
It's worth mentioning that outperformed CRIS(CRI, 2022) by a large margin, around 4%, on all datasets and splits. CRIS(CRI, 2022) utilizes the same pre-trained parameters as our method and it also employs the cascade framework; however. it keeps input language features for every stage static. These results demonstrate the effectiveness of our method.
Implementation Details The text and image encoder were initiated by CLIPPRadford et al., 2021b). For the image encoder, we employed ResNet-101(He et al., 2016). All the experi- ments are conducted on two A100 GPUs. In what follows, n represents the number of stages of the Joint Mask and Contextual Embedding Learning Network and it is set to n=6. Q represents the number of Contextual Embedding. Here, set Q=20. The channel dimension C=1024 unless otherwise stated. The input images were resized to 416x416. For all three datasets, Refcoco, Ref- coco+ and Refcocog, we set all sentence lengths to 30. All multi-head attentions had eight heads. The model was trained by the MlamW(Loshchiloo, and Hutter, 2017) optimizer for 60 epochs with batch size of 20. The initial learning rate was 0.000006 and it was decreased by polynomial learning rate decay with power 0.9. ar argmaæ along the channel dimension of the score maps was used as the predic- tion during inference. Following previous methods,
4.2 Ablation Study
The Proposed Components Tab.2 show the exper- iments Refcoco testB to verify the effectiveness of our proposed model. We train our model on its training set, choose the model's parameters with the best mloU on the val split, and evaluate on test set. Due to the space limit, we report the ablation ex- periments on Refcoco testB as it is considered the most difficult one, which better demonstrates the effectiveness of our proposed modules. 1 denotes our baseline whose design only contains image coder. language encoder, neck and upsampleing
"more details can be found in supplymentary material.

---

Refcoco
Refcoco+
Refeocog
Year
val
testA
testB
val
testA
testß
test
RMI(Liu et al., 2017)
ICCV'17 17
41.98
43.46
42.56
24.98
26.89
24.91
31.78
31.70
RRN(LI ct al., 2018)
CVPR 18
54.33
56.12
52.78
37.22
40.24
34.12
41.22
41.90
CMSA(Ye et 2019)
CVPR 19
59.00
62.22
55.99
40.41
45.19
34.60
40.07
40.27
BRINet(HU et al., 2020)
CVPR 20
62.74
65.23
60.80
49.32
54.20
43.20
47.55
47.74
CMPC(Huang et al., 2020)
CVPR 20
63.82
66.78
61.49
52.08
56.83
45.84
52.66
53.27
LSCM(HHi et al., 2020)
ECCV'20
63.46
65.88
61.52
52.01
56.51
45.24
50.87
51.15
MCN(Luo et al., 2020b)
CVPR 20
62.91
64.88
60.22
52.19
56.43
46.99
48.64
49.22
CGAN(Luo et al., 2020a)
MM'20
64.86
68.04
62.07
51.03
55.51
44.06
51.01
51.69
LTS(Jing et al., 2021)
CVPR 21
65.43
67.76
63.08
54.21
58.32
48.02
54.40
54.25
EFN(Feng et al., 2021)
CVPR'21 21
65.02
65.98
63.28
53.18
56.45
47.31
55.01
54.85
VLT(Ding ct al., 2021)
ICCV'21
65.65
68.29
62.73
55.50
59.20
49.36
52.99
56.65
CRIS(CRI, 2022)
CVPR'22
70.28
73.39
65.41
62.92
67.75
53.24
60.72
61.04
LAVT(Yang et al., 2022)
CVPR'22
73.70
75.67
70.44
64.22
68.31
56.84
62.24
62.88
Ours
74.40
77.69
70.43
66.99
72.69
57.34
64.08
64.99
Table Comparison with state-of of the-art methods on three widely used datasets. We trained all methods on their released code with 416x416 input resolution and used mloU as the evaluation metric.
Cascade framework, we slight improve mloU, tar- geting rate and mask quality. To address the issue of error propagation from early stages, we intro- duce the Progressive Alignment Network (PAN) at the end of each stage. In comparison to the results presented in Experiment 4, Experiment 5 showcases the effectiveness of PAN rectifying error propagation. Specifically, PAN demonstrates its ability to improve the mean mloU by approxi- mately 1%, increase the targeting rate by 1.02%, and enhance mask quality. These findings high- light the important role of PAN in mitigating the impact of error propagation and further improving the performance the overall system. In Experiment 6, we investigate the effectiveness of the feedback mechanism by excluding the up- date of Learnable Contextual Embedding (LCE) based on each stage's ' output. Instead, we input the previous stage's fused features, denoted Fm, and F0 mnt into each subsequent stage. The purpose of this evaluation is assess the impact of feedback on the overall performance. Surprisingly, the re- sults reveal a decrease mloU by approximately 1.32% compared to previous experiments. This indicates that updating the Learnable Contextual Embedding based on each stage's output plays a crucial role in refining the reasoning process and improving the accuracy of the segmentation task.
# Methods
mloU
Pr@ 50
Pr@90
Baseline
62.57
70.11
23.73
+LSRAN
68.03
77.53
27.36
+Cascade
69.23
78.08
29.54
Ours(w/o PAN)
69,46
78.86
30.56
5
Ours(w/ PAN)
70.43
79.88
31.54
6
w/o adaptive LCE
69.11
77.60
29.22
Table Ablation study on Refcoco testB.
module. In order to enhance the baseline perfor- mance, we introduce LSRAN, a novel approach that incorporates the capability to capture relations between objects within single image. Our pro- posed method not only achieves significant im- provement of approximately 5% in mloU, but also enhances the target rate (Pr@50) and mask quality (Pr@90). We further investigate the impact of in- tegrating the Cascade framework with Learnable Contextual Embedding, and present the results
The Cascade framework with Learnable Contextual Embedding leads to significant improvement of approximately 7% in mloU which demonstrate the effectiveness of combining the Cascade framework and Learnable Contextual Embedding in improving the quality of reasoning outcomes.
In 4, take step further by combining LSRAN with the Cascade framework. In this hy- brid approach, the outputs from the neck compo- nent solely provide image and language informa- tion, while LSRAN introduces additional relation information. By incorporating LSRAN into the
5 Visualization
Fig.3 shows some qualitative results, comparing our method and previous ofate-of oftte----te--rr the-art methods.

---

steamboot sitingg near the dook
black pot with plant in 1t"
"An old dirty scooter parked next mental fence
(4) Input
(b) Ground Truth (c) Ours
(d) LAVT
(c) CRIS
(a) Input
(b) Ground Truth
(c)
(d) LAVT
Figure 3: Qualitative comparison of our work and previous work: LAVT and CRIS.
"robertson" "
"girt right with hair up
blue car
*Player second from left in the foreground"
Input
(b) GT
(c) Raselinc
(d) #4
(c) #5
(a) Input
(b) GT
(c) Baseline
(d) #4
(e) #5
Figure 4: Qualitative comparison of ablation study. #4 and #5 indicates the number in Table 2.
Model
Split
mloU
Pr
Pr699
Limitations
LAVT
68.31
78.31
26.77
Ours
testA
72.69(44.38)
83.74(45.43)
29.71(42.94)
As shown Tab.3, our methods produce a relative improvement in the mIoU metric 4.38% in testA and 0.50% in testB. Moreover, in testA, there is 4.38% improvement in Pr@50,, while Pr@ 90 only improves by 2.94%. The improvement in Pr@ 50 in testB is 1.07%, while Pr@90 shows no improve- ment. By analyzing testA and testB, find the target in testA is the human instances whose size are larger while the target in testB is mainly ob- jects whose size are smaller. This indicates that our methods have more ability to identify larger target objects, but for the smaller objects, the im- provement is limited. And our method is not good at producing high-quality output mask. Compared with LAVT, cascade stages only uses 52 52 feature maps for reasoning, while LAVT utilizes clear-sighted early fusion method, in which the query fused with larger feature map with a 104 x 104 size in the first stage. The larger feature map is good for finding smaller objects and keep- ing the boundary. However, because the number of layers in image encoding backbone designed for other task is fixed, it is difficult for carly fu- sion methods define their own reasoning stages, which limits their performance. For example, Swin- Transformer(L.iu iiu et al., 2021b) has four layers, so LAVT performs image-text alignment four times. By comparison, we adopt the late fusion cascade
LAVT
56.84
63.24
24.34
testB
Ours
57.34(+0.50)
64.31(+1.07)
24.34(+0.00)
Table 3: Results for refcoco+, testA and testB.
While LAVT and CRIS encountered difficulties and made mistakes in these four challenging cases, our method demonstrated robust performance and successfully tackled them.
Figure 4 presents qualitative results obtained from the ablation study. Starting from the base- line and progressing experiment #4, the Cascade framework with Learnable Contextual Embedding is introduced. This addition results in improved per- formance, as observed the case of the "blue car" where the Cascade framework with Learnable Con- textual Embedding predicts a more compact mask compared to the baseline. Experiment #5 intro- duces the Progressive Alignment Network (PAN). Notably, in the cases of "robertson" and "girl front right with hair up". PAN demonstrates its ability rectify incorrect predictions during the reason- ing process. This showcases the effectiveness of PAN refining and improving the accuracy of the segmentation outputs.

---

framework which is more flexible in defining the number of reasoning stages fit for this lask.
Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. 2021. Encoder fusion network with co-attention em- bedding for referring image segmentation. In CVPR.
7 Conclusion and Future Work
Michael A Goodrich, Alan C Schultz, et al. 2008. Human-robot interaction: survey, Foundations and Trends in Human Muaman-Computer Interaction.
We proposed a Joint Mask and Contextual Em- bedding Learning Network that dynamically up- dates the Contextual Embedding the basis of the reasoning output and long-short relation-aware network learn objects relations. Experimental results demonstrate the effectiveness of our meth- ods. In the future, we want to handle the limitation mentioned in 6ec.6 and improve the targeting rate for small objects and recover boundary information better.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In CVPR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
Ronghang Hu, Mareus Rohrbach, and Trevor Darrell. 2016. Segmentation from natural language expres- sions. In ECCV.
Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. 2020. Bi-directional relationship infer- ring network for referring image segmentation. In CVPR.
8 Acknowledge
Ziling Huang was supported by Japanese Gov- emmment(MEXT) Scholarship and this work was supported by JSPS KAKENHI JP22H03620 and JP22H05015.
Shaofei Huang. Tianrui Hui. Si Liu, Guanbin Li, Yun- chao Wei, Jizhong Han, Luoqi Liu, and Bo Li. 2020. Referring image segmentation via cross-modal pro- gressive comprehension. In CVPR.
References
Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han. 2020. Linguistic structure guided context modeling for referring image segmentation. In ECCV.
2022. Cris: Clip-driven referring image segmentation. In CVPR.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In ECCV.
Sergey loffe and Christian Szegedy. 2015. Batch nor- malization: Accelerating deep network training by reducing internal covariate shift. In ICML.
Ya Jing, Tao Kong. Wei Wang. Liang Wang. Lei Li, and Tieniu Tan. 2021. Locate then segment: A strong pipeline for referring image segmentation. In CVPR.
Liang-Chieh Chen, George Papandreou, lasonas Kokki- nos, Kevin Murphy, and Alan I. Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected crfs. IEEE TPAMI.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder- decoder with atrous separable convolution for seman- tic image segmentation. In ECCV.
Tao Lei, Yu Zhang, and Yoav Artzi. 2018. Training rnns fast
In ICLR.
Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xi- aojuan Qi, Xiaoyong Shen, and Jiaya Jia. 2018. Re- ferring image segmentation via recurrent refinement networks. In CVPR.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal im- age segmentation. In CVPR.
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. 2022. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS.
Kerstin Dautenhahn. 2007. Socially intelligent robots: dimensions human-robot interaction. Philosoph- ical transactions of the royal society B: Biological sciences.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision-language transformer and query generation for referring segmentation. In ICCV.
Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. 2017. Recurrent multimodal teraction for referring image segmentation. In ICCV.

---

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV.
Zhao Yang, Jiagi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. 2022. Lavt: Language-aware vision transformer for referring im- age segmentation. In CVPR.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 202 lb. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. 2019. Cross-modal self-attention network for refer- ring image segmentation. In CVPR.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. 2016. Modeling context in referring expressions. In ECCV.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In ICLR.
Gen Luo, Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jin- song Su, Chia- Wen Lin, and Qi Tian. 2020a. Cascade grouped attention network for referring expression segmentation. In ACM MM.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. 2021. K-net: Towards unified image segmentation. In NewrIPS.
Gen Luo. Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao. Chenglin Wu, Cheng Deng, Rongrong 2020b. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR.
Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. 2019. Differentiable learning-to- normalize via switchable normalization. In ICLR.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In CVPR.
Edgar Margffoy- Tuayy, Juan Pérez, Emilio Botero, and Pablo Arbeláez. 2018. Dynamic multimodal instance segmentation guided by natural language queries. In ECCV.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 202 la. Learning transferable visual models from natural language supervision. In ICML.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b. Learning transferable visual models from natural language supervision. In ICML.
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, and Jiwen Lu. 2022. Hornet: Efficient high-order spatial interactions with recur- sive gated convolutions. In NeurIPS.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. 2021. Segmenter: Transformer for semantic segmentation. ICCV.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NewrIPS.
Sibei Yang, Meng Xia, Guanbin Li, Hong- Yu Zhou, and Yizhou Yu. 2021. Bottom-up shift and reasoning for referring image segmentation. In CVPR.


Reviews:
[{"review_id": "U0rAgIPGEo", "reviewer": "Reviewer_i668", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply value your feedback and its contribution to enhancing the clarity of our paper.\n\n(1) Highlight the difference of the proposed techniques:\n(a) Learnable Contextual Embeddings are inspired by the use of random/zero initial queries in K-net and Segformer. However, it differs from queries in K-net and Segformer in two aspects: i) In K-Net and Segformer, the number of queries represents the number of segmentation maps in the final outputs. In contrast, the size of our Learnable Contextual Embeddings represents the capacity to carry information for reasoning.  ii) Our Learnable Contextual Embeddings can be updated based on the output of the current layer, whereas queries in K-Net and Segformer cannot be updated adaptively; they only take in information layer by layer. (b) The Progressive Alignment Network draws inspiration from the concept of progressive refinement as seen in HorNet. While HorNet operates with a single input, our Progressive Alignment Network is newly designed for utilizing two inputs from different stages, facilitating a progressive comparison of their differences. By integrating these two components, we introduce a new error correction mechanism for cascade network which is not appear in any previous work.\n\n(2) Description of Challenges and Motivations:\nThank you for your valuable suggestions. We will incorporate the following in our introduction:  In summary, cascade network presents two key challenges: the repetition of static text accumulates noise throughout reasoning and misunderstanding reasoning at earlier stages propagate, causing inaccurate predictions in subsequent stages. To avoid introduce too much noise, we introduce Learnable Contextual Embedding. Learnable Contextual Embedding aims to provide each stage dynamic and the most pertinent information, while PAN combats inconsistencies between stages to avoid error propagation. Learnable Contextual Embedding and Progressive Alignment Network ensure the successful operation of our error correction mechanism within the cascade network.\n\n(3) Inclusion of Limitations and Failure Cases:\nWe has examined failure cases, revealing insights into specific challenges. Two primary categories emerge: (a) Small Object: We already confirmed that our method can properly handle the cases when the target objects are small, but final segmentation map can not find its precise boundary because input feature maps are too small.  (b) Incorrect Text: In cases where text contains errors or abbreviations, such as \"appl\" instead of \"apple,\" our model encounters difficulties in establishing accurate connections. To offer a well-rounded assessment, we visually present these failure scenarios in our paper. These insights deepen our understanding of our approach's strengths and limitations, guiding future advancements.\n\nWe sincerely appreciate your thoughtful critique and the opportunity to enhance the clarity, thoroughness, and robustness of our paper. Your insights are invaluable in driving us to refine and elevate the quality of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The introduction of JMCELN, which replaces static-text with Learnable Contextual Embeddings, is a significant contribution. By incorporating Learnable Contextual Embeddings, the model gains the ability to capture pertinent information at every stage of reasoning. This approach enhances the network's capacity to understand the context and generate more accurate mask predictions.\n2) The proposal of the Progressive Alignment Network is another noteworthy contribution. The PAN plays a crucial role in improving the accuracy of mask predictions by addressing inconsistencies between predictions from different stages. This iterative alignment mechanism reduces the response of the later output and encourages the generation of more discriminative information, resulting in more precise masks.\n3) The paper's methods, JMCELN, and PAN, achieve state-of-the-art performance on three widely used datasets in referring expression comprehension tasks: Refcoco, Refcoco+, and Refcocog. The demonstration of superior performance, especially on intricate datasets like Refcoco+ and Refcocog, highlights the effectiveness of the proposed approach.\n4) The paper reinforces the validity of its proposed components through extensive ablation studies. By conducting these studies, the authors demonstrate the importance of each proposed component and its positive impact on performance.", "reasons_to_reject": "1) While the core ideas of the proposed modules, Contextual Embeddings, and Progressive Alignment, are cited as widely referenced in related fields, the paper lacks a clear and thorough clarification of how these concepts differ from existing approaches in the context of the specific problem addressed in this paper. It would be beneficial to provide a more detailed comparison with related works to highlight the novelty of the proposed techniques.\n2) The paper could benefit from a more explicit and concise description of the challenges faced in the stage reasoning problem and the precise motivations behind the design of JMCELN and PAN. \n3) The visualizations presented in Figures 3 and 4 illustrate the effectiveness of the proposed method. However, the paper could be improved by providing additional examples where the proposed approach might fail or show limitations. This would allow readers to gain a better understanding of the proposed method's capability boundaries and its performance in various scenarios.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel approach to address the problem of stage reasoning in Mask Predicting tasks, particularly in the context of referring expression comprehension. The paper introduces an innovative solution to the stage reasoning problem in Mask Predicting tasks by leveraging Learnable Contextual Embeddings and Progressive Alignment Network. The proposed approach achieves state-of-the-art results on challenging referring expression comprehension datasets, providing valuable insights into the field of visual reasoning and language understanding.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2jbUlJWLs", "reviewer": "Reviewer_mgwW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper.\n\n(1) Enhancement for Smaller Objects:\nOur proposed framework can not only handle large objects, but also effectively in finding small object. After carefully check, the Table 3 in our paper, in Refcoco testA(first row),  around 2.31% targeting rate(Pr@50) improvements is contributed by small objects while large   objects contribute 3.12% improvement in targeting rate(Pr@50). While in Refcoco testB(second row),  the small objects contribute 0.43% improvements in targeting rate(Pr@50), and large objects contribute 0.64% improvements in targeting rate(Pr@50). These results mean our proposed model also improve the targeting rate for small objects. While in mask quality(Pr@90), small objects contribute 1.2% improvement in Refcoco testA, and around 1.7% mask quality(Pr@90) improvement is contributed by larger objects. (The larger objects and small objects are defined by area provided by mscoco dataset, area<481 is small object while area>=481 is larger object. )\n\n(2) Acknowledging Recent SOTA Development in Early 2023:\nThank you for your suggestions. We also notice Polyformer on arxiv in Feb. 2023.  However, it is unfair to compare our work with Polyformer and two work, SeqTR[1] published in ECCV2020 and RefTr[2] published in NIPS2021, because of two reasons: (a) all these work use large-scale referring grounding datasets Visual Genome and Flickr30k-entities to pretrain their model. (b) all these methods are designed for multitask. As shown in SeqTR[1] and RefTr[2],  the pretraining can bring approximately 5% improvement in mIoU for refcoco and an impressive 10% improvement in mIoU for refcoco+ and refcocog. Moreover, in multitask methods, training referring grounding and referring segmentation jointly would significantly impact the performance of referring segmentation, as evidenced in Table 5 of the PolyFormer and Table 4 in RefTr[2]. Because of these reasons, most of previous work[3][4] in referring segmentation do not compare with these work.\n\nIt's crucial to highlight that our work only focuses on referring segmentation and does not incorporate the large-scale referring grounding dataset for pretraining. Therefore, a direct comparison between our work and PolyFormer would be unfair.\n\nWe greatly value your engagement and the insights you provide. Your comments substantially enrich the depth and quality of our research, propelling us to continuously refine and enhance our work.\n\n[1] SeqTR: A Simple Yet Universal Network for Visual Grounding. ECCV2020\n\n[2] A one-step approach to multi-task visual grounding. NIPS2021\n\n[3] CRIS: CLIP-Driven Referring Image Segmentation. CVPR2022\n\n[4] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation.CVPR2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written, making it easy to follow the proposed method and its contributions.\n2. The drawn diagram visually presents the proposed approach, enhancing understanding.\n3. JMCELN demonstrates the capability to improve the segmentation of large objects effectively.", "reasons_to_reject": "1. The main concern is that JMCELN appears to be restricted to improving the segmentation of large objects, rendering it unsuitable for generating scenarios involving smaller objects.\n2. The paper compares JMCELN with works up to 2022, disregarding newer approaches like PolyFormer [1], which achieve superior performance in early 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a joint mask contextual embedding learning network, JMCELN, to solve the challenges in both maintaining a strong focus on the most relevant information at specific stages and  rectifying errors propagated from early stages. JMCELN proposes to use a learnable contextual embedding module to enable adaptive capturing pertinent information, and a progressive alignment network to reduce inconsistencies between stages. Experiments show that the proposed methods outperform baseline methods on three benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I5FXRctxb0", "reviewer": "Reviewer_hADV", "summary": "", "questions": "Question A: While the performance gains on the Refcoco testB dataset are noteworthy, could you provide further results from other datasets to help validate the method's effectiveness and generalizability?\nQuestion B: In terms of transformer-based segmentation methods, particularly the recently popular Meta-SAM, could you outline the advantages of the proposed approach? It would be beneficial to highlight the unique strengths of your method.\nQuestion C: Visualization of instances where the method does not perform as expected might offer a valuable perspective on the current limitations. It could also help in identifying potential areas for improvement.\nQuestion D: Some parts of the paper could benefit from further refinement to enhance the readability and logical flow. For instance, it might be helpful to consolidate some sections in the methods part.\nI hope these queries will aid the authors in enhancing the strength of the paper.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful feedback, which has played a crucial role in improving the quality of our paper.\n\n(1) Ablation Study for JMCELN:  When remove JMCELN from original cascade network, the model become original static-text stage reasoning network. We obtained 68.77 mIoU in Refcoco testB, and 76.32 mIoU in Refcoco testA dataset. Compared with results in our paper, our error correction mechanism can largely improve the final results of cascade network.\n\n(2) Question A:\n***\n|# | Methods              | mIoU | Pr@50| Pr@90 |\n| --- | --- | --- | --- | --- |\n| 1| Baseline              | 71.68 | 81.46  | 26.29   |\n| 2| +LSRAN              | 75.37 | 87.29  | 29.10   |\n| 3| +Cascade            | 76.85 | 88.70  | 30.35.  |\n| 4| Ours(w/o PAN)    | 77.21 | 88.99  |  31.15  | \n| 5| Ours(w/ PAN)      | 77.69 | 89.27  |  31.31  |\n| 6| w/o adaptive LCE | 77.04| 89.04  | 29.57   |\n***\n\nDue to time limitation, the table provided represents the ablation study conducted on the refcoco testA dataset, which is known as the easiest split within the refcoco dataset. From table, it is obvious, the proposed LSRAN, LCE and PAN can help model improve mIoU, targeting rate(Pr@50) and boundary precision(Pr@90), which help validate the method's effectiveness and generalizability. In camera ready version, we will provide more ablation results for refcoco+ and refcocog dataset. \n\n(3) Question B: Strengths compared with Meta-SAM\n    (a) Our main advantage over Meta-SAM lies in the fact that our model can understand the meaning of text inputs better and segment the object text referred to. In the context of Meta-SAM, although they offer prompts in the form of points, bounding boxes, and text, they have not yet released the model specifically designed for text prompts. Consequently, the current Meta-SAM framework remains incapable of effectively performing the task of referring segmentation. To assess the efficacy of their text prompts, we extracted BERT text features as prompts and input them into the Meta-SAM framework. We utilized the refcoco dataset for fine-tuning and implemented various training strategies. However, even after these rigorous efforts, the highest achieved mIoU on the validation set only approximately 58%, a considerable gap compared to our model's performance at 74.40%.\n    (b) Although Meta-SAM achieved notable segmentation results, much of its performance stems from the availability of extensive training data and abundant computing resources. They employed 128 GPUs for training the Meta-SAM model, a resource-intensive endeavor that might be financially impractical for ordinary research institutes. In contrast, our model relies solely on publicly available datasets such as refcoco, refcoco+, and refcocog for training. Remarkably, our entire training process is accomplished within a day on 4 GPUs.\n\n(4) Question C: Visualization of Failure Cases\nWe has examined failure cases, revealing insights into specific challenges. Two primary categories emerge: (a) Small Object: We already confirmed that our method can properly handle the cases when the target objects are small, but final segmentation map can not find its precise boundary because input feature maps are too small.  (b) Incorrect Text: In cases where text contains errors or abbreviations, such as \"appl\" instead of \"apple,\" our model encounters difficulties in establishing accurate connections. To offer a well-rounded assessment, we visually present these failure scenarios in our paper. These insights deepen our understanding of our approach's strengths and limitations, guiding future advancements.\n\n(5) Question D: \nWe greatly appreciate your valuable feedback regarding the refinement of certain sections within our paper. Your insights are instrumental in guiding us toward improving the overall readability and logical coherence. We are committed to carefully examining these sections and making necessary adjustments to ensure paper's readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A Joint Multi-stage Contextual Embedding Learning Network, which can dynamically update context embeddings and better perform multi-stage inference, offering an effective framework for referring image segmentation tasks.\nA Progressive Alignment Network has been designed to effectively reduce error propagation across different stages and improve the consistency of prediction results. This design concept can also be generalized to other visual tasks that require multi-stage inferences.\nComprehensive validation was conducted on multiple referring image segmentation datasets, with all results outperforming the current state-of-the-art methods, proving the effectiveness of the proposed method.\nA Long-Short Relation Awareness Network was introduced to better represent the relationships between objects within images. This component design can also be generalized to other tasks that require object relationship modeling.\nThe results of the paper can provide reference on how to better integrate visual and language information and carry out multi-stage inference in other image understanding tasks.\nThe code and model of the paper will be publicly released, providing other researchers with powerful and effective pre-trained models to advance the development of this field.\nOverall, this paper has made valuable explorations in advancing multi-stage inference, error propagation correction, and the combination of image understanding and natural language, and has the potential to be published at top-tier conferences.", "reasons_to_reject": "The proposed Joint Mask Contextual Embedding Learning Network (JMCELN) expands upon previous cascade frameworks. It would be beneficial to further elucidate the unique aspects that separate JMCELN from these existing frameworks.\nWhile the Learnable Contextual Embedding (LCE) and Progressive Alignment Network (PAN) demonstrate improved performance, more extensive analysis and additional ablation studies could provide further insight into their effectiveness and indispensability.\nThe performance improvements on some datasets are notable but could be more consistent across all tested datasets. A deeper exploration of the method's generalizability could strengthen the paper.\nCertain areas of the paper could benefit from further refinement to enhance readability and logical flow.\nIn summary, this paper tackles an important issue and contributes valuable advancements. To further strengthen the paper, consider focusing on demonstrating the novelty of the method, ensuring completeness in evaluation, and enhancing the clarity and organization of the writing. I hope these suggestions will assist the authors in improving the paper.", "questions_to_authors": "Question A: While the performance gains on the Refcoco testB dataset are noteworthy, could you provide further results from other datasets to help validate the method's effectiveness and generalizability?\nQuestion B: In terms of transformer-based segmentation methods, particularly the recently popular Meta-SAM, could you outline the advantages of the proposed approach? It would be beneficial to highlight the unique strengths of your method.\nQuestion C: Visualization of instances where the method does not perform as expected might offer a valuable perspective on the current limitations. It could also help in identifying potential areas for improvement.\nQuestion D: Some parts of the paper could benefit from further refinement to enhance the readability and logical flow. For instance, it might be helpful to consolidate some sections in the methods part.\nI hope these queries will aid the authors in enhancing the strength of the paper.", "paper_topic_and_main_contributions": "This paper investigates the task of referring image segmentation, aiming to predict the pixel-level occlusion of the objects mentioned in the image based on natural language descriptions. The main contributions of this paper are:\nProposing a Joint Multi-stage Contextual Embedding Learning Network (JMCELN), which can dynamically store and utilize multi-stage inference information through learnable context embedding.\nIntroducing a Progressive Alignment Network (PAN), which can correct the inconsistencies in predictions at different stages, reducing error propagation.\nConducting experiments on three referring segmentation datasets, the results outperform multiple state-of-the-art methods, demonstrating the effectiveness of the proposed approach.\nProposing a Long-Short Relation Awareness Network (LSRAN), which can better capture the relationships between objects within the image.\nThis paper addresses the issues of over-reliance on static language expressions and error propagation inherent in current multi-stage methods. By implementing learnable context embedding and the Progressive Alignment Network, it achieves more accurate and robust multi-stage inferences, representing high-quality innovative work in this field.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Here are some suggestions to improve spelling, grammar, writing style, and presentation of the paper:\nLine 78: Rephrase \"its defects also obvious\" to \"its limitations are also apparent\" for better clarity.\nLine 61: Add \"the\" before \"early fusion framework\".\nLine 139: Change \"Endow the network\" to \"Endows the network\" to fix grammar error.\nLine 316: Simplify phrase \"obtained before\" to just \"obtained\"."}}]

Author rebuttal:

Meta-review: Three reviewers provided feedback for this paper and their reviews were in consensus. Overall, reviewers found the work to be novel and significant, particularly the new network architecture. They appreciated the results, i.e. the high scores on multiple benchmarks, as well as the ablation analysis provided by the authors. The main complaints included the lack of extensive qualitative examples and a clear comparison/clarification to recent related work. There was also a concern that the method does not work well for small objects, but I think this isnt a strong concern since it is unreasonable to expect a single paper to address all concerns within a problem. Overall, post rebuttal, reviewers continue to be happy with this paper and fairly excited about it. I agree with this feedback.
Decision: Accept-Main