Electronics 2024, 13, x. https://doi.org/10.3390/xxxxx  www.mdpi.com/journal/electronics 
Article  1  
Text-to-Image Segmentation with Open-Vocabulary and Multi- 2  
tasking  3  
Lihu Pan  1, *, Yunting Yang  1 , Zhengkui Wang  2  and Rui Zhang  1   4  
1   School of Computer Science and Technology, Taiyuan University of Science and Technology, Taiyuan,  5  
030024, China; <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(Y.Y.); <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(R.Z.)  6  
2   InfoComm Technology Cluster, Singapore Institute of Technology, 138683, Singapore; zhengkui.wang@sin- 7  
gaporetech.edu.sg  8  
*  Correspondence: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>  9  
Abstract: Open-vocabulary learning has recently gained prominence as a means to enable image  10  
segmentation for arbitrary categories based on textual descriptions. This advancement has extended  11  
the applicability of segmentation systems to a broader range of generally purpose scenarios. How- 12  
ever, current methods often revolve around specialized architectures and parameters tailored to  13  
specific segmentation tasks, resulting in a fragmented landscape of segmentation models. In re- 14  
sponse to these challenges, we introduce OVAMTSeg, a versatile framework designed for Open- 15  
Vocabulary and Multitask Image Segmentation. OVAMTSeg harnesses adaptive prompt learning  16  
to empower the model to capture category-sensitive concepts, enhancing its robustness across di- 17  
verse multi-task and scenario contexts. Text prompts are employed to effectively capture semantic  18  
and contextual features of the text, while cross-attention and cross-modal interactions enable the  19  
fusion of image and text features. Furthermore, a transformer-based decoder is incorporated for  20  
dense prediction. Extensive experimental results underscore the effectiveness of OVAMTSeg, show- 21  
casing its state-of-the-art performance and superior generalization capabilities across three segmen- 22  
tation tasks. Notable achievements include a 47.5 mIoU in referring expression segmentation, 51.6  23  
mIoU on Pascal-VOC with four unseen classes, 46.6 mIoU on Pascal-Context in zero-shot segmen- 24  
tation, 65.9 mIoU on Pascal-5i, and 35.7 mIoU on COCO-20i datasets for one-shot segmentation.  25  
Keywords: image segmentation; open vocabulary; multitask; multi-modal interaction  26  
  27  
1. Introduction  28  
Image segmentation represents a deeply explored and pivotal domain within the  29  
field  of  computer  vision.  Its  primary  objective  is  the  simultaneous  categorization  and  30  
grouping of pixels belonging to distinct objects within an image. Recent strides in image  31  
segmentation owe their success largely to the availability of expansive dataset [1-3], me- 32  
ticulously annotated to include pixel-level masks and object category labels. However,  33  
these annotations, although invaluable, come at a significant cost in terms of time and  34  
labor. Consequently, the predefined categories within current segmentation tasks remain  35  
restricted in scope, far removed from the vast and diverse lexicon that humans employ to  36  
describe the complexities of the real world. Such limitations in the learning objectives of  37  
existing segmentation systems impose a substantial impediment to scalability, particu- 38  
larly when attempting to accommodate richer and more encompassing semantic nuances.    39  
To address the inherent constraints of predefined categories and unlock the potential  40  
for handling custom-defined classes beyond the confines of training data, the paradigm  41  
of open-vocabulary learning has gained prominence. Open-vocabulary learning leverages  42  
the power of large-scale visual-language pre-training models, exemplified by prominent  43  
models like CLIP[4] and ALIGN[5], to compute semantic similarity between visual con- 44  
cepts and textual descriptions. Notably, a burgeoning body of research in segmentation  45  
Citation: To be added by editorial 
staff during production. 
Academic Editor: Firstname Last-
name 
Received: date 
Revised: date 
Accepted: date 
Published: date 
 
Copyright:  ©  2024  by  the  authors. 
Submitted  for  possible  open  access 
publication under the terms and con-
ditions of the Creative Commons At-
tribution (CC BY) license (https://cre-
ativecommons.org/licenses/by/4.0/). Electronics 2024, 13, x FOR PEER REVIEW  2  of  14 
 
 
based open-vocabulary studies[6, 7] has emerged with the goal of devising task-specific  46  
architectures and parameters tailored to single segmentation tasks. For instance, ZSSeg[6]  47  
harnesses the capabilities of off-the-shelf pre-trained CLIP models and demonstrates com- 48  
petitive performance in open-vocabulary semantic segmentation. However, when extend- 49  
ing these methods to a broader spectrum of segmentation scenarios, they exhibit signifi- 50  
cant limitations. Firstly, a unified model cannot be seamlessly applied to address multiple  51  
segmentation  tasks,  necessitating  retraining  and  the  deployment  of  numerous  custom  52  
models for diverse tasks. Additionally, while CLIPSeg   [8] successfully handles multiple  53  
segmentation tasks within a compact framework, its reliance on fixed-format text prompts  54  
(e.g., "photos of...") may impose restrictions on the generalization of human language un- 55  
derstanding in practical applications. Moreover, it lacks the adaptability to dynamically  56  
adjust the modality correlation degree for different tasks and data, and it is not inherently  57  
designed for open-vocabulary tasks.    58  
In response to the challenges outlined above, we propose OVAMTSeg, a framework  59  
tailored for open-vocabulary and multitask image segmentation. OVAMTSeg, designed  60  
with precision and versatility in mind, is driven by two primary objectives:    61  
(1) Multitasking: OVAMTSeg seamlessly adapts to a spectrum of tasks, encompass- 62  
ing referring expression segmentation, zero-shot, and one-shot image segmentation.    63  
(2) Open-Vocabulary: OVAMTSeg exhibits the capacity to generalize across a wide  64  
array of segmentation categories, embracing the flexibility to accommodate arbitrary cat- 65  
egories.  66  
OVAMTSeg unfolds as a two-stage segmentation paradigm. The initial stage entails  67  
the extraction of universal mask proposals, while the subsequent stage is dedicated to the  68  
precise segmentation of these masks. Crucially, OVAMTSeg operates as a unified frame- 69  
work, cultivating a profound understanding of both textual and visual features for seg- 70  
mentation tasks, driven by text and image prompts. An adaptive prompt learning mech- 71  
anism is introduced to encode category-specific concepts into the textual abstraction, en- 72  
dowing OVAMTSeg with the versatility to tackle diverse segmentation tasks spanning  73  
arbitrary categories, all within a single, unified model.    74  
To further elevate its performance, OVAMTSeg augments the text encoder and inte- 75  
grates a multimodal interaction  module. This strategic enhancement facilitates the dy- 76  
namic adjustment of modality correlations, enabling a more adaptable fusion of text and  77  
image features across distinct tasks and datasets.    78  
In summary, OVAMTSeg emerges as a task-flexible, category agnostic, and perfor- 79  
mance-driven framework. The following concisely lists our contributions:    80  
• We introduce OVAMTSeg, a universal open-vocabulary framework renowned for  81  
its  capacity  to  efficiently  segment  images  based  on  arbitrary  text  or  image  prompts.  82  
OVAMTSeg effectively addresses the intricate challenges posed by zero-shot, one-shot,  83  
and referring expression segmentation tasks.    84  
•Adaptive prompt  learning empowers OVAMTSeg  to explicitly encode category- 85  
specific information into a compact textual abstraction, facilitating the model’s adeptness  86  
in generalizing to diverse textual descriptions. Additionally, we enhance the text encoder  87  
and introduce a multimodal interaction module to optimize cross-model fusion.    88  
• Our model’s efficiency and effectiveness are meticulously demonstrated through  89  
comprehensive evaluations across various benchmark datasets. Extensive experimental  90  
results conclusively establish that our proposed model surpasses current standards by a  91  
substantial margin, rendering it a highly viable choice for multitask deployment.    92  
2. Related Work    93  
2.1. Open Vocabulary Segmentation    94  
In recent years, deep learning techniques [9-14] have advanced image segmentation  95  
[15-20], especially in open-vocabulary segmentation, addressing unseen categories. Re- 96  
search divides into two main areas: mapping visual features to semantics and cross-modal  97  Electronics 2024, 13, x FOR PEER REVIEW  3  of  14 
 
 
alignment with pre-trained models. SPNet [7] uses a unique mapping strategy to project  98  
visual features onto a fixed semantic word coding matrix, facilitating the prediction of  99  
category probability distributions. ZS3Net [3] extends SPNet by mapping semantic space  100  
to visual space, generating pixel-level features for previously unseen categories and su- 101  
pervising the visual segmentation model. On the other hand, cross-modal alignment uti- 102  
lizes the strong zero-shot abilities of pre-trained cross-modal models like CLIP[4] for exe- 103  
cuting open vocabulary segmentation tasks. LSeg [21] calculates pixel-wise image features  104  
using a convolutional neural network and aligns them with text embeddings from a pre- 105  
trained text model. These methods leverage cross-modal alignment and pre-trained mod- 106  
els to innovate open lexical segmentation.    107  
2.2. Multitask Image Segmentation Architecture  108  
Multitask image segmentation architecture aims to unify various segmentation tasks,  109  
eliminating the need for separate training modes. Leading universal segmentation meth- 110  
ods like MaskFormer [22] treat segmentation as a mask classification problem, excelling  111  
in semantic and panoptic tasks. CLIPSeg [8] offers adaptability to new tasks using text or  112  
image prompts during inference, sparing the cost of retraining. This hybrid approach ac- 113  
commodates referring expression, zero-shot, and one-shot segmentation, informing our  114  
multitask segmentation model.    115  
2.3. Prompt Learning  116  
Prompt learning, originally a prominent concept in natural language processing [22-  117  
24],  has  gained  widespread  recognition  in  vision  and  visual  language  models  [25,  6].  118  
CoOp [25] presents ongoing immediate enhancement of subsequent data to synchronize  119  
with pre-trained visual language models. DenseCLIP [6] fine-tunes the pre-trained text  120  
encoder by employing distinct prompt templates for functions like detection and segmen- 121  
tation, ensuring precise alignment of text and visual features. MAPLE [26] argues that  122  
prompts should be applied in all modes or branches to achieve optimal performance, de- 123  
viating from the conventional practice of applying prompts in only one mode.    124  
In the context of open-vocabulary segmentation tasks, prompt templates are meticu- 125  
lously crafted based on provided category labels and subsequently transformed into text  126  
embeddings. These embeddings are then deployed for alignment with the representations  127  
of unseen classes, facilitating the achievement of remarkable results.    128  
3. Methods  129  
Our proposed framework  is centered on a  unified open-vocabulary segmentation  130  
pardigm aimed at optimizing an all-encompassing model to excel in referring expression,  131  
one-shot, and zero-shot segmentation tasks spanning arbitrary categories. At its core, this  132  
framework is founded upon the CLIP model, serving as the foundational backbone as  133  
depicted in Figure 1. To enhance its capabilities, we have made significant improvements  134  
to both the text prompt and text encoder. Moreover, we have incorporated an efficient  135  
multi-modal interaction module for seamless fusion of image and text features. Addition- 136  
ally, we employ a compact and parametrically efficient transformer decoder to extend the  137  
model’s functionality. We establish a connection between the decoder and the CLIP en- 138  
coder, inspired by the U-Net network structure[27]. Activations at specific layers S are  139  
extracted from the visual encoder and then mapped onto the token embedding size D our  140  
decoder. To guide the decoder in segmenting the target, we employ the FiLM module[28]  141  
to modulate the input activation. The decoder generates binary segmentation by linearly  142  
projecting the output of its last transformer block. In our experiments, we use a projection  143  
dimension of 64. To leverage the semantic information of deep features, we extract the  144  
activation information of CLIP in the transformer module using S=[3,7,9], resulting in a  145  
three-layer decoder. While keeping the improved CLIP encoder frozen, we train the de- 146  
coder to perform the image segmentation task.  147  Electronics 2024, 13, x FOR PEER REVIEW  4  of  14 
 
 
  148  
Figure 1. Architecture of OVAMTSeg: Our enhancement of the static CLIP model(green  149  
and blue) involves adding a transformer that divides the query image according to either  150  
a support image or a text prompt. Fusing text and visual features through a multi-modal  151  
interaction module(gray).    152  
3.1. Adaptive Prompt Learning    153  
Traditional approaches to prompt template design often rely on human linguistic ex- 154  
pertise to craft smooth and effective semantic templates. While such hand-crafted tem- 155  
plates  are  intuitive  and  easy  to  comprehend,  they  come  with  inherent  limitations,  de- 156  
manding substantial manual effort and expertise, thus incurring high costs. In response  157  
to  this  challenge,  we  introduce  an  innovative  adaptive  prompt  learning  module.  This  158  
module transforms diverse classes of text into a set of learnable vectors, which are subse- 159  
quently amalgamated into text embeddings, simplifying the training of the model.    160  
The introduction of adaptive prompts has not only broadened the applicability of our  161  
OVAMTSeg framework to a wider spectrum of unseen categories but has also signifi- 162  
cantly  enhanced  performance  in  open-domain  scenarios.  Specifically,  the  adaptive  163  
prompt P is generated based on the following template, where  [𝑥]   represents semantic  164  
categories or phrases.  165   𝑃 = [𝑉] 1 [𝑉] 2 ⋯ [𝑉] 𝑀 [𝑥][𝑉] 𝑀 +1 ⋯ [𝑉] 𝑁 ,  (1)
Where each [𝑉] 𝑛 (𝑛 ∈ {1,2, . . . , 𝑁}) has the same dimension (512 dimensions) as the word  166  
embedding [𝑥] , is a learnable vector, and  N   is a hyperparameter that specifies the number  167  
of context tokens. During the training, we are given the semantic categories or phrases  168  
involved. These adaptive prompts are then embedded in the pre-trained text encoder Ψ :  169   𝐸 = 𝛹(𝑃([𝑥])) .  (2) 𝐸   represents the obtained text embedding vector. Given the high flexibility of the input  170  
categories or phrases, it can be seamlessly adapted to unseen categories in the Open-Vo- 171  
cabulary segmentation task.  172  
3.2. Feature Extraction Dual-Encoder  173  
Text Encoder.    Text encoders are employed to convert text information into a high  174  
dimensional vector representation, allowing text and images to be compared in the singu- 175  
lar embedded area. The text encoder adopts the ResNet-50 structure. Our modifications  176  
involve replacing the replacement of the global average pooling layer with an attention  177  
pooling mechanism, utilizing multi-head QKV attention. This enhancement enables the  178  
text encoder to effectively capture semantic and contextual features of the text, including  179  
cross-word associations, which improves its semantic representation capacity. The text  180  
encoder embeds  N   possible labels into an unbroken vector space  T ∈ ℝ nt×dt , resulting in  181  Electronics 2024, 13, x FOR PEER REVIEW  5  of  14 
 
 
N   vectors T 1 , T 2 , . . . , T N ∈ ℝ nt×dt   , with their arrangement unaffected by the input label se- 182  
quence. The number N is flexible and can vary freely.  183  
Image Encoder.    Our image encoder utilizes the Visual Transformer It begins  by  184  
converting  the  three-dimensional  image  𝑥 ∈ ℝ H×W×C into  a  two-dimensional  sequence  185   𝑥 1 ∈ ℝ 𝑁×(𝑝2×𝐶) , where (𝐻 × 𝑊 ) represents the original image size,  C   is the channel count,  186  
and  the  image  is  divided  into  evenly  sized  (𝑃, 𝑃)   blocks.  This  results  in  𝑁 = 𝐻𝑊 /𝑃 2      187  
blocks, which also function as the actual length of input sequence for the Visual Trans- 188  
former.  Next,  the  patch  sequence  undergoes  transformation  into  1𝐷   tokens  {𝑓 𝑖𝑣 } 𝑖=1 𝑛   189  
through a trainable linear projection. To capture positional information, positional em- 190  
beddings and an additional  [𝐶 𝐿𝑆] token are introduced. These tokens  𝑓 𝑐 𝑙𝑠 𝑣 , 𝑓 1𝑣 , ⋯ , 𝑓 𝑁𝑣     are  191  
then fed into  𝑆 𝑁  -layer transformer blocks to model correlations among individual patches,  192  
where  𝑁 = 11 . Finally, a linear projection maps  𝑓 𝑐 𝑙𝑠 𝑣   to the combined space of image and  193  
text embedding, acting as the overarching representation of the image.  194  
3.3. Multimodal Interaction Module    195  
In order to achieve complete interaction between image and text modes, we designed  196  
an proficient multi-modal for merging image and text embedding. In contract to other  197  
well-known multi-modal interaction modules, our design of the model enhances the ac- 198  
curacy of image and text correspondence more accurately and improves the semantic un- 199  
derstanding ability of the model. The multi-modal interaction module consists of a multi- 200  
head cross-attention layer and four transformer modules. Given text and supporting im- 201  
ages. The text hidden representation  𝑇 = {𝑡 1 , 𝑡 2 , ⋯ , 𝑡 𝑠 } ,  𝑇 ∈ ℝ 𝑛𝑡×𝑑𝑡   and image hidden rep- 202  
resentation  𝐼 ∗ ∈ ℝ 𝑛𝑖×𝑑𝑖   are obtained by the text and image encoder respectively.  𝐼 ∗ is the  203  
image feature mapping output at the last layer of the image encoder. Since the dimension  204  
of  𝐼 ∗ is different from the dimension of  𝑇 , the hidden representation dimension needs to  205  
be converted to the same dimension as  𝑇 . It can be converted by the following formula:  206   𝐼 = 𝐼 ∗ 𝑊 𝐼 + 𝑏 𝐼 ,  (3)
  207  
Where  𝐼 = {𝑖 1 , 𝑖 2 , ⋯ 𝑖 𝑛𝑖 } ,  𝐼 ∈ ℝ 𝑛𝑖×𝑑𝑖 .  Then,  the  text  𝑇   and  image  𝐼   are  fed  into  the  208  
multi-modal interaction module. To fuse image and text representations more effectively,  209  
the text representation T is used as the query (Q), and the image representation I is used  210  
as the key (K) and value (V). This allows the model to adjust the correlation between pat- 211  
terns, thereby achieving more flexible text-image fusion for different tasks and data. Full  212  
interaction between image and text representations can be achieved by:    213   𝐹 = 𝑇 𝑟 𝑎 𝑛 𝑠 𝑓𝑜𝑟𝑚𝑒𝑟(𝑀 𝐴𝐶(𝑄, 𝐾, 𝑉)) ,  (4)
Where  𝑀 𝐴𝐶 (∙)   is multi-head cross-attention, which can be achieved in the following  214  
ways:    215   𝑀 𝐴𝐶(𝑄, 𝐾, 𝑉) = 𝑠 𝑜 𝑓 𝑡𝑚𝑎𝑥 ( 𝑄𝐾𝑇
√𝑑 ) 𝑉 .  (5)
where d is the embedding dimension of tokens.    216  
4. Experimental Results  217  
4.1. Experimental Settings  218  
The experiments were conducted using the PyTorch [29] deep learning framework  219  
in Python 3.8, within the Anaconda3 environment, and implemented using the PyCharm  220  
compiler. The training of our model consisted of 20,000 iterations, utilizing the PhraseCut+  221  
dataset. We employed a batch size of 64 and set the image size to 352×352 pixels. During  222  
the model’s training process, we employed the AdamW optimization algorithm to itera- 223  
tively optimize the network. The initial learning rate was set to 0.001. As the model ap- 224  Electronics 2024, 13, x FOR PEER REVIEW  6  of  14 
 
 
proaches the global minimum of the loss function, the corresponding learning rate grad- 225  
ually decreases towards a minimum point. To achieve this, we utilized the Cosine An- 226  
nealing method for dynamic learning rate adjustment without warm-up. The minimum  227  
learning rate was set to 0.0001. Simultaneously, we adopted the technique of automatic  228  
mixed precision to enhance the training process. Our model’s loss function exclusively  229  
employed the binary cross-entropy function.  230  
4.2. Datasets    231  
PhraseCut+ is employed as the primary dataset for our training process. It is an ex- 232  
tension of the PhraseCut dataset   [30], encompassing 345,486 phrase regions meticulously  233  
labeled with category names, attributes, and object relationships within the images. Each  234  
phrase corresponds to a binary segmentation mask in the image. PhraseCut+ enhances the  235  
original dataset by introducing visual support samples and negative samples. This aug- 236  
mentation makes it particularly well-suited for models that jointly process text and visual  237  
inputs. The dataset extends the phrase regions using automatically learned templates and  238  
ensures partial object visibility by considering object position information during random  239  
cropping.    240  
Pascal-VOC2012 [31]serves as the basis for our zero-shot image segmentation exper- 241  
iments. Pascal-VOC2012     is a comprehensive dataset featuring images for training, vali- 242  
dation, and testing. The validation subset consists of 1,449 images, each accompanied by  243  
pixel-level semantic segmentation masks used for image segmentation tasks. It includes  244  
annotations for 20 object categories along with a background category. We follow previ- 245  
ous studies’ practice [1, 28] by partitioning these 20 categories into 16 seen classes and 4  246  
unseen classes ("cow," "motorbike," "airplane," and "sofa") to evaluate open word segmen- 247  
tation performance. Additionally, we employ the Pascal-5i dataset [21], a one-shot seg- 248  
mentation dataset derived from Pascal-VOC2012, which contains 20 object classes evenly  249  
distributed across four sections.    250  
For our zero-shot segmentation experiment, we employ the Pascal-Context dataset  251  
[32], which consists of 5,105 validation samples, spanning 59 item classes in addition to a  252  
background class. Similarly to Pascal-VOC, we select four unseen categories ("cow," "mo- 253  
torbike," "sofa," and "cat") from the foreground category for evaluation.    254  
The COCO-20i dataset [33] is commonly used for one-shot segmentation tasks and  255  
is derived from the COCO dataset. It includes annotations for 80 object classes evenly  256  
divided into four sections, each containing 20 categories.    257  
The test results involve selecting the best-performing group from among these four  258  
sections. These datasets provide a diverse and comprehensive foundation for evaluating  259  
the  performance  of  our  proposed  open-vocabulary  and  multitask  image  segmentation  260  
framework.  261  
4.3. Evaluation Metrics.  262  
In our experimental evaluation, we employ several key metrics to assess the perfor- 263  
mance of our model: Mean Intersection over Union ( 𝑚 𝐼𝑜𝑈 ), Intersection over Union for  264  
Foreground ( 𝐼 𝑜𝑈 𝐹𝐺 ), Intersection over Union for Binary Segmentation ( 𝐼 𝑜𝑈 𝐵 𝐼𝑁 ), and Aver- 265  
age Precision ( 𝐴𝑃 ).    266  
Mean Intersection over Union ( 𝑚 𝐼𝑜𝑈 ):  𝑚 𝐼𝑜𝑈  is calculated as the average of the In- 267  
tersection over Union ( 𝐼 𝑜𝑈 ) values across different foreground object categories. It is de- 268  
fined as:    269   𝑚 𝐼𝑜𝑈 = 1
𝐶 ∑ 𝐼 𝑜𝑈 𝐶 𝐶𝐶=1 .  (6)
Where  𝐶   represents the number of object categories, and  𝐼 𝑜𝑈 𝐶   signifies the Intersection  270  
over Union for category  𝐶 . This metric quantifies the degree of overlap between the pre- 271  
dicted segmentation and the ground truth labels for each category, providing an overall  272  
measure of segmentation quality.    273  Electronics 2024, 13, x FOR PEER REVIEW  7  of  14 
 
 
Intersection over Union for Binary Segmentation ( 𝐼 𝑜𝑈 𝐵 𝐼𝑁 ):  𝐼 𝑜𝑈 𝐵 𝐼𝑁  assesses segmen- 274  
tation performance without considering specific object classes. It computes the average of  275  
the  𝐼 𝑜𝑈  values for both foreground and background regions across all test images:    276   𝐼 𝑜𝑈 𝐵 𝐼𝑁 = 1
2 (𝐼 𝑜𝑈 𝐹𝐺 + 𝐼 𝑜𝑈 𝐵𝐺 ) .  (7)
Where  𝐼 𝑜𝑈 𝐹𝐺   represents the  𝐼 𝑜𝑈  for the foreground, and  𝐼 𝑜𝑈 𝐵𝐺   represents the  𝐼 𝑜𝑈  for the  277  
background. This metric provides insights into the segmentation quality when class in- 278  
formation is disregarded.    279  
Average Precision ( 𝐴𝑃 ):  𝐴𝑃  is determined by the area beneath the recall-precision  280  
curve (ROC). The assessment focuses on the model's capacity to differentiate between ac- 281  
curate and inaccurate matches, shedding light on its balance between precision and recall.  282  
4.4. Comparison to State-of-the-art Methods  283  
Referring Expression Segmentation Comparative Experiment. Referring expression  284  
segmentation associates natural language reference expressions with corresponding ob- 285  
jects in the image to perform semantic segmentation on the image. We conducted a com- 286  
prehensive comparison of OVAMTSeg with contemporary state-of-the-art open-vocabu- 287  
lary referring expression segmentation methods, including MDETR [34], HulaNet [35],  288  
Mask-RCNN [35], RMI [35], and CLIPSeg   [8]. The results are summarized in Table1. Our  289  
approach  outperforms  the  two-stage  HulaNet  approach.  However,  the  𝑚 𝐼𝑜𝑈   of  290  
OVAMTSeg is worse than MDETR, which operates at full image resolution and received  291  
two rounds of fine-tuning on PhraseCut. Notably, OVAMTSeg outperforms CLIPSeg with  292  
the same training method in all aspects, indicating that our model is effective.    293  
Table 1. Referring Expression Segmentation performance on PhraseCut+.  294  
Model    mIoU    IoU 𝐹𝐺     AP 
MDETR[34]  53.7  -  - 
HulaNet[35]  41.3  50.8  - 
Mask-RCNN top[35]  39.4    47.4  - 
RMI[35]    21.1    42.5    -   
CLIPSeg[8]    43.4    54.7    76.7   
Ours    47.5    57.1    80.4   
  295  
Zero-Shot Segmentation Comparative Experiment. As shown in Table 2 and Table  296  
3, we compare the open vocabulary zero-shot segmentation performance on Pascal-VOC  297  
and  Pascal-Context  datasets,  including  SPNet[7],  ZS3Net[3],  CSRL[6],  CaGNet[36],  298  
OSR[37], JoEm[38], CLIPSeg[8]   . Tab.2 and Tab.3 can be condened as the following obser- 299  
vations: i) OVAMTSeg achieves 51.6% and 46.6% mIoU towards unseen classes on Pascal- 300  
VOC and Pascal-Context, which goes beyond the best method CLIPSeg and OSR by +4.3%  301  
and +3.5%, respectively. It suggests that OVAMTSeg can be adapted to a wider range of  302  
scenarios. ii) Among the baselines, OSR demonstrates promising results for the seen clas- 303  
ses  but  encounters  challenges  with  respect  to  the  unseen  classes.  In  comparison,  our  304  
model performs well in unseen classes compared to models trained on Pascal-VOC and  305  
Pascal-Context datasets. This variation in performance can be attributed to the fact that  306  
other models are trained on datasets with fixed classes, whereas OVAMTSeg is capable of  307  
discerning a wider range of classes.iii) OVAMTSeg and CLIPSeg perform better on unseen  308  
classes compared to seen classes. This could be because seen classes pose inherent diffi- 309  
culties in segmentation, while unseen classes tend to be relatively larger and easier to seg- 310  
ment.     311  Electronics 2024, 13, x FOR PEER REVIEW  8  of  14 
 
 
Table 2. Performance in zero-shot segmentation using Pascal-VOC with 4 unseen classes.  𝑚 𝐼 𝑜𝑈𝑠   312  
and  𝑚 𝐼 𝑜𝑈𝑢  denote the mIoU(%) of classes that are observed and those that are not. The training of  313  
our model occurs on PhraseCut+ with the Pascal classes removed. The term IN-seen refers to the  314  
pre-training phase of ImageNet where previously unseen classes are eliminated.  315  
Model    pre-train    mIoU 𝑠     mIoU 𝑢  
SPNet[7]  IN  67.3  21.8 
ZS3Net[3]    IN-seen  66.4  23.2 
CSRL[6]  IN-seen  69.8  31.7 
CaGNet[36]  IN  69.5  40.2 
OSR[37]  IN-seen    75.0  44.1 
JoEm[38]    IN-seen    67.0    33.4 
CLIPSeg[8]    CLIP    20.8    47.3   
Ours    CLIP  28.3                51.6 
Table 3. Zero-shot segmentation performance on Pascal-Context with 4 unseen classes. Our model  316  
is trained on PhraseCut+ with the Pascal classes removed.  317  
Model    pre-train    mIoU 𝑠     mIoU 𝑢  
SPNet[7]  IN  36.3    18.1 
ZS3Net[3]    IN-seen  37.2    24.9 
CSRL[6]  IN-seen  39.8      23.9 
CaGNet[36]  IN  24.8    18.5 
OSR[37]  IN-seen    41.1      43.1 
JoEm[38]    IN-seen    36.9      30.7 
CLIPSeg[8]    CLIP    16.8      40.2 
Ours    CLIP  25.3    46.6 
  318  
One-Shot  Segmentation  Comparative  Experiment.  Differing  from  zero-shot  seg- 319  
mentation, one-shot segmentation requires the understanding of both text prompts and  320  
annotated  support  images.  To  aid  in  this  comprehension,  we  used  the  same  visual  321  
prompts methods as CLIPSeg, including object cropping, background blur, and darken- 322  
ing. We evaluate the proposed model on PASCAL-5i, COCO-20i, and compare the results  323  
with recent methods. To ensure fairness during training, classes that overlap with these  324  
datasets are removed. As shown in Table 4, OVAMTSeg achieves 85.6% AP, outperform- 325  
ing CLIPSeg by 4.3%, respectively. Table 5 presents a performance comparison when the  326  
model is trained on the COCO-20i dataset. Similarly, we note consistent findings where  327  
our model performs effectively. OVAMTSeg also attains the highest results with an AP of  328  
86.6%. The COCO-20i results show that OVAMTSeg also performs well when trained on  329  
other datasets than PhraseCut+. Particularly, HSNet and PFENet exhibit superior perfor- 330  
mance in the performance metric of mIoU, which can be attributed to their explicit design  331  
for one-shot segmentation.    332  
Tian et al. [41] replace visual samples in the network with textual label word vectors  333  
and employ PFENet for zero-shot segmentation, adhering to the protocol of zero-shot seg- 334  
mentation. and utilize PFENet for zero-shot segmentation following the one-shot segmen- 335  
tation  protocol.  In  this  context,  OVAMTSeg  significantly  surpasses  their  performance  336  
scores, as evidenced in Table 6. This indicates the challenge in applying one-time oriented  337  
techniques such as PFENet to different tasks, in contrast to our OVAMTSeg, which exhib- 338  
its significant generalization potential.  339  
Table 4. One-shot performance on Pascal-5i.  340  
Model    vis.backb.    mIoU    IoU 𝐵 𝐼𝑁     AP 
PPNet[39]    RN50  52.8  69.2  - 
RePRI[40]  RN50  59.1    -  - Electronics 2024, 13, x FOR PEER REVIEW  9  of  14 
 
 
PFENet[41]  RN50  60.8  73.3  - 
HSNet[42]  RN50  64.0  76.7  - 
MGNet[43]    RN50    52.1    68.2  - 
HCNet[44]    RN50    62.1    71.7  - 
DRNet[45]    RN50    53.3    72.8    -   
SRPNet[46]    RN50    61.5    -    -   
CLIPSeg[8]    ViT(CLIP)    59.5    75.0  82.3 
Ours    ViT(CLIP)  65.9  77.1  86.8 
Table 5. One-shot performance on COCO-20i(OVAMTSeg trained on COCO-20i).  341  
Model    vis.backb.    mIoU    IoU 𝐵 𝐼𝑁     AP 
PPNet[39]    RN50  29.0  -  - 
RePRI[40]  RN50  34.0    -  - 
PFENet[41]  RN50  35.8  -  - 
HSNet[42]  RN50  39.2  68.2  - 
MGNet[43]    RN50    34.9    63.9  - 
HCNet[44]    RN50    40.7                63.4  - 
DRNet[45]    RN50    36.5    60.9    -   
CLIPSeg[8]    ViT(CLIP)    33.2    58.4  40.5 
Ours    ViT(CLIP)  35.7  62.5  46.3 
Table 6. Zero-shot performance on Pascal-5i. Scores were derived by adhering to the one- shot seg- 342  
mentation assessment method.  343  
Model    vis.backb.    mIoU    IoU 𝐵 𝐼𝑁   AP 
PFENet[41]    VGG16    54.2    -    -   
LSeg[21]    ViT(CLIP)    52.3    67.0    -   
CLIPSeg[8]    ViT(CLIP)    72.4    83.1  93.5 
Ours    ViT(CLIP)  78.5  87.3  93.8 
4.5. Ablation Study  344  
Adaptive Prompt Analysis. We evaluate various prompt configurations to assess the  345  
importance of the adaptive prompt in the context of open vocabulary segmentation, as  346  
presented in Table 7. The static template prompt utilizes the sentence structure "A photo  347  
of class," where "class" is replaced with specific class names. These class-specific prompts  348  
are then encoded into the text features. As shown in Table 7, the adaptive prompt yields  349  
a noteworthy enhancement of 2.2% mIoU and 2.3% AP performance for referring expres- 350  
sion segmentation, respectively, when compared to the fixed prompt. Similarly, the adap- 351  
tive prompt leads to 2.6% mIoU performance improvement tover the fixed prompt for  352  
unseen classes in zero-shot segmentation, and achieves a lead of 2.6% mIoU and 1.2% AP  353  
for one-shot segmentation. This underscores the role of the adaptive prompt in facilitating  354  
the capture of category-sensitive concepts through learnable parameters.    355  
Multi-Task Analysis. To assess the benefits of the multitask approach in OVAMTSeg,  356  
we performed a comparative examination by comparing the model's performance with  357  
single-task training for individual tasks. As depicted in Table 7, the outcomes under the  358  
multitask category are derived from a single unified model, while the single-task results  359  
are obtained from three separate individual models. Our model achieves remarkable re- 360  
sults, with a 47.5%  𝑚 𝐼𝑜𝑈  and an 80.4% AP on referring expression segmentation, surpas- 361  
sing the performance of the one-shot counterparts. Furthermore, open-vocabulary zero- 362  
shot and one-shot segmentation exhibit consistent and impressive results, particularly in  363  
terms of performance on unseen classes. OVAMTSeg consistently improves metrics across  364  Electronics 2024, 13, x FOR PEER REVIEW  10  of  14 
 
 
almost all tasks, highlighting the effectiveness of the multi-task training approach in en- 365  
hancing network generalization and surpassing the performance of one-shot models.  366  
Text Encoders. OVAMTSeg inherently accommodates various text encoders. We il- 367  
lustrate the impact of employing different text encoders in Table 7. It is important to high- 368  
light that all text encoders feature the same transformer-based architecture that purely  369  
operates on text prompts. The main difference between the encoders is the image encoder  370  
that was paired during CLIP pre-training (for example, the text encoder denoted by "ViT- 371  
B/32" was trained in conjunction with a ViT-B/16 image encoder) and the size of the em- 372  
bedding dimension. We observed that employing RN50×16 achieves the highest perfor- 373  
mance among all text encoders. We hypothesize that this is due to the larger embedding  374  
dimension provided by this encoder.  375  
Table 7. Comparison of various task paradigms, prompt strategies, and text encoders. Evaluation  376  
includes Referring Expression Segmentation performance on PhraseCut+, zero-shot segmentation  377  
performance on Pascal-VOC, and one-shot performance on Pascal-5i.  378  
Method  Referring Ex-
pression 
Zero-Shot  One-Shot 
mIoU  AP  mIoUs  mIoUu  mIoU  AP 
Prompt  Fixed  45.4  78.1  57.8  49.0  63.3  85.4 
Adaptive  47.5  80.4  28.3  51.6  65.9  86.6 
Task 
Para-
digm 
Single-Task  41.3  -  75.0  44.1  64.0  - 
Multi-Task  47.5  80.4  28.3  51.6  65.9  86.6 
Text En-
coder 
ViT-B/32(512)  43.6  79.0  25.5  49.2  63.5  85.2 
ViT-B/16 (512)  44.4  79.3  26.0  49.8  64.1  85.4 
RN50 × 4(640)  44.8  79.4  26.2  50.1  64.5  85.6 
RN50 × 16(768)  46.2  79.8  27.4  51.0  65.6  85.9 
  379  
Component Analysis. We conducted ablation studies on OVAMTSeg, and the re- 380  
sults are presented in Table 8. When the model operates without any adaptive prompt,  381  
text encoder extension, and replaces the attention-based multi-modal interaction with a  382  
simple concatenation sum, its performance across various metrics is notably subpar. The  383  
introduction of adaptive prompt learning significantly enhances the model’s performance,  384  
particularly in the context of unseen classes during one-shot segmentation. Subsequently,  385  
the text encoder extension and multi-modal interaction components are gradually incor- 386  
porated into the framework. This incremental integration yields performance improve- 387  
ments across various tasks during inference. These results underscore the critical contri- 388  
butions of each component. Adaptive prompt learning facilitates OVAMTSeg in captur- 389  
ing category-sensitive characteristics, while the text encoder extension aids in effectively  390  
capturing textual features. Additionally, the multi-modal interaction component plays a  391  
pivotal role in enhancing the cross-modal alignment of visual and text features.    392  
Table 8. Ablation studies were conducted to analyze the effectiveness of the proposed modules.  393  
Referring Expression Segmentation performance on PhraseCut+. Zero-shot segmentation perfor- 394  
mance on Pascal-VOC, and one-shot performance on Pascal-5i.  395  
Adap-
tive 
Prompt 
Text En-
coder Exten-
sion 
Multi-
modal In-
teraction 
Referring Ex-
pression 
Zero-Shot  One-Shot 
mIoU  AP  mIoUs  mIoUu  mIoU  AP 
✗  ✗  ✗  43.6  72.8  23.1  25.3  60.2  78.1 
✓  ✗  ✗  45.9  77.6  25.9  48.8  63.7  83.5 
✓  ✓  ✗  46.6  78.3  26.8  49.9  64.6  84.3 
✓  ✓  ✓  47.5  80.4  28.3  51.6  65.9  86.6 
  396  Electronics 2024, 13, x FOR PEER REVIEW  11  of  14 
 
 
4.6. Qualitative Results    397  
Figure 2 presents qualitative results, highlighting the notable differences in predic- 398  
tion accuracy between OVAMTSeg and CLIPSeg for identical text prompts. For example,  399  
in the second image where the text prompt is "glass," OVAMTSeg’s predictions exhibit  400  
precise  delineation  of  the  two  glasses,  accurately  capturing  their  outlines.  In  contrast,  401  
CLIPSeg’s predictions result in a less refined separation of the glasses’ shapes. Similarly,  402  
when the text prompt is "bottle," OVAMTSeg correctly identifies and predicts two small  403  
bottles, demonstrating its ability to discern subtle distinctions. However, CLIPSeg’s pre- 404  
dictions in this scenario are less accurate, as it erroneously identifies both the wine glass  405  
and the bottle. Despite these successes, certain challenges persist. In the second image,  406  
where the text indicates "fork," both OVAMTSeg and CLIPSeg incorrectly predict the knife  407  
to be a fork, in addition to accurately predicting the strengths and  limitations  of both  408  
OVAMTSeg and CLIPSeg in their responses to specific text prompts.the presence of an  409  
actual fork. These visual comparisons underscore the strengths and limitations of both  410  
OVAMTSeg and CLIPSeg in their responses to specific text prompt.  411  
                        412  
  413  
Figure 2. Qualitative results of the multi-task open vocabulary segmentation. We compare  414  
the segmentation results of the proposed OVAMTSeg and CLIPSeg.  415  
5. Conclusions    416  
In this paper, we have presented OVAMTSeg, a universal framework designed to  417  
excel in multitask open-vocabulary segmentation. Our investigation delved into a novel  418  
text prompt method, showcasing competitive performance across a spectrum of tasks, in- 419  
cluding referring expression, zero-shot, and one-shot image segmentation. Furthermore,  420  
we have enhanced the text encoder to bolster its semantic representation capabilities. Ad- 421  
ditionally, we introduced a multi-modal interaction module that dynamically adjusts the  422  
correlation between modalities. This adaptation enables more flexible fusion of textual  423  
and image features across diverse tasks and datasets. Our contributions stand as a testa- 424  
ment to the potential of open-vocabulary and multitask segmentation. We are convinced  425  
that our work not only provides valuable insights but also outlines a promising direction  426  
for future research in this dynamic field. 

---
Act as AI expert in mathematics. Write professional-garage summarization and expert recenze. 
Find few advantages of proposed approach.
Special attention focus on disadvantages/cons of proposed approach