Multi‐similarity based hyperrelation network for few‐shot segmentation

IET Image ProcessingVolume 17, Issue 1 p. 204-214 ORIGINAL RESEARCHOpen Access Multi-similarity based hyperrelation network for few-shot segmentation Xiangwen Shi, Xiangwen Shi Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China University of Chinese Academy of Sciences, Beijing, China Contribution: Formal analysis, ​Investigation, Methodology, Software, Validation, Visualization, Writing - original draftSearch for more papers by this authorZhe Cui, Corresponding Author Zhe Cui [email protected] Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Correspondence Zhe Cui, Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu 610081, China. Email: [email protected] Xianghong Tang, State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China. Email: [email protected] Contribution: Funding acquisition, Methodology, Validation, Writing - review & editingSearch for more papers by this authorShaobing Zhang, Shaobing Zhang Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: Methodology, Validation, Writing - review & editingSearch for more papers by this authorMiao Cheng, Miao Cheng Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: ​Investigation, Validation, Writing - review & editingSearch for more papers by this authorLian He, Lian He Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: Methodology, Validation, Writing - review & editingSearch for more papers by this authorXianghong Tang, Corresponding Author Xianghong Tang [email protected] State Key Laboratory of Public Big Data, Guizhou University, Guiyang, China Correspondence Zhe Cui, Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu 610081, China. Email: [email protected] Xianghong Tang, State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China. Email: [email protected] Contribution: ​Investigation, Writing - review & editingSearch for more papers by this author Xiangwen Shi, Xiangwen Shi Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China University of Chinese Academy of Sciences, Beijing, China Contribution: Formal analysis, ​Investigation, Methodology, Software, Validation, Visualization, Writing - original draftSearch for more papers by this authorZhe Cui, Corresponding Author Zhe Cui [email protected] Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Correspondence Zhe Cui, Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu 610081, China. Email: [email protected] Xianghong Tang, State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China. Email: [email protected] Contribution: Funding acquisition, Methodology, Validation, Writing - review & editingSearch for more papers by this authorShaobing Zhang, Shaobing Zhang Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: Methodology, Validation, Writing - review & editingSearch for more papers by this authorMiao Cheng, Miao Cheng Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: ​Investigation, Validation, Writing - review & editingSearch for more papers by this authorLian He, Lian He Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu, China Contribution: Methodology, Validation, Writing - review & editingSearch for more papers by this authorXianghong Tang, Corresponding Author Xianghong Tang [email protected] State Key Laboratory of Public Big Data, Guizhou University, Guiyang, China Correspondence Zhe Cui, Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu 610081, China. Email: [email protected] Xianghong Tang, State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China. Email: [email protected] Contribution: ​Investigation, Writing - review & editingSearch for more papers by this author First published: 30 September 2022 https://doi.org/10.1049/ipr2.12628AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Few-shot semantic segmentation aims at recognizing the object regions of unseen categories with only a few annotated examples as supervision. The key to few-shot segmentation is to establish a robust semantic relationship between the support and query images and to prevent overfitting. In this paper, an effective multi-similarity hyperrelation network (MSHNet) is proposed to tackle the few-shot semantic segmentation problem. In MSHNet, a new generative prototype similarity (GPS) is proposed, which, together with cosine similarity, establishes a strong semantic relationship between supported images and query images. In addition, a symmetric merging block (SMB) in MSHNet is proposed to efficiently merge multi-layer, multi-shot, multi-similarity features to generate hyperrelation features for semantic segmentation. Experimenting on two benchmark semantic segmentation datasets (Pascal − 5i and COCO − 20i) shows that this method achieves a mean intersection-over-union score of 72.3% and 56.0%, respectively, which outperforms the state-of-the-art methods by 1.9% and 6.5%. 1 INTRODUCTION The image semantic segmentation aims to classify each pixel of an image into a specific class. It is important fundamental processing in medical analysis and industry defect detection. With the great success of deep learning, image semantic segmentation has made great progress, such as FCN [4], U-Net [7], DeepLab [9], and so on. But training these segmentation models requires a lot of pixel-wise annotated images, which is tedious and costly. Moreover, a trained model can only make predictions within a set of pre-defined classes. To perform semantic segmentation for new categories with only a few labelled samples, many few-shot semantic segmentation methods [24-27] have been proposed. Most of the few-shot segmentation methods are based on two-branch architecture, as shown in Figure 1, with a support branch and a query branch. Support images are labelled with corresponding categories, and query images are segmented according to the labels of the support image. The two-branch structure uses a pre-trained convolutional neural network (CNN) to extract the high-level features of support and query images, and then uses a relational transformation block to establish the semantic relationship between support and query images to perform semantic segmentation on query images. The key to few-shot semantic segmentation is to find the robust relationship between support images and query images in high-dimensional features. FIGURE 1Open in figure viewerPowerPoint Two branches few-shot semantic segmentation structure There are two ways to establish the semantic relationship between support and query images. The first method [24, 25] is that concatenate support features with query feature-map in deep layers to generate the relationship. Another way [26, 27] is to seek the cosine similarity between the deep query feature and support feature as a prior probability, and then concatenate it with the query feature map to segment the query image. However, objects are diverse, and it is difficult to describe the relationship between the query and support images robustly using only just one relational function. The HSNet [30] proposed by Min et al. tells us that using more relations between feature layers is helpful to establish a robust relationship between support and query images. Motivated by these, we propose a multi-similarity hyperrelation network (MSHNet) to address few-shot semantic segmentation. First, we use generative prototype similarity and cosine similarity to construct a multi-layer and multi-scale high-dimensional spatial relationship between the query and support images. Compared with feature-based methods [24, 38], our prototype-similarity is universal and can generalize models to new categories better. We then use symmetric merging block (SMB) to merge these multiple relationships and establish robust hyperrelation for the support and query images. The locally generated prototype similarity based on global feature is logically complementary to the global cosine similarity based on local feature, and the relationship between the query image and the supported image can be expressed more comprehensively by using the two similarities simultaneously. In summary, MSHNet is based on the similarity relationship, rather than the features of the query image, which reduces the impact of the bias between train and test classes. In addition, the network efficiently integrates the multi-layer and multi-scale relationship features, which can better solve the problem of the diversity of object shapes and sizes. We experiment with the proposed method on Pascal − 5i dataset and COCO − 20i dataset, and we achieve a new the-state-of-art on both datasets. Through experiments, we find that even if we only use one of the two similarities, we can get results close to the existing optimal methods. The contributions of this paper can be summarized as follows. We propose a new prototype similarity method, which can effectively describe the similarity between query features and support vectors. We use multiple similarities to establish a more robust relationship between support and query images. We proved through experiments that the proposed prototype similarity and cosine similarity are complementary. The proposed symmetric merging block (SMB) module is not only simple but also efficiently merges the similarity feature of multi-layer, multi-shot, and multi-similarity. MSHNet achieves a new state-of-the-art performance on both Pascal − 5i[1] and COCO − 20i[32] datasets. 2 RELATED WORK In this chapter, we briefly review some work related to this paper, such as few-shot learning, image semantic segmentation, few-shot semantic segmentation, and so on. Image semantic segmentation Image semantic segmentation is an essential component in many visual understanding systems, such as autonomous vehicles, and medical image diagnostics. The goal of image semantic segmentation is to classify each pixel of an image into a set of predefined semantic classes. Long et al. [4] proposed fully convolutional networks which trained end-to-end, and pixels-to-pixels on semantic segmentation exceed the previous best results. Encoder-decoder [5-7] is another popular semantic segmentation architecture. The encoder uses convolution and down-sampling to obtain deep features of different scales of the input image, while the decoder merges and upsamples these features to obtain the final semantic segmentation result. The DeepLab family [8-10] uses atrous convolution and atrous spatial pyramid pooling (ASPP) to merge the multi-scale information, which can address the decreasing resolution in the encoder and robustly segment objects at multiple scales. Although these methods can achieve image semantic segmentation well, they need a large number of pixel-level annotated images to supervise the network, and the trained model cannot be used for new categories. Few-shot learning for image classification Few-shot image classification [11-14] aims to classify new images with a few examples. Meta-learning [3, 19–23] is a common method to solve the few-shot problem. It iteratively trains few-shot tasks on the train set to learn meta-knowledge and adapt the model to the situation with only a few samples. Many meta-learning methods have been proposed based on metric learning and optimization algorithms. Metric-based methods [15-19] aim to learn a general metric function to infer the distances between the query image and samples. The optimization-based [20-23] method aims to design effective learning strategies for few-shot tasks. For example, Chelsea et al. [20] proposed a good parameter initialization learning method, which can make the model adapt to new tasks quickly. Most of the few-shot learning methods are developed for other image tasks. Few-shot semantic segmentation Few-shot segmentation [24–27, 30] is the extension of meta-learning methods, which can perform semantic segmentation within a few labelled samples. This research problem was introduced by Shaban et al. [1], who proposed a classical two-branch network. The most challenge for few-shot segmentation is how to design a transformation block and establish a class-agnostic relationship between the support and query images. The relationship is mainly based on generative similarity and cosine similarity. The generative-based methods [24, 25, 38] directly concatenate the support and query features into the merge block to estimate the prediction. For example, Zhang et al. [24] upsample the prototype feature vector from the support set and concatenate it with the query feature for dense predictions. However, the training category is different from the test category. Using a large number of parameters to express the relationship between the support and query features will result in overfitting. In the cosine-based methods, refs. [26, 27] compute the cosine similarity between the prototypes vector of support feature and pixels of query feature, and refs. [28, 41] introduce a graph attention mechanism to establish pixel-to-pixel similarity between support and query features. However, these models only use a few feature layers, so it is difficult to establish a robust relationship between the support and query features. Recently, refs. [30, 46] used 4D convolution to establish the hyperrelation between multi-layer features, but 4D convolution has high spatial complexity and time complexity. In this paper, we use multi-similarity to build a more robust semantic relationship between support and query images. 3 PROBLEM DESCRIPTION The few-shot segmentation aims to learn a model to perform segmentation on novel classes with a few pixel-level annotated samples. In the K-shot segmentation task, each class has K pixel-level labelled samples. Suppose we have two non-overlapping data sets Dtrain and Dtest. The train set Dtrain is constructed by classes Ctrain and the test set Dtest is constructed by Ctest, where Ctrain and Ctest are non-overlapping ( C t r a i n ∩ C t e s t = ∅ ${C}_{train} \cap {C}_{test} = \emptyset $ ). The model only uses the Dtrain to train the network, and the categories in the Ctest are not involved in the training process. Following the previous few-shot methods [2, 30], we train and test the model in episodes. In each episode, we randomly sample a support set S = ( X s , M s ) $S = (X_s,{M}_s)$ and a same class query set Q = ( X q , M q ) $Q = ({X}_q,{M}_q)$ , where X ∈ R 3 × H × W $X \in {R}^{3 \times H \times W}$ indicates the RGB image and M ∈ { 0 , 1 } H × W $M \in {\{ 0,1\} }^{H \times W}$ is the corresponding binary segmentation masks. The support set S consists of N classes and each class contains K samples S = { ( x c 1 , m c 1 ) … ( x c K , m c K ) } c = 1 N $S = {\{ (x_c^1,m_c^1)\ldots(x_c^K,m_c^K)\}}_{c = 1}^N$ . In this paper, we train and tested on 1-shot and 5-shot tasks. We are supposed to learn a mapping function M ̂ = f θ { X q | ( X s , M s ) } $\hat{M} = {f}_\theta \{ {X}_q|({X}_s,{M}_s)\} $ on Dtrain and evaluate it on Dtest, where θ is the parameter of the model and M ̂ $\hat{M}$ is the predicted mask. Once the mapping function is learned, the parameters in the model are fixed and we do not optimize the model on the Dtest. 4 OUR METHOD OVERVIEW We propose a novel few-shot semantic segmentation architecture, multi-similarity hyperrelation network (MSHNet), which is based on generative prototype similarity and cosine similarity. The overall architecture is illustrated in Figure 2, which is built upon a two-branch architecture. The support branch and query branch share a CNN block (pretrained on ImageNet [40]) which is used to obtain the high-dimensional semantic features of the input images. We then used these high-level features to calculate the prototype similarity and cosine similarity between the support and query images. Each symmetrical merging block (SMB) takes the prototype similarity and cosine similarity as input to generate hyperrelation features for semantic segmentation. To reduce the risk of overfitting, we train the model in episode [34]. The overview of the whole process is provided in Algorithm 1. FIGURE 2Open in figure viewerPowerPoint Overview of MSHNet framework. We extract image features by Resnet-50 or Resnet-101 in multi-scale and layers. In each symmetric merging block, we get the layer relationship of the same scale between the support image and query image. Finally, we merge the hyperrelational features of different scales with pyramid structures In each episode, we first input the random sampled X s ∈ R 3 × H × W ${X}_s \in {R}^{3 \times H \times W}$ and X q ∈ R 3 × H × W ${X}_q \in {R}^{3 \times H \times W}$ to support and query branch respectively. The pre-trained CNN block extracts multi-scale support deep features F s b , l ∈ R d l × h b × w b $F_s^{b,l} \in {R}^{{d}_l \times {h}_b \times {w}_b}$ and query deep features F q b , l ∈ R d l × h b × w b $F_q^{b,l} \in {R}^{{d}_l \times {h}_{\rm{b}} \times {w}_b}$ , where d l ${d}_l$ is the feature dimensionality of layer l, and w b ${w}_b$ , h b ${h}_b$ denote the width and height of feature maps in the block b. Then we apply the mask to each support feature map to get object features F s b , l , o $F_s^{b,l,o}$ and use average pooling to obtain the class prototype vector P s b , l , c $P_s^{b,l,c}$ of the annotated object, which is described in Section 4.1. Next, we apply CS() to get pixel-to-pixel cosine similarity between F s b , l , o $F_s^{b,l,o}$ and F q b , l $F_q^{b,l}$ , and apply GPS() to get prototype similarity between P s b , l , c $P_s^{b,l,c}$ and F q b , l $F_q^{b,l}$ , which is described in Section 4.2. In the end, the transformation block merges the multi-layer and multi-scale features, as described in Section 4.3. In this paper, we use ResNet-50 [31] and ResNet-101 [31] as the CNN backbone to extract the feature of support images and query images. We select the standard cross-entropy loss L ( m ̂ q , m q ) $L({\hat{m}}_q,{m}_q)$ as our loss function. As shown in Figure 2, each symmetric merging block of the transformation block has an intermediate output and an intermediate loss L i n n e r i ( i ∈ { 1 , 2 , 3 } ) $L_{{\rm{i}}nner}^i(i \in \{ 1,2,3\} )$ to help optimize the network. The final prediction of MSHNet generates another loss function L o u t ${L}_{{\rm{o}}ut}$ . The total loss L is the sum of Linnerand L o u t ${L}_{{\rm{o}}ut}$ as: L = 1 n ∗ ∑ i = 1 n L i n n e r i + L o u t $$\begin{equation}L = \frac{1}{n} {*}\sum_{i = 1}^n {L_{inner}^i} + {L}_{out}\end{equation}$$ (1)where n is the number of symmetric merging blocks. The output of each symmetric merging block is segmentation at a specific scale, and the final output of the network merges the segmentation results at a variety of scales. The model parameters are only updated during training and fixed during testing. Object feature extractor and prototype The ResNet [31] is composed of many layers, and each convolutional layer has a specific convolution kernel to generate feature maps. In general, the more pairs of intermediate feature maps { ( F s b , l , F q b , l ) l = 0 L } b = 1 B ${\{ {({\rm{F}}_s^{b,l}{\rm{,F}}_q^{b,l})}_{l = 0}^L \}} _{b = 1}^B$ have been used, the relationship between support and query image is more robust. To balance computational complexity and relational complexity, we only select the features marked in the support image to calculate similarity, as F s b , l , o = F s b , l M s b , l ( x , y ) = c ( x = 1 , y = 1 ) ( x = h , y = w ) $$\begin{equation}F_s^{b,l,o} = \left\{ {F_s^{b,l}\left[ {M_s^{b,l}(x,y) = c} \right]} \right\}_{(x = 1,y = 1)}^{(x = h,y = w)}\end{equation}$$ (2)where (x, y) is the spatial position of the feature map; [] is a selector function, if true, the index value is selected, otherwise discarded; F s b , l , o ∈ R n × d l $F_s^{b,l,o} \in {R}^{n \times {d}_l}$ . The prototype vector of the specific class object is computed by: P s b , l , c = F s b , l , o ∑ i = 0 F s b , l , o , i F s b , l , o if F s b , l , o > 0 O otherwise $$\begin{equation} P_s^{b,l,c} = \left\{ { \def\eqcellsep{&}\begin{array}{l} {\left| {F_s^{b,l,o}} \right|} \\[12pt] {\dfrac{{\sum_{i = 0}^{F_s^{b,l,o,i}} }}{{\left| {F_s^{b,l,o}} \right|}}\quad{\rm{ if }}\left| {F_s^{b,l,o}} \right|{ > 0}}\\[20pt] {\mathrm O}\qquad\qquad{\rm{ otherwise}} \end{array} } \right. \end{equation}$$ (3)where |*| means the number of pixel features in F s b , l , o $F_s^{b,l,o}$ ; F s b , l , o , i $F_s^{b,l,o,i}$ is the i-th value in the support feature set. Prototype similarity and cosine similarity Because the class of the test set is different from that of the train set, the feature-based methods [27, 38, 47] tend to be associated with specific category features. We reduce this bias by class-independent similarity, and use only a few parameters in prototype similarity to further reduce the influence of overfitting. We upsample the prototype vector P s b , l , c $P_s^{b,l,c}$ to the size of query feature map and concatenate it to F q b , l $F_q^{b,l}$ , and then we learn a projection function to get the generative prototype similarity (GPS) between prototype and query features, as G P S ( x p , x q ) = s i g m o i d ( W ∗ [ x p | | x q ] ) $$\begin{equation}GPS({x}_p,{x}_q) = sigmoid(W*[{x}_p{\rm{||}}{x}_q])\end{equation}$$ (4)where x p ${x}_p$ = P s b , l , c $P_s^{_{b,l,c}}$ is the class prototype vector of the annotated object, x q ${x}_q$ is a vector in query feature map, and W ∈ R 1 × 2 d $W \in {R}^{1 \times 2d}$ is a full connection layer weight. The output of GPS is a one-dimensional feature map, and the value represents the probability that the position is the target class. Objects in nature come in a variety of shapes and sizes that are difficult to represent with just a global average prototype. We use the cosine similarity to establish a more detailed relation between query and support in pixel by pixel. The cosine similarity between each point in query feature map and support features is defined as below: C S ( x q , x s ) = ∑ i = 1 F s b , l , o Re L U ( x q T · x s , i x q x s , i ) F s b , l , o $$\begin{equation}CS({x}_q,{x}_s) = \frac{{\sum_{i = 1}^{\left| {F_s^{b,l,o}} \right|} {{\mathop{\rm Re}\nolimits} LU(\frac{{x_q^T \cdot {x}_{s,i}}}{{\left\| {{x}_q} \right\|\left\| {{x}_{s,i}} \right\|}})} }}{{\left| {F_s^{b,l,o}} \right|}}\end{equation}$$ (5)where x q ${x}_q$ is a vector of a point in F q b , l $F_q^{b,l}$ , x s , i ${x}_{s,i}$ is the i-th value in x s = F s b , l , o ${x}_s = F_s^{b,l,o}$ , and | F s b , l , o | $| {F_s^{b,l,o}} |$ is the number of elements in F s b , l , o $F_s^{b,l,o}$ . As done in [30, 46], we use the ReLU() function to make the network focus only on how the query point is similar to the specific class object in the supporting image, rather than how they are different. The CS() describes the similarity between each point in query feature map and all feature points in the support set. After the prototypical similarity and cosine similarity of each layer were obtained, we used a symmetrical merge block to refine the two similarities between the same scale layers. Symmetric merging block To merge the two similarities more consistently and stably, we propose a two-branch symmetric merging block as shown in Figure 3. The input prototype similarity and cosine similarity are one-dimensional feature maps, which were calculated from GPS and CS functions respectively. In Block-Conv, a 1 × 1 convolution layer was used to integrate similarity features from multiple layers, and then a 3 × 3 convolution kernel was used to optimize similarity features within the local range. In prototype Shot-Conv, we use multi-scale atrous convolution to obtain the similarity features of different scales in a larger range. Since cosine similarity is stable, we only use a 3 × 3 convolution layer to merge features in cosine Shot-Conv. In Merge-Conv we use the channel-attention mechanism [29] to highlight the layers of similarity we need to focus on. The symmetric merging block (SMB) merges two similarities and outputs hyperrelational features for semantic segmentation. ALGORITHM 1. Testing an episode for a K-shot semantic segmentation task Input: Support set S, Support mask M s ${M}_s$ and query set Q Extract query deep features F q b , l $F_q^{b,l}$ from Q for b = 2 , . . , B $b = 2,..,B$ do Extract support deep features F s , k b , l $F_{s,k}^{b,l}$ from S k ${S}_k$ for k = 1 , … , K $k = 1,\ldots,K$ do for l = 1 , … , L $l = 1,\ldots,L$ do Apply M s , k ${M}_{s,k}$ to obtain F s , k b , l , o $F_{s,k}^{b,l,o}$ by Equation (2) Obtain P s , k b , l , c $P_{s,k}^{b,l,c}$ from F s , k b , l , o $F_{s,k}^{b,l,o}$ by Equation (3) Compute the generative prototype similarity G P S k b , l $GPS_k^{b,l}$ from ( F q b , l $F_q^{b,l}$ , P s , k b , l , c $P_{s,k}^{_{b,l,c}}$ ) by Equation (4) Compute the cosine similarity C S k b , l $CS_k^{b,l}$ from ( F q b , l $F_q^{b,l}$ , F s , k b , l , o $F_{s,k}^{b,l,o}$ ) by Equation (5) end Concatenate [ G P S k b , l $GPS_k^{b,l}$ ,…, G P S k b , L $GPS_k^{b,L}$ ] and input it to Psimilarity Block Conv to get b l c _ G P S k b $blc\_GPS_k^b$ Concatenate [ C S k b , l $CS_k^{b,l}$ ,…, C S k b , L $CS_k^{b,L}$ ] and input it to Csimilarity Block Conv to get b l c _ C S k b $blc\_CS_k^b$ end Compute the average of [ b l c _ G P S k b $blc\_GPS_k^b$ ,…, b l c _ G P S K b $blc\_GPS_K^b$ ] and input it to Psimilarity Shot Conv to get s h o t _ G P S b $shot\_GP{S}^b$ Compute the average of [ b l c _ C S k b $blc\_CS_k^b$ ,…, b l c _ C S K b $blc\_CS_K^b$ ] and input it to Csimilarity Shot Conv to get s h o t _ C S b $shot\_C{S}^b$ Input ( s h o t _ G P S b $shot\_GP{S}^b$ , s h o t _ C S b $shot\_C{S}^b$ ) to Block Merge Conv to get hyperrelation h y p e r _ s i m b $hyper\_si{m}^b$ end Merge the multi-scale hyperrelational features [ h y p e r _ s i m b $hyper\_si{m}^b$ ,.., h y p e r _ s i m B $hyper\_si{m}^B$ ] to get segmentation result. FIGURE 3Open in figure viewerPowerPoint We proposed symmetric merging block. The upper part is used for prototype similarity, and the lower part is used for cosine similarity. Block-Merge-Conv synthesizes these similarity and outputs the hyperrelation features 5 EXPERIMENT In this section, we experiment our proposed model on two benchmark datasets and compare the results with current excellent methods. Pascal-5i Pascal − 5iusing images and annotations in Pascal-VOC 2012 [44] and extended annotations from SDS [45]. Following ref. [30], the 20 object categories in Pascal-VOC 2012 dataset is sub-divided into four folds i = {0, …, 3}, each fold having five classes Lfold = {4x+i}, where x∈{0, …, 4}. Following [24, 28, 30, 35, 36], we use cross-validation to evaluate the proposed model, taking the classes in one of the folds as test set Ctest and the classes in the other fold as the train set Ctrain. As shown in Table 1, the sample number distribution of different categories in the dataset is very uneven, which will bias the prediction result to the category with more samples. To solve this problem, we randomly select 750 images of each train class to form the train set. Another potential problem is that if objects belonging to the test class appear in the image of the training set, the category will always be predicted as the background during training, and the network will also tend to predict such objects as the background during testing. Therefore, the pictures containing the class of the test set are not used in the training of the network in this paper. TABLE 1. Pascle-5i data set category and corresponding quantity distribution Fold0 Object name aero-plane bicycle bird boat bottle Number 528 428 583 371 376 Fold1 Object name bus car cat chair cow Number 367 848 987 988 235 Fold2 Object name dining-tables dog horse motorbike person Number 504 1101 370 440 3386 Fold3 Object name potted-plant sheep sofa train tv-monitor Number 359 276 487 488 476 COCO-20i COCO − 20iis based on the MS COCO2014 dataset [32]. We also use cross-validation to test our model. Following ref.