SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuan Wang $^{1*}$ , Naisong Luo $^{1*}$ , Tianzhu Zhang $^{1\dagger}$
<sup>1</sup>Deep Space Exploration Laboratory/School of Information Science and Technology,
University of Science and Technology of China, Hefei 230026, China.
wy2016,lns6}@mail.ustc.edu.cn， tzzhang@ustc.edu.cn
Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention to the mining of the critical query branch. In this paper, we rethink the importance of support information and propose a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer  $(G)$  that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer  $(D)$  to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train  $G$  and  $D$  via an adversarial process, where  $G$  is optimized to generate more accurate masks approaching ground truth to fool  $D$ . We conduct extensive experiments on commonly used Pascal-5 $^i$  and COCO-20 $^i$  benchmarks and achieve state-of-the-art results across all settings. In addition, the decent performance with weak support labels in our query-centric paradigm may inspire the development of more general FSS models. Code will be available at https://github.com/Wyxdm/AMNet
As a fundamental computer vision task, semantic segmentation [1] has drawn decades of research interest in both academia and industry. However, the underlying driving force of fully supervised segmentation models is massive, densely labeled data. The inherent category sensitivity of these models leads to severe performance degradation when dealing with previously unseen categories. To equip segmentation models with generalization to novel classes, few-shot segmentation (FSS) has been proposed [2], which achieves segmentation of new category images (called query images) with only a handful of annotated samples (called support images) without retraining the model.
The current top-performing FSS methods focus on adequately exploring the information contained in the support samples to guide the segmentation of the query images. Among them, prototypical learning methods [3-6] concentrate support features into one or several representative prototypes to direct the classification of query pixels, while the affinity learning approaches [7-9] try to equip the query features with support semantic clue of pixel granularity. Although considerable progress has been made in the direction of probing support information, we noticed that significant intra-class diversities (as illustrated in Figure 2) between support and query images are common occurrences. In these cases, the target feature from the query image may be quite distinct from the support one.
† Corresponding Author.
(a) Support-centric approaches
(b) Our query-centric approach
Figure 1: Comparison between the previous methods and ours. (a) Most of the previous methods focus their efforts on extracting more support information. They condense support information into prototypes [3, 4, 6, 12] or directly explore pixel-level support features [7-9, 13]. (b) The proposed AMFormer focus on the query features and exploits intra-object similarity to mining the complete target, forming a query-centric FSS method. Only rough category guidance from support is needed in our approach.
Hence there raises a natural question: is exhaustive support information extraction indispensable for query image segmentation?
We revisit the role of support information in few-shot segmentation through a pilot study. As shown in Table 1, we randomly erode support foreground features with varying proportions to explore how the support information completeness affects query segmentation accuracy. It can be observed that the performances do not degrade significantly or even surpass the complete one. We deem two main reasons contribute to the phenomenon: 1) Rough category guidance from support is sufficient to guide the accurate query image segmentation. 2) Forcing fine-grained support information may introduce redundancy, or even bias, which confuses the prediction of query images, particularly when intra-class diversities are prevalent. Motivated by the above insights, we adjust the direction and focus on the query branch. Building on the observation of intra-object similarity, i.e., pixels within the same object are more similar than those across different objects [10, 11] (as shown in Figure 2 and Table 8, we argue that partially activated object regions can serve as cues to infer whole objects. In this paper, we construct a pseudo support image that has low intra-class variation with the query image by activating query image with a rough category-level support clue. Then we exploit the pseudo support feature to guide the query segmentation, forming a novel query-centric exploration strategy as illustrated in Figure 1.
We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. The holistic semantic clue from the labeled support image is utilized to roughly locate the most discriminative target area on the query image. (2) Local to global expansion. We utilize intra-object feature similarity to explore contextual information and then highlight the less discriminative object parts. (3) Coarse to fine alignment. The coarse activated region is further refined by eliminating the detailed difference with ground truth. Different from previous methods that focus on extracting fine-grained support information, Our method is less susceptible to intra-class diversity as only a rough holistic support guidance is needed in the first procedure to generate the pseudo support. More attention is paid to the last two query-focused procedures that enable growing incomplete local regions into accurate segmentations.
We propose an end-to-end Adversarial Mining Transformer(AMFormer) to couple procedures (2) and (3) and mutually enhance each other, which optimizes an object mining transformer  $G$  and a detail mining transformer  $D$  via an adversarial process. AMFormer aims at fully excavating the target area and aligning it in detail under the guidance of the pseudo support features. To dig out the entire object
Figure 2: t-SNE visualization of the intra-class diversity and the intra-object similarity. As shown by the blue and green circles, different instances of the same category may be far apart in the feature space, e.g., dog. While pixels from the same object share high feature similarity, as shown by the three circles within the blue one.
region, in the  $G$ , we conduct multi-scale pixel-level correlation between the query features and pseudo support features. The exploration of the intra- and inter-scale contextual information in  $G$  progressively highlights the latent target parts that share high intra-object similarities. To effectively align the prediction details, e.g., object boundaries,  $D$  models multiple detail proxies to accurately distinguish the subtle local differences between the ground truth and the prediction generated by  $G$ . By means of adversarial training,  $G$  is optimized to predict more accurate segmentations to fool  $D$  and only the well-learned  $G$  is required to produce accurate predictions in the inference stage.
We evaluate our AMFormer on two widely used benchmarks COCO-  $20^{i}$  and Pascal-  $5^{i}$  with different backbones and the AMFormer consistently sets new state-of-the-art on different settings. Moreover, our query-centric approach does not reckon on elaborate support annotations. It achieves remarkable performance even with weak support labels such as scribbles or bounding boxes, formulating a more practical FSS model. We also hope that our work can
Table 1: The performance (mIoU) under different support feature erosion ratios. We conduct this pilot study on  $1^{st}$  split of Pascal-  $5^i$  using ResNet50 and 1-shot setting. \* means reproduced results.
Erosion Ratio (%)	80	65	50	35	20	0*
PFENet [14]	60.6	61.0	61.5	61.2	61.0	61.2
HDMNet [13]	69.3	70.0	70.5	70.4	69.9	70.2
inspire more research on query-centric FSS methods. To recap, our contributions are concluded as follows: (i) We re-evaluate the importance of support information in FSS and demonstrate that a coarse category hint suffices for accurate query segmentation. This motivated us to put forward a novel query-centric FSS method. (ii) We propose a novel Adversarial Mining Transformer (AM-Former) that optimizes an object mining transformer  $G$  and a detail mining transformer  $D$  for region expansion and detail alignment, respectively. (iii) Extensive experiments show that AMFormer significantly outperforms previous SOTAs. The conspicuous performance with weak support label also sheds light for future research of more general FSS models.
2.1 Semantic Segmentation
Semantic segmentation has been widely applied to autonomous driving, medical image processing [15-17] and so on. The aim of semantic segmentation is to assign each pixel within the given image to a predefined category label. The seminal FCN [1] sparked a wave of remarkable advances in semantic segmentation [18-20] based on convolutional neural networks (CNN). Various networks focus on better context exploration by enlarging the receptive field of CNN via dilated convolutions [21, 22], global pooling [23] and pyramid pooling [21, 24]. In addition to CNN-based models, the success of Vision Transformer (ViT) [25] encourages a series of transformer-based segmentation models [26-29]. For instance, maskformer [30] adopts the transformer decoder [31] to conduct the mask classification based on a set prediction objective. Many subsequent works improve this framework [32-35] and gradually formed a unified segmentation model that can address different image segmentation tasks. Despite the success, these methods cannot generalize to novel classes in the low-data regime.
2.2 Few-Shot Semantic Segmentation
Few-shot segmentation (FSS) [2] is established to segment new category images (query images) with only a few labeled samples (support images). Owing to the reliable annotations in the support set, current FSS methods mainly focus on effectively excavating support information, which can be roughly divided into two categories: prototypical learning methods and affinity learning methods. Motivated by PrototypicalNet [36], many previous works condense the support information into single [37-39, 6, 40] or multiple prototypes [41-43, 5, 4, 44-46] and then conduct feature comparison or aggregation. For example, ASGNet [3] adopts superpixel-guided clustering to adaptively construct multiple prototypes, which are concatenated with the most relevant query features to guide the pixel classification. Differently, for the seek of fine-grained support guidance, affinity learning methods [13, 47, 8, 48-50] are designed to establish pixel-level associations between support and query features via attention mechanism [7, 9] or cost volume aggregation [8, 48]. For instance, CyCTR [7] introduces a cycle-consistent attention mechanism to equip query features with relevant support information. Though achieving promising results, these methods depend heavily on support information and are prone to segmentation errors in the presence of large intra-class variations. Some methods[11, 6] try to solve this problem by mining the class-specific representation from the
query branch but remain at relatively coarse prototype granularity. In this paper, we propose a novel query-centric approach that exploits intra-object similarity to probe query targets with only overall category-level guidance from support.
Few-shot segmentation (FSS) tackles novel class object segmentation with only a few densely-annotated samples. Episodic meta-training [51] is widely used to enhance the generalization of FSS models. Specifically, the dataset is divided into the training set  $\mathcal{D}_{train}$  and the testing set  $\mathcal{D}_{test}$ . The category sets of  $\mathcal{D}_{train}$  and  $\mathcal{D}_{test}$  are disjoint, i.e.,  $\mathcal{C}_{train} \cap \mathcal{C}_{test} = \emptyset$ . A series of episodes are sampled from  $\mathcal{D}_{train}$  to train the model, each of which is composed of a support set  $S = \{I_s^k, M_s^k\}_{k=1}^K$  and a query set  $\mathcal{Q} = \{I_q, M_q\}$  in the  $K$ -shot setting, where  $I$  and  $M$  denote the RGB image and corresponding binary masks, respectively. Under the supervision of  $M_q$ , the model is trained to predict the query mask conditioned on the  $S$  and  $I_q$ . After that, the trained model is evaluated on the episodes sampled from  $\mathcal{D}_{test}$  without further optimization.
3.2 Revisiting of Transformer-based Feature Aggregation
Transformer layers [52] are widely used in computer vision tasks for feature aggregation. The critical component of transformer layer is the attention mechanism that enables the long-range modeling capability. Specifically, the attention layer is first applied to compute the attention weight of the source feature sequence  $\mathbf{S} \in \mathbb{R}^{N_1 \times C}$  and the target sequence  $\mathbf{T} \in \mathbb{R}^{N_2 \times C}$ , where the  $N_1$ ,  $N_2$  denote the length of sequences and  $C$  is the embedding dimension, formulaic as:
$$
\mathbf {S} = \operatorname {S o f t m a x} \left(\frac {\mathbf {Q} (\mathbf {K}) ^ {\mathsf {T}}}{\sqrt {\mathrm {d}}}\right), \quad \mathbf {Q} = \mathbf {T} \mathbf {W} ^ {\mathsf {Q}}, \quad \mathbf {K} = \mathbf {S} \mathbf {W} ^ {\mathsf {K}}, \tag {1}
$$
in which  $\mathbf{W}^{\mathrm{Q}}$  and  $\mathbf{W}^{\mathrm{K}}\in \mathbb{R}^{C\times d}$  are learnable linear projections and  $\sqrt{d}$  is the scaling factor. The source information is adaptively transported to the target sequence according to attention weight, and a feed-forward network (FFN) is further applied to transform the fused features:
$$
\widehat {\mathbf {T}} = \mathbf {F F N} (\mathbf {S V}), \quad \mathbf {V} = \mathbf {S W} ^ {\mathrm {V}}, \tag {2}
$$
where the  $\mathbf{W}^{\mathrm{V}}$  is the linear linear projection and  $\widehat{\mathbf{T}}$  is the enhanced target feature sequence. We can abbreviate the feature aggregation process as:
$$
\widehat {\mathbf {T}} = \operatorname {F e a t} \mathbf {A g g} [ \mathbf {T}, \mathbf {S} ]. \tag {3}
$$
Note that when  $\mathbf{T} = \mathbf{S}$ , the  $\mathrm{FeatAgg}(,)$  explores contextual information within the feature sequence, i.e., acts as the self-attention mechanism. The above processes are implemented with the multi-head mechanism to enhance performance further.
3.3 Adversarial Mining Transformer
3.3.1 Overview
The proposed framework tailored for query-centric FSS consists of three procedures, i.e., 1) discriminative region localization, 2) local to global region expansion, 3) coarse to fine mask alignment. We roughly locate the object of novel class under a rough support guidance via nonparametric cosine similarity in procedure 1). The resulting incompleted region is expanded by the object miner  $G$  to cover the whole target in procedure 2). The detail miner  $D$  in procedure 3) is tasked with identifying local discrepancies between the ground truth and the expanded mask generated by  $G$ , thereby aligning the coarse prediction in detail. We optimize the  $G$  and  $D$  via an adversarial process and forming the end-to-end AMNet as illustrated in Fig. 3. The details are as follows.
3.3.2 Discriminative region localization
Considering the significant intra-class diversity between samples, we contend that support information is more suitable as overall guidance indicating the novel class rather than detailed reference. Given
Figure 3: Illustration of the proposed AMFormer. We approach the query-centric FSS with three procedures, i.e., discriminative region localization, local to global region expansion, and coarse to fine mask alignment. AMFormer optimizes an object mining transformer and a detail mining transformer via an adversarial process to couple these procedures.
the support image features  $\mathbf{F}_s\in \mathbb{R}^{H\times W\times C}$  and corresponding mask  $\mathbf{M}_s\in \mathbb{R}^{H\times W}$ , where  $H$  and  $W$  describe the feature size and  $C$  is the feature dim, we apply mask average pooling (MAP) to obtain the mean target feature  $\mathbf{F}_h\in \mathbb{R}^{1\times C}$  that serves as the holistic support guidance, formally:
$$
\mathbf {F} _ {h} = \mathbf {M A P} \left(\mathbf {F} _ {s}, \mathbf {M} _ {s}\right). \tag {4}
$$
We exploit the  $\mathbf{F}_h$  to indicate the target region in the  $\mathbf{F}_q$  based on the cosine similarity:
$$
\mathbf {M} _ {q \tau} = \mathbb {1} ^ {\tau} (\operatorname {S o f t m a x} (\cos (\mathbf {F} _ {\mathrm {h}}, \mathbf {F} _ {\mathrm {q}}))), \quad \mathbb {1} ^ {\tau} (\mathrm {x}) = \left\{ \begin{array}{l l} 1, & x > \tau \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
A relatively high threshold  $\tau$  is set to locate the most class-relevant regions and suppress activation to backgrounds. In our experiment,  $\tau$  is set to 0.7.
3.3.3 Local to global region expansion
Adaptive feature aggregation. The high-confidence object regions  $\mathbf{M}_{q\tau} \in \mathbb{R}^{H \times W}$  obtained by Eqn (5) are incomplete. We introduce the object mining transformer  $G$  to expand the local region to the entire target area based on the intra-object similarity. Specifically,  $G$  aims at aggregating the representative target features from the pseudo support to less discriminative object parts in the query, formally,
$$
\widehat {\mathbf {F}} _ {q} = \mathcal {F} ^ {- 1} \left(\operatorname {F e a t A g g} \left[ \mathcal {F} \left(\mathbf {F} _ {\mathbf {q}}\right), \mathcal {F} \left(\mathbf {F} _ {p s d}\right) \right]\right), \tag {6}
$$
where the  $\mathcal{F}:\mathbb{R}^{H\times W\times C}\mapsto \mathbb{R}^{HW\times C}$  is the spatial flatten operation.  $\mathbf{F}_{psd}$ , which serve as the source sequence in Eqn (1), are essentially query features filtered by  $\mathbf{M}_{q\tau}$ :
$$
\mathbf {F} _ {p s d} = \mathbf {F} _ {q} \odot \mathbf {M} _ {q \tau}. \tag {7}
$$
Homogeneous feature guidance enables our aggregation to circumvent the effects of intra-class variance. Note that the original attention mechanism applies Softmax activation along the source sequence dimension. Directly adopting this scheme in our aggregation makes the query background features inevitably attend to foreground features since all the non-zero features in the pseudo support belong to the foreground. Inspired by [53, 13], we adjust to implement Softmax along the axis of  $\mathbf{F}_q$ . In this way, the discriminative object features are more likely to be aggregated target areas that share high intra-object similarities with  $\mathbf{F}_{psd}$ .
Cross-scale information transportation. We observe that in many cases, the query image contains different objects with varying scales, which are to be segmented in other episodes. To handle this spatial inconsistency, we construct the  $G$  in a multi-scale form. Concretely, we follow [13] to establish the hierarchical query features  $\{\mathbf{F}_{q,l}\}_{l = 1}^{L}$  with down-sampling and self-attention layers, in which
$$
\mathbf {F} _ {q} ^ {l} \in \mathbb {R} ^ {\frac {H}{2 ^ {l - 1}} \times \frac {W}{2 ^ {l - 1}} \times C}, \quad l = 1, \dots , L. \tag {8}
$$
The corresponding pseudo support features  $\{\mathbf{F}_{psd,l}\}_{l=1}^{L}$  is obtained by the Hadamard product of  $\{\mathbf{F}_{q,l}\}_{l=1}^{L}$  and downsampled  $\mathbf{M}_{q\tau}$ . Then Eqn (6) is implemented independently in each scale to obtain the aggregated query features  $\{\widehat{\mathbf{F}}_{q,l}\}_{l=1}^{L}$ . Finally, we fuse the multi-scale aggregated query features in a top-down manner as down in [14], specifically,
$$
\widehat {\mathbf {F}} _ {q, l} ^ {\prime} = \operatorname {C o n v} _ {3 \times 3} \left(\operatorname {C o n v} _ {1 \times 1} \left(\widehat {\mathbf {F}} _ {q, l} + \mathcal {R} \left(\widehat {\mathbf {F}} _ {q, l + 1}\right)\right) + \mathcal {R} \left(\widehat {\mathbf {F}} _ {q, l + 1}\right)\right). \tag {9}
$$
The fused features from the last stage  $\widehat{\mathbf{F}}_{q,1}^{\prime}\in \mathbb{R}^{H\times W\times C}$  are exploited to predict the expanded target area  $\mathbf{M}_e\in \mathbb{R}^{H\times W}$  via a small convolution head. Benefiting from the adaptive feature aggregation and cross-scale information transportation in the object mining transformer  $G$ , less discriminative target parts can be highlighted in the query image, forming the expanded object mask  $\mathbf{M}_e$ .
3.3.4 Coarse to fine mask alignment
Detail mining transformer  $D$  is designed to discriminate subtle differences between the ground truth  $\mathbf{M}_{gt}$  and the expanded object mask  $\mathbf{M}_e$  generated by  $G$ , since  $\mathbf{M}_e$  adequately covers the object but still exhibits misalignments in fine-grained details, e.g., boundaries. By training  $G$  and  $D$  via an adversarial process,  $G$  is optimized to generate more accurate target masks approaching ground truth to fool  $D$ , thus achieving coarse to fine prediction alignment.
To capture comprehensive details,  $D$  models multiple local proxies, each of which is tasked with exploring the object features respectively specified by  $\mathbf{M}_e$  and  $\mathbf{M}_{gt}$  to obtain a pair of local features. The most different pair is selected to output real/fake results. Concretely, we concatenate the learnable proxies as a feature sequence  $\mathbf{P} \in \mathbb{R}^{N \times C}$ , where  $N$  denotes the number of proxies. We perform feature adaptive aggregation through the attention mechanism to construct local features, In specific,
$$
\boldsymbol {\Omega} _ {f} = \operatorname {F e a t A g g} [ \mathbf {P}, \mathcal {F} \left(\mathbf {F} _ {\mathbf {q}} \odot \mathbf {M} _ {e}\right) ], \tag {10}
$$
where the "fake" object features specified by predicted mask  $(\mathbf{F}_{\mathbf{q}}\odot \mathbf{M}_{e})$  serves as the source sequence of aggregation process. Note that the  $\mathbf{M}_e$  is not binarized. Similarly, local features originate from "real" object features are obtained by:
$$
\boldsymbol {\Omega} _ {r} = \operatorname {F e a t} \mathbf {A g g} [ \mathbf {P}, \mathcal {F} (\mathbf {F} _ {\mathbf {q}} \odot \mathbf {M} _ {g t}) ]. \tag {11}
$$
Finally, we calculate the cosine similarity among the real local features  $\Omega_r = \{\omega_f^i\}_{i=1}^N$  and the fake ones  $\Omega_f = \{\omega_f^i\}_{i=1}^N$ . The most different pair  $(\omega_f^k, \omega_r^k)$  is selected and fed into a fully-connected layer to predict the fake/real results for adversarial training. By this way,  $D$  is optimized to discriminate detailed local differences, while the  $G$  will generate more precise masks to fool  $D$  by adjusting itself.
Since there is no explicit supervision, different proxies may focus on the same object part. We impose a diversity loss to avoid this degradation by expanding the discrepancy among local features:
$$
\mathcal {L} _ {d i v} = \frac {1}{N (N - 1)} \sum_ {i = 1} ^ {N} \sum_ {j = 1, i \neq j} ^ {N} \left(\frac {\langle \omega_ {f} ^ {i} , \omega_ {f} ^ {j} \rangle}{\| \omega_ {f} ^ {i} \| _ {2} \| \omega_ {f} ^ {j} \| _ {2}} + \frac {\langle \omega_ {r} ^ {i} , \omega_ {r} ^ {j} \rangle}{\| \omega_ {r} ^ {i} \| _ {2} \| \omega_ {r} ^ {j} \| _ {2}}\right). \tag {12}
$$
The intuition of Eqn (12) is trivial. If different proxies focus on the same object region,  $\mathcal{L}_{div}$  will be large and adjust the learning of proxies.
3.3.5 Training and Inference