SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: Intra- and inter-object similarity of each class within Pascal-  $5^{i}$ ,
Pascal-50	Pascal-51	Pascal-52	Pascal-53
class	Intra-	Inter-	class	Intra-	Inter-	class	Intra-	Inter-	class	Intra-	Inter-
aeroplane	0.449	0.008	bus	0.453	0.011	diningtable	0.496	0.010	potted plant	0.515	0.012
bicycle	0.460	0.009	bus	0.453	0.011	dog	0.571	0.009	sheep	0.536	0.007
bird	0.493	0.010	cat	0.643	0.021	horse	0.489	0.012	sofa	0.535	0.006
boat	0.483	0.009	chair	0.519	0.015	motorbike	0.445	0.008	train	0.509	0.008
bottle	0.504	0.120	cow	0.573	0.016	person	0.529	0.039	tv/monitor	0.513	0.024
10.2 More visualization results.
Visualization of segmentation at different stages. We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. (2) Local to global expansion. (3) Coarse to fine alignment. To illustrate the effects of the above three steps, in Figure 7, we visualize the outcomes of different stages. Procedure (1) can only roughly local the discriminative region of the target ( $2^{nd}$  column of Figure 7). In procedure (2), the object mining transformer  $G$  exploits the intra-object similarity to explore multi-scale contextual information, thus highlighting the whole object ( $3^{rd}$  column of Figure 7). Segmentation from  $G$  can roughly cover the entire target but there still exist misalignments as shown in the yellow boxes in the  $3^{rd}$  column of Figure 7. The detail mining transformer  $D$  is responsible for discriminating those detailed misalignments, i.e., procedure (3). The proposed AMFormer couples procedures (2) and (3) via adversarial training. In this way, the G can be optimized to generate more accurate segmentations ( $4^{th}$  column of Figure 7) to fool  $D$ .
Visualization of activation maps. We visualize the attention weight of query features between the support target and pseudo support. Specifically, the attention matrix  $\mathbf{S} \in \mathbb{R}^{H_qW_q \times H_sW_s}$  is computed according to the Eqn (1) of the original manuscript. Then we compute the average activation of each query pixel over all support foreground pixels:
$$
\mathbf {A c t} (i) = \frac {\sum_ {j = 1} ^ {H _ {s} W _ {s}} \mathbf {S} (i , j) \cdot [ \mathcal {F} (\mathbf {M} _ {s}) (j) > 0 ]}{\sum_ {j = 1} ^ {H _ {s} W _ {s}} [ \mathcal {F} (\mathbf {M} _ {s}) (j) > 0 ]}, \tag {16}
$$
where the  $\mathbf{M}_s$  is the (pseudo) support mask. In the baseline, the  $\mathbf{S}$  is oriented from the cross attention between the query features and the support features (support-centric). While in our query-centric
Table 13: Intra- and inter-object similarity of each class within COCO-20  ${}^{i}$
COCO-201	COCO-201	COCO-202	COCO-203
class	Intra- Inter-	class	Intra- Inter-	class	Intra- Inter-	class	Intra- Inter-	class	Intra- Inter-
Person	0.553 0.002	Bicycle	0.398 0.017	Car	0.514 0.012	Motorcycle	0.485 0.007
Airplane	0.582 0.020	Bus	0.440 0.019	Train	0.544 0.017	Truck	0.552 0.021
Boat	0.585 0.013	T.light	0.474 0.013	Fire H.	0.467 0.007	Stop	0.544 0.008
Park meter	0.476 0.014	Bench	0.459 0.044	Bird	0.530 0.016	Cat	0.549 0.010
Dog	0.482 0.010	Horse	0.574 0.018	Sheep	0.487 0.010	Cow	0.584 0.015
Elephant	0.467 0.012	Bear	0.460 0.012	Zebra	0.510 0.012	Giraffe	0.556 0.013
Backpack	0.479 0.013	Umbrella	0.571 0.020	Handbag	0.519 0.032	Tie	0.559 0.006
Suitcase	0.542 0.007	Frisbee	0.384 0.005	Skis	0.432 0.007	Snowboard	0.509 0.014
Sports ball	0.538 0.009	Kite	0.470 0.016	B.bat	0.532 0.023	B.glove	0.576 0.006
Skateboard	0.537 0.018	Surfboard	0.589 0.024	T.racket	0.373 0.012	Bottle	0.545 0.007
W.glass	0.434 0.011	Cup	0.606 0.009	Fork	0.451 0.017	Knife	0.589 0.013
Spoon	0.562 0.027	Bowl	0.563 0.010	Banana	0.511 0.015	Apple	0.656 0.010
Sandwich	0.469 0.013	Orange	0.474 0.028	Broccoli	0.471 0.026	Carrot	0.539 0.007
Hot dog	0.558 0.008	Pizza	0.515 0.015	Donut	0.477 0.300	Cake	0.597 0.006
Chair	0.442 0.015	Couch	0.542 0.013	P.plant	0.433 0.032	Bed	0.560 0.007
D.table	0.474 0.046	Toilet	0.503 0.006	TV	0.456 0.014	Laptop	0.480 0.021
Mouse	0.447 0.021	Remote	0.583 0.011	Keyboard	0.542 0.007	Cellphone	0.498 0.018
Microwave	0.454 0.019	Oven	0.567 0.011	Toaster	0.433 0.007	Sink	0.375 0.027
Fridge	0.476 0.023	Book	0.586 0.025	Clock	0.526 0.018	Vase	0.567 0.013
Scissors	0.456 0.027	Teddy	0.583 0.016	Hairdrier	0.373 0.007	Toothbrush	0.479 0.013
AMFormer, the S is computed from the pseudo support and the query features. As shown in Figure 8, the support targets not only cannot fully activate the target in the query image, but also frequently activates the background categories. While the pseudo support can well excavate the full object attribute to intra-object similarity.
Visualization of local proxies. To explore the regions of interest for learnable local proxies, Figure 9 visualizes the activation maps of a part of proxies. It can be observed that different proxies tend to focus on different local regions, and most proxies attend to the boundaries of the object, which is usually the most ambiguous region. In addition, a particular proxy consistently focuses on the boundaries in a specific direction, e.g., proxy 1 always activates the right border ( $5^{th}$  column). Through the cooperation of multiple proxies, our detail mining transformer  $G$  can effectively detect the detailed local differences between the prediction of the object mining transformer  $G$  and the ground truth. By means of adversarial training,  $G$  will produce more accurate segmentations, especially in ambiguous regions, by adjusting itself to fool D.
Figure 7: Visualization of the segmentation of the proposed AMFormer at different stages
activation (baseline)
activation (ours)
Figure 8: Visualizations of the attention weight of query features between the support target (support-centric baseline) and pseudo support (ours).
Figure 9: Visualizations of the activated regions of local proxies.