SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenchao Jin $^{1}$ , Xiaowei Hu $^{2}$ , Lingting Zhu $^{1}$ , Luchuan Song $^{4}$ , Li Yuan $^{3}$ , Lequan Yu $^{1}$ $^{1}$ The University of Hong Kong  $^{2}$ Shanghai AI Laboratory  
 $^{3}$ Peking University  $^{4}$ University of Rochester  
{blwx96@connect., ltzhu99@connect., lqyu@}hku.hk  
huxiaowei@pjlab.org.cn  
yuanli-ece@pku.edu.cn  
lsong11@ur.ROCHESTER.edu
Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, e.g., multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or ineffective contextual information aggregation due to reliance on large amounts of predetermined priors. To alleviate the issues, we propose a novel Intervention-Driven Relation Network (IDRNet), which leverages a deletion diagnostics procedure to guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, a deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and the extracted relations are utilized to guide the semantic-level representations to interact with each other. Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments are conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-driven context scheme brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes. Code is available at https://github.com/SegmentationBLWX/sssegmentation.
Table 1: Performance comparison between existing context schemes and our IDRNet.
Benchmark	Multi-Scale-Driven	Similarity-Driven	Intervention-Driven
ASPP [8]	PPM [83]	UPerNet [70]	Non-Local [69]	Ann [89]	OCR [79]	IDRNet	IDRNet+
ADE20K [87]	43.19	42.64	43.02	42.15	41.75	42.47	43.61	44.84
Cityscapes [17]	79.62	79.05	79.08	78.34	78.36	79.40	79.91	80.81
Semantic image segmentation, which aims at assigning a semantic category to each pixel, is one of the most fundamental research areas in computer vision. Recent years have witnessed great progress in semantic segmentation using deep convolutional neural networks [44] and transformers [64]. The state-of-the-art semantic segmentation approaches [8, 12, 30, 39, 40, 50, 67, 79, 86] mostly follow the encoder-decoder paradigm [51] where the decoder is defined by replacing the fully-connected layers in classification tasks with some convolution layers.
Figure 1: Diagram of our intervention-driven relation network. Deletion diagnostics is leveraged to build relations between semantic-level representations. With the built relation matrix and semantic-level representations, pixel representations can be augmented for pixel prediction.
Inspired by co-occurrent visual patterns [25, 63, 83], a popular class of boosting the encoder-decoder architecture can be regarded as incorporating various contextual modules [8, 40, 41, 69, 79, 83] into decoders. Consequently, the segmentation performance improvements are conferred by enabling each pixel to perceive information from other pixels in the input image. For instance, ASPP [8] and PPM [83] exploited dilated convolutions or pyramid pooling layers to aggregate pixel representations at different spatial positions. OCRNet [79] proposed to augment pixel representations with object-level representations. MCIBI++ [39, 41] differentiated the beyond-image contextual information from the within-image contextual information and thus improved pixel representations from the perspective of the whole dataset. To summarize, the existing contextual modules can be grouped into two batches, i.e., multi-scale-driven [8, 70, 83] and similarity-driven [69, 79, 89] context schemes. Despite these methods achieved astonishing results, both schemes suffer from inadequate or ineffective contextual information aggregation as pixel relations are modeled with masses of predetermined priors. Specific to similarity-driven methods, contextual modules are designed to make pixels aggregate semantically similar contextual representations, which is inadequate for building the co-occurrent visual patterns. Also, some rigorous empirical analysis [5, 75] demonstrates that the contexts of pixels modeled by similarity-driven operations are indistinguishable for different query positions within an input image. As for multi-scale-driven approaches, pixel contexts are assumed to be fixed with certain geometric structures, which may compel pixels to capture some uninformative feature representations. Besides, previous studies [19, 67, 88] indicate that this fixed context assumption prevents the generalization of one-scenario-trained segmentors to other scenarios with unknown geometric transformations.
Different from conventional pipelines of building contextual modules, this paper investigates a more effective paradigm to build relationships among pixels. The proposed intervention-driven paradigm utilizes deletion diagnostics [16] to guide building pixel relations and thus aggregate more meaningful contextual information. In practice, there are two main challenges in constructing such a paradigm: (1) it is time-consuming to perform deletion diagnostics on all pixels and (2) it is memory-consuming to maintain a pixel relation matrix as one image contains plenty of pixels. Motivated by the observation that pixel relationships are closely connected to object relationships since a pixel semantic label is determined by which object region the pixel belongs to, we propose to simplify pixel-level relations modeling as object-level relations modeling to address the above-mentioned challenges.
Figure 1 depicts the diagram of our proposed intervention-driven relation network (IDRNet). We first group pixel representations into semantic-level representations according to predicted pseudo labels. Then, a relation matrix is constructed to make the semantic-level representations interact with each other and thus enhance themselves, where the relation matrix is updated by our deletion diagnostics mechanism. Finally, original pixel representations are augmented with the interacted representations and used to produce pixel labels. Compared with simply performing pixel-level deletion diagnostics, the GPU memory and computational complexity of our method can be significantly reduced. For example, on Cityscapes [17], we only need to maintain a relation matrix with a size  $19 \times 19$  rather than a relation matrix with an input image size of  $1024 \times 512$  during training. As the proposed deletion diagnostics mechanism is designed for semantic-level representations, it is also worth noting that our paradigm is general and can be easily extended to broader applications in other vision tasks, e.g., object detection [27] and instance segmentation [12].
Table 1 compares the segmentation performance of various context modeling paradigms on Cityscapes and ADE20K benchmark datasets under fair settings (e.g., backbone networks and training schedules). As observed, a simple FCN segmentation framework equipped with our intervention-driven context module (termed IDRNet) can achieve  $43.61\%$  and  $79.91\%$  mIoU on the datasets, respectively, which outperforms both dominant similarity-driven and multi-scale-driven context aggregation schemes. To further demonstrate the superiority of our method, we also integrate the proposed context scheme into UPerNet segmentation framework (termed IDRNet+), and it is observed that our paradigm with ResNet-50 backbone yields more impressive results on both datasets under single-scale testing.
In principle, our paradigm is fundamentally different from and would complement most similarity-driven and multi-scale-driven contextual modules. The main contribution can be summarized as,
- To our best knowledge, for the first time, this paper presents an intervention-driven paradigm, i.e., deletion diagnostics mechanism, to help model pixel relations.
- This paper indicates that, in semantic segmentation task, pixel-level relation modeling can be simplified as building semantic-level relations.
- Our intervention-driven context modeling paradigm is conscientiously designed. It can be seamlessly integrated into various state-of-the-art semantic segmentation frameworks and achieve consistent performance improvements on various benchmark datasets, including ADE20K, Cityscapes, COCO-Stuff, LIP and PASCAL-Context.
We expect that IDRNet can provide insights for the future development of semantic segmentation and other related vision tasks.
Semantic segmentation is a fundamental computer vision task, which is an indispensable component for many applications such as autopilot and medical diagnosis. Modern semantic image segmentation methods (in deep learning era) are believed to be derived from FCN [51] which formulates semantic segmentation as an encoder-decoder architecture. After that, numerous efforts have been devoted to improving the pipeline. For example, some tailored encoders like Res2Net [26] and HRNet [66] are designed for semantic segmentation to replace classification networks [33, 72]. In addition, automating network engineering [10, 47, 49] is also one of the most efficient ways to boost the encoder structure. Since transformer [64] attains growing attention, many recent works [1, 15, 21, 30, 50, 55, 62, 67, 71, 78, 86] also introduce transformer structure to help the encoder extract more effective pixel representations. Furthermore, there are also plenty of studies focusing on utilizing co-occurrent patterns to boost the decoder performance, e.g., modeling long-range dependency [5, 31, 37, 40, 69, 75, 76] and enlarging the receptive field [7, 8, 9, 74, 77, 83]. Besides, some researchers also find that introducing context-aware optimization objectives [2, 42, 84] can strengthen context cues. Apart from the above, some studies also choose to break new ground, e.g., introducing contrastive learning mechanism [68, 85], formulating the instance, panoptic and semantic segmentation tasks in a unified framework [12, 13, 82], leveraging boundary information [3, 20, 57, 65, 81], to name a few.
This paper studies how to leverage co-occurrent patterns. The key differences are that (1) we transform the pixel relation modeling problem to the semantic-level relation modeling problem and (2) for the first time, we propose to introduce deletion diagnostics mechanism to help model pixel relations.
2.2 Pixel Relation Modeling
As there exist co-occurrent patterns (i.e., how likely two semantic classes can exist in a same image) in semantic segmentation, pixel relation modeling becomes a crucial component of recent state-of-the-art semantic segmentation algorithms [5, 36, 40, 70, 69, 75, 77, 79, 83]. For example, non-local network [69] built non-local blocks for capturing long-range dependencies. GCNet [5] unified the non-local and squeeze-excitation operations into a general framework for aggregating more efficacious global contextual information. DNL [75] presented the disentangled non-local block to help learn more discriminative pixel representations. ISA [36] was motivated to improve the efficiency of the self-attention operations and proposed to factorize one dense affinity matrix as the product of two sparse affinity matrices. Besides, OCRNet [79] proposed to emphasize utilizing the contextual information in the corresponding object class. ISNet [40] further improved OCRNet by incorporating image-level and semantic-level contextual information.
Figure 2: Illustration of our intervention-driven relation network (IDRNet). We first extract pixel representations  $R_{p}$  using a backbone network  $\mathcal{F}_B$ , e.g., ResNet [33] or SwinTransformer [50]. Then,  $R_{p}$  is grouped into semantic-level representations  $R_{sl}$  based on a coarse pixel prediction  $P_{c}$ . Next, we leverage a relation matrix  $\mathcal{M}_r$  to make  $R_{sl}$  interact with each other and further obtain enhanced semantic-level representations  $R_{esl}$ . Finally,  $R_{p}$  is augmented by  $R_{esl}$  and utilized for final pixel predictions  $P_{o}$ . After each training iteration, the deletion diagnostics is conducted to update  $\mathcal{M}_r$ .
Succinctly, these contextual modules utilize feature similarities or fixed geometric structures to model pixel relations, which injects an unavoidable human priors bias. Distinctively, this paper introduces deletion diagnostics mechanism to guide pixel relation modeling.
2.3 Object Relation Modeling
In object detection, co-occurrence is first applied in DPM [23] to refine object scores. After that, several works [14, 24, 53] were proposed to take object position and size into account for better building object relations in pre-deep learning era. In deep learning era, object relation modeling is also one of the most popular techniques to augment object representations. For instance, relation networks [34] extended original attention module to model both object similarities and relative geometry as the relations between objects. Motivated by relation networks, SGRN [73] took advantage of graph convolutional networks [43] to better bridge object regions by defining them as graph nodes. SMN [11] proposed to adopt a memory mechanism to perform object-object context reasoning.
Considering that each semantic-level representation in our paradigm plays a similar role to an object region in object detection, our intervention-driven relation modeling paradigm can also be extended to object detection tasks to help object relation modeling.
3.1 Overview of IDRNet
Figure 2 illustrates our proposed intervention-driven relation network (IDRNet). The initial pixel-wise representations are grouped into semantic-level representations to facilitate relationship modeling. In detail, the semantic-level representations are first processed with a feature enhancement module and a feature interaction module with the guidance of a relation matrix  $\mathcal{M}_r$ , where the relation matrix is learned with the proposed deletion diagnostics paradigm. We subsequently augment the initial pixel representations with the processed semantic-level representations for final predictions.
Semantic-level Representation Generation. Following conventional FCN-based architecture [51], we first adopt a backbone network  $\mathcal{F}_B$  to extract the pixel representations  $R_{p}\in \mathbb{R}^{Z\times \frac{H}{8}\times \frac{W}{8}}$  from an input image  $\mathcal{L}\in \mathbb{R}^{3\times H\times W}$ , where  $H$  and  $W$  are height and width of the image and  $Z$  is the number of feature channels. To facilitate the seamless integration of our paradigm with existing segmentation frameworks, we formulate the pseudo-label generation process as follows,
$$
P _ {c} = \mathcal {F} _ {C} \left(\mathcal {C} _ {e} \left(R _ {p}\right)\right), Y _ {c} = \operatorname {a r g m a x} \left(P _ {c}\right), \tag {1}
$$
where  $\mathcal{C}_e$  denotes an identity function (instantiated as FCN [51]) or a previous similarity- / multi-scale-driven context module (instantiated as OCRNet [79] and PSPNet [83]) and  $\mathcal{F}_C$  is a convolution layer
to project pixel representations into probability space.  $P_{c}\in \mathbb{R}^{K\times \frac{H}{8}\times \frac{W}{8}}$  is the predicted probabilities of  $K$  classes and  $Y_{c}\in \mathbb{R}^{\frac{H}{8}\times \frac{W}{8}}$  is the pseudo label.
As it is computationally impracticable to direct model pixel-level relationships with deletion diagnostics, we propose to simplify pixel-level relations modeling as semantic-level relations modeling, i.e., we group  $R_{p}$  into semantic-level representations  $R_{sl}$  by aggregating the corresponding input pixel representations with the same pseudo label,
$$
R _ {s l} = \left\{\mathcal {F} _ {M} \left(\left\{R _ {p, [ *, i, j ]} \mid Y _ {c, [ i, j ]} = k \right\}\right), k \in [ 0, K) \right\}, \tag {2}
$$
where  $R_{sl} \in \mathbb{R}^{N_e \times Z}$  and  $N_e$  is the number of classes existed in  $\mathcal{I}$ .  $\mathcal{F}_M$  is a weighted summation operation to merge pixel representations with the corresponding probabilities in  $P_c$  as the weights.
Feature Enhancement. We further introduce a feature enhancement module to improve the discrimination of  $R_{sl}$  to help augment pixel representations  $R_{p}$ . Specifically, we modify Eq (2) as
$$
R _ {s l} = \left\{\mathcal {F} _ {M} \left(\left\{R _ {p, [ *, i, j ]} \oplus R _ {i e, [ k, * ]} \mid Y _ {c, [ i, j ]} = k \right\}\right), k \in [ 0, K) \right\}, \tag {3}
$$
where  $\oplus$  denotes a concatenation operation and  $R_{ie} \in \mathbb{R}^{K \times Z}$  denotes discriminative vectors for each class in the training dataset. In our experiments, we investigate two strategies to obtain  $R_{ie}$ . The first one is to yield a random orthogonal matrix [52] to represent  $R_{ie}$ , while the second one is to learn a dataset-level representation for each category to construct  $R_{ie}$  [39]. For the second strategy, we simplify  $R_{ie}$  update strategy for class id  $k$  as
$$
R _ {i e, [ k, * ]} = R _ {i e, [ k, * ]} \times (1 - m) + R _ {g t} \times m, \tag {4}
$$
where  $m$  is an update momentum and set as 0.1 by default. Before training, we simply initialize  $R_{ie}$  as a zero matrix and  $R_{gt} \in \mathbb{R}^{1 \times Z}$  is calculated as
$$
R _ {g t} = \left\{\operatorname {M e a n} \left(\left\{R _ {p, [ *, i, j ]} \mid G T _ {[ i, j ]} = k \right\}\right), k \in [ 0, K) \right\}, \tag {5}
$$
where  $GT$  denotes the ground truth segmentation mask and  $\mathrm{Mean}(\cdot)$  is used to average the extracted pixel representations with the same ground truth label.
Feature Interaction. After the feature enhancement module, we propose to make  $R_{sl}$  interact with each other to enrich themselves by using a semantic-level relation matrix  $\mathcal{M}_r \in \mathbb{R}^{K \times K}$ , which is updated by the proposed deletion diagnostics mechanism. The procedure of the feature interaction module can be represented as
$$
R _ {e s l} = \operatorname {S o f t m a x} \left(\mathcal {T} \left(\mathcal {M} _ {r}\right), d i m = 1\right) \otimes R _ {s l}, \tag {6}
$$
where we adopt  $\otimes$  to denote matrix multiplication and  $R_{esl} \in \mathbb{R}^{N_e \times Z}$  is the enhanced semantic-level representations.  $\mathcal{T}$  is employed to transform  $\mathcal{M}_r$  to adapt the shape of  $R_{sl}$ , and it is formulated as
$$
\hat {\mathcal {M}} _ {r} = \mathcal {T} \left(\mathcal {M} _ {r}\right) = \left\{\mathcal {S} \left(\mathcal {M} _ {r, [ i, j ]}, t\right) \mid i, j \in \{c l a s s i d s i n R _ {s l} \} \right\}, \tag {7}
$$
where  $\mathcal{S}$  is introduced to control that there are no interactions between two semantic-level representations whose relationship in  $\mathcal{M}_r$  is weak.  $t = 0$  is a threshold utilized to identify those weak relationships. In practice, we set  $\mathcal{M}_{r,[i,j]}$  be negative infinity in the calculation if  $\mathcal{M}_{r,[i,j]}$  is smaller than  $t$ . Subsequently, we re-arrange  $R_{esl}$  into the original pixel representation shape to augment  $R_{p}$ ,
$$
R _ {a, [ *, i, j ]} = R _ {e s l, \left[ k ^ {i d x}, * \right]} \text {i f} Y _ {c, [ i, j ]} = k, k \in \{c l a s s i d s i n R _ {s l} \}, \tag {8}
$$
where  $k^{idx}$  is the corresponding matrix index for class id  $k$  in  $\mathcal{R}_{sl}$ .  $R_{a} \in \mathbb{R}^{Z \times \frac{H}{8} \times \frac{W}{8}}$  are the feature representations initialized as a zero matrix, and we fill it with the representations in  $R_{esl}$  according to the pseudo labels in each pixel position.
Final Prediction. We use  $R_{a}\in \mathbb{R}^{Z\times \frac{H}{8}\times \frac{W}{8}}$  to augment  $R_{p}$  for final predictions,
$$
\widetilde {R} = R _ {a} \oplus \mathcal {C} _ {e} (R _ {p}), R = S A (\widetilde {R}), \tag {9}
$$
where we use the self-attention mechanism  $SA$  [64] to further process the augmented representations to balance the diversity and discriminative of the pixels with the same class. Based on  $R$ , the final pixel predictions  $P_{o} \in \mathbb{R}^{K \times H \times W}$  is produced as
$$
P _ {o} = \mathcal {U} _ {8 \times} \left(\mathcal {F} _ {O} (R)\right), \tag {10}
$$
where  $\mathcal{F}_O$  is implemented by a convolution layer, which is utilized to produce pixel class probabilities and  $\mathcal{U}$  is a bilinear interpolation operation.
During training, the overall objective function for backpropagation algorithm [45, 58] is defined as
$$
\mathcal {L} = \alpha \mathcal {L} _ {c} + \mathcal {L} _ {o}, \tag {11}
$$
where  $\mathcal{L}_c$  and  $\mathcal{L}_o$  are the cross entropy losses between  $P_{c}$  and  $GT$ , and  $P_{o}$  and  $GT$ , respectively.  $\alpha$  is a hyperparameter for balancing  $\mathcal{L}_c$  and  $\mathcal{L}_o$ . We empirically set it as 0.4 in our experiments.
3.2 Deletion Diagnostics
Deletion diagnostics is proposed to update the relation matrix  $\mathcal{M}_r$ , which is initialized as an identity matrix. As shown in Figure 2, we randomly delete one semantic-level representation in  $R_{sl}$  with a deletion distribution Prob and obtain  $R_{sl}^{\prime}$ . The deletion probability Prob of the  $i$ -th semantic class representation is related to the number of times (i.e., count) that this category has been deleted and can be represented as (termed balanced deletion in Figure 2)
$$
P r o b _ {i} = \frac {\frac {1}{\text {c o u n t} _ {i}}}{\sum \frac {1}{\text {c o u n t} _ {i}}}, i \in \{\text {c l a s s i d s i n R} _ {s l} \}. \tag {12}
$$
We calculate the pixel predictions  $P_{o}^{\prime}$  by re-conducting Eq (6) - (10) with  $R_{sl}^{\prime}$  as input and further calculate the pixel-wise cross-entropy loss  $l$  and  $l^{\prime}$  for  $P_{o}$  and  $P_{o}^{\prime}$ , respectively. After that, we extract the loss values from  $l$  with respect to semantic class id  $j$  as follows,
$$
l _ {j} = \left\{l _ {[ h, w ]} \right| G T _ {[ h, w ]} = j \}. \tag {13}
$$
We also conduct the same operation on  $l^{\prime}$  to obtain  $l_j^\prime$
Given the extracted two sets of loss values  $l_{j}$  and  $l_{j}^{\prime}$ , we can model the relationship between the deleted semantic class id  $i$ -th and one reserved class id  $j$  in  $R_{sl}$ . Specifically, we calculate the mean and variance differences between  $l_{j}$  and  $l_{j}^{\prime}$ ,
$$
r _ {m e a n} ^ {i, j} = \frac {\sum l _ {j} ^ {\prime} - \sum l _ {j}}{K \times H \times W},
$$
$$
r _ {v a r} ^ {i, j} = \frac {\sum \left(l _ {j , [ \cdot ]} ^ {\prime} - \bar {l} _ {j} ^ {\prime}\right) ^ {2} - \sum \left(l _ {j , [ \cdot ]} - \bar {l} _ {j}\right) ^ {2}}{K \times H \times W}, \tag {14}
$$
where  $\bar{l}_j$  and  $\bar{l}_j^\prime$  denote the mean of  $l_{j}$  and  $l_j^{\prime}$ , respectively. Eq (14) indicates that the built semantic-level relations are supposed to help pixel recognition from the perspective of the mean and variance of the loss variation. To utilize such modeled relationship, we maintain two relation matrices  $\mathcal{M}_{r_{mean}}$  and  $\mathcal{M}_{r_{var}}$  during training and update them as
$$
\mathcal {M} _ {r _ {\text {m e a n}}, [ i, j ]} = m _ {\text {m e a n}} \times r _ {\text {m e a n}} ^ {i, j} + (1 - m _ {\text {m e a n}}) \times \mathcal {M} _ {r _ {\text {m e a n}}, [ i, j ]}, \tag {15}
$$
$$
\mathcal {M} _ {r _ {v a r}, [ i, j ]} = m _ {v a r} \times r _ {v a r} ^ {i, j} + (1 - m _ {v a r}) \times \mathcal {M} _ {r _ {v a r}, [ i, j ]},
$$
where  $m_{mean}$  and  $m_{var}$  are the momentum for two matrices and they are both empirically set as 0.1. By re-conducting Eq (6)-(8) with  $\mathcal{M}_{r_{mean}}$  and  $\mathcal{M}_{r_{var}}$  as the relation matrix, we can obtain  $R_{a,mean}$  and  $R_{a,var}$ , respectively. Finally, we combine them to obtain  $R_{a}$  in Eq (9):  $R_{a} = R_{a,mean} \oplus R_{a,var}$ .