Title: IDRNet: Intervention-Driven Relation Network for Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: semantic segmentation

relation modeling

object detection
Abstract: Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, \emph{e.g.}, multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or ineffective contextual information aggregation due to reliance on large amounts of predetermined priors. To alleviate the issues, we propose a novel \textbf{I}ntervention-\textbf{D}riven \textbf{R}elation \textbf{Net}work (\textbf{IDRNet}), which leverages a deletion diagnostics procedure to guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, a deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and the extracted relations are utilized to guide the semantic-level representations to interact with each other.  Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments are conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-driven context scheme brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes.

IDRNet: Intervention-Driven Relation Network for Semantic Segmentation

Abstract

Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, e.g. , multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or ineffective contextual information aggregation due to reliance on large amounts of predetermined priors. To alleviate the issues, we propose a novel I nterventionD riven R elation Net work ( IDRNet ), which leverages a deletion diagnostics procedure to guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, a deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and the extracted relations are utilized to guide the semantic-level representations to interact with each other. Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments are conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-driven context scheme brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes. Code is available at https://github.com/SegmentationBLWX/sssegmentation .

1 Introduction

Semantic image segmentation, which aims at assigning a semantic category to each pixel, is one of the most fundamental research areas in computer vision. Recent years have witnessed great progress in semantic segmentation using deep convolutional neural networks [44] and transformers [64]. The state-of-the-art semantic segmentation approaches [8, 12, 30, 39, 40, 50, 67, 79, 86] mostly follow the encoder-decoder paradigm [51] where the decoder is defined by replacing the fully-connected layers in classification tasks with some convolution layers.

balanced delete

input image

𝑟

!!

⋯

𝑟

!"

⋮

⋱

⋮

𝑟

"!

⋯

𝑟

""

prediction

relation matrix

Group

pixel representations

semantic-level representations

update

semantic-level representations

diagnostic

×

Inspired by co-occurrent visual patterns [25, 63, 83], a popular class of boosting the encoder-decoder architecture can be regarded as incorporating various contextual modules [8, 40, 41, 69, 79, 83] into decoders. Consequently, the segmentation performance improvements are conferred by enabling each pixel to perceive information from other pixels in the input image. For instance, ASPP [8] and PPM [83] exploited dilated convolutions or pyramid pooling layers to aggregate pixel representations at different spatial positions. OCRNet [79] proposed to augment pixel representations with object-level representations. MCIBI++ [39, 41] differentiated the beyond-image contextual information from the within-image contextual information and thus improved pixel representations from the perspective of the whole dataset. To summarize, the existing contextual modules can be grouped into two batches, i.e. , multi-scale-driven [8, 70, 83] and similarity-driven [69, 79, 89] context schemes. Despite these methods achieved astonishing results, both schemes suffer from inadequate or ineffective contextual information aggregation as pixel relations are modeled with masses of predetermined priors. Specific to similarity-driven methods, contextual modules are designed to make pixels aggregate semantically similar contextual representations, which is inadequate for building the co-occurrent visual patterns. Also, some rigorous empirical analysis [5, 75] demonstrates that the contexts of pixels modeled by similarity-driven operations are indistinguishable for different query positions within an input image. As for multi-scale-driven approaches, pixel contexts are assumed to be fixed with certain geometric structures, which may compel pixels to capture some uninformative feature representations. Besides, previous studies [19, 67, 88] indicate that this fixed context assumption prevents the generalization of one-scenario-trained segmentors to other scenarios with unknown geometric transformations.

Different from conventional pipelines of building contextual modules, this paper investigates a more effective paradigm to build relationships among pixels. The proposed intervention-driven paradigm utilizes deletion diagnostics [16] to guide building pixel relations and thus aggregate more meaningful contextual information. In practice, there are two main challenges in constructing such a paradigm: (1) it is time-consuming to perform deletion diagnostics on all pixels and (2) it is memory-consuming to maintain a pixel relation matrix as one image contains plenty of pixels. Motivated by the observation that pixel relationships are closely connected to object relationships since a pixel semantic label is determined by which object region the pixel belongs to, we propose to simplify pixel-level relations modeling as object-level relations modeling to address the above-mentioned challenges.

Figure 1 depicts the diagram of our proposed intervention-driven relation network (IDRNet). We first group pixel representations into semantic-level representations according to predicted pseudo labels. Then, a relation matrix is constructed to make the semantic-level representations interact with each other and thus enhance themselves, where the relation matrix is updated by our deletion diagnostics mechanism. Finally, original pixel representations are augmented with the interacted representations and used to produce pixel labels. Compared with simply performing pixel-level deletion diagnostics, the GPU memory and computational complexity of our method can be significantly reduced. For example, on Cityscapes [17], we only need to maintain a relation matrix with a size 19 × 19 rather than a relation matrix with an input image size of 1024 × 512 during training. As the proposed deletion diagnostics mechanism is designed for semantic-level representations, it is also worth noting that our paradigm is general and can be easily extended to broader applications in other vision tasks, e.g. , object detection [27] and instance segmentation [12].

Table 1 compares the segmentation performance of various context modeling paradigms on Cityscapes and ADE20K benchmark datasets under fair settings ( e.g. , backbone networks and training schedules). As observed, a simple FCN segmentation framework equipped with our intervention-driven context module (termed IDRNet ) can achieve 43 . 61% and 79 . 91% mIoU on the datasets, respectively, which outperforms both dominant similarity-driven and multi-scale-driven context aggregation schemes. To further demonstrate the superiority of our method, we also integrate the proposed context scheme into UPerNet segmentation framework (termed IDRNet+ ), and it is observed that our paradigm with ResNet-50 backbone yields more impressive results on both datasets under single-scale testing.

In principle, our paradigm is fundamentally different from and would complement most similaritydriven and multi-scale-driven contextual modules. The main contribution can be summarized as,

To our best knowledge, for the first time, this paper presents an intervention-driven paradigm, i.e. , deletion diagnostics mechanism, to help model pixel relations.

This paper indicates that, in semantic segmentation task, pixel-level relation modeling can be simplified as building semantic-level relations.

Our intervention-driven context modeling paradigm is conscientiously designed. It can be seamlessly integrated into various state-of-the-art semantic segmentation frameworks and achieve consistent performance improvements on various benchmark datasets, including ADE20K, Cityscapes, COCO-Stuff, LIP and PASCAL-Context.

We expect that IDRNet can provide insights for the future development of semantic segmentation and other related vision tasks.

2 Related Work

2.1 Semantic Segmentation

Semantic segmentation is a fundamental computer vision task, which is an indispensable component for many applications such as autopilot and medical diagnosis. Modern semantic image segmentation methods (in deep learning era) are believed to be derived from FCN [51] which formulates semantic segmentation as an encoder-decoder architecture. After that, numerous efforts have been devoted to improving the pipeline. For example, some tailored encoders like Res2Net [26] and HRNet [66] are designed for semantic segmentation to replace classification networks [33, 72]. In addition, automating network engineering [10, 47, 49] is also one of the most efficient ways to boost the encoder structure. Since transformer [64] attains growing attention, many recent works [1, 15, 21, 30, 50, 55, 62, 67, 71, 78, 86] also introduce transformer structure to help the encoder extract more effective pixel representations. Furthermore, there are also plenty of studies focusing on utilizing cooccurrent patterns to boost the decoder performance, e.g. , modeling long-range dependency [5, 31, 37, 40, 69, 75, 76] and enlarging the receptive field [7, 8, 9, 74, 77, 83]. Besides, some researchers also find that introducing context-aware optimization objectives [2, 42, 84] can strengthen context cues. Apart from the above, some studies also choose to break new ground, e.g. , introducing contrastive learning mechanism [68, 85], formulating the instance, panoptic and semantic segmentation tasks in a unified framework [12, 13, 82], leveraging boundary information [3, 20, 57, 65, 81], to name a few.

This paper studies how to leverage co-occurrent patterns. The key differences are that (1) we transform the pixel relation modeling problem to the semantic-level relation modeling problem and (2) for the first time, we propose to introduce deletion diagnostics mechanism to help model pixel relations.

2.2 Pixel Relation Modeling

As there exist co-occurrent patterns ( i.e. , how likely two semantic classes can exist in a same image) in semantic segmentation, pixel relation modeling becomes a crucial component of recent state-of-the-art semantic segmentation algorithms [5, 36, 40, 70, 69, 75, 77, 79, 83]. For example, non-local network [69] built non-local blocks for capturing long-range dependencies. GCNet [5] unified the non-local and squeeze-excitation operations into a general framework for aggregating more efficacious global contextual information. DNL [75] presented the disentangled non-local block to help learn more discriminative pixel representations. ISA [36] was motivated to improve the efficiency of the selfattention operations and proposed to factorize one dense affinity matrix as the product of two sparse affinity matrices. Besides, OCRNet [79] proposed to emphasize utilizing the contextual information in the corresponding object class. ISNet [40] further improved OCRNet by incorporating image-level and semantic-level contextual information.

ℒ

!

Assign using

𝑃

"

𝑅

#

Group using

𝑃

"

ℱ

$

𝑟

!!

⋯

𝑟

!"

⋮

⋱

⋮

𝑟

"!

⋯

𝑟

""

𝑃

"

𝑅

%

Balanced deletion

ℳ

&

ℒ

"

𝑃

!

Assign using

𝑃

"

𝑅

'

𝑃

!

'

Diagnostic and update

: Only perform during training

{…}

: Optional operations

{similarity-/multi-scale-driven context module}

𝑅

()

…

𝑅

*+

'

…

𝑅

,()

…

×

𝑅

,*+

'

…

×

!

Succinctly, these contextual modules utilize feature similarities or fixed geometric structures to model pixel relations, which injects an unavoidable human priors bias. Distinctively, this paper introduces deletion diagnostics mechanism to guide pixel relation modeling.

2.3 Object Relation Modeling

In object detection, co-occurrence is first applied in DPM [23] to refine object scores. After that, several works [14, 24, 53] were proposed to take object position and size into account for better building object relations in pre-deep learning era. In deep learning era, object relation modeling is also one of the most popular techniques to augment object representations. For instance, relation networks [34] extended original attention module to model both object similarities and relative geometry as the relations between objects. Motivated by relation networks, SGRN [73] took advantage of graph convolutional networks [43] to better bridge object regions by defining them as graph nodes. SMN [11] proposed to adopt a memory mechanism to perform object-object context reasoning.

Considering that each semantic-level representation in our paradigm plays a similar role to an object region in object detection, our intervention-driven relation modeling paradigm can also be extended to object detection tasks to help object relation modeling.

3 Methodology

3.1 Overview of IDRNet

Figure 2 illustrates our proposed intervention-driven relation network (IDRNet). The initial pixel-wise representations are grouped into semantic-level representations to facilitate relationship modeling. In detail, the semantic-level representations are first processed with a feature enhancement module and a feature interaction module with the guidance of a relation matrix M r , where the relation matrix is learned with the proposed deletion diagnostics paradigm. We subsequently augment the initial pixel representations with the processed semantic-level representations for final predictions.

Semantic-level Representation Generation. Following conventional FCN-based architecture [51], we first adopt a backbone network F B to extract the pixel representations R p ∈ R Z × H 8 × W 8 from an input image I ∈ R 3 × H × W , where H and W are height and width of the image and Z is the number of feature channels. To facilitate the seamless integration of our paradigm with existing segmentation frameworks, we formulate the pseudo-label generation process as follows,

where C e denotes an identity function (instantiated as FCN [51]) or a previous similarity- / multi-scaledriven context module (instantiated as OCRNet [79] and PSPNet [83]) and F C is a convolution layer

to project pixel representations into probability space. P c ∈ R K × H 8 × W 8 is the predicted probabilities of K classes and Y c ∈ R H 8 × W 8 is the pseudo label.

As it is computationally impracticable to direct model pixel-level relationships with deletion diagnostics, we propose to simplify pixel-level relations modeling as semantic-level relations modeling, i.e. , we group R p into semantic-level representations R sl by aggregating the corresponding input pixel representations with the same pseudo label,

where R sl ∈ R N e × Z and N e is the number of classes existed in I . F M is a weighted summation operation to merge pixel representations with the corresponding probabilities in P c as the weights.

Feature Enhancement. We further introduce a feature enhancement module to improve the discrimination of R sl to help augment pixel representations R p . Specifically, we modify Eq (2) as

where ⊕ denotes a concatenation operation and R ie ∈ R K × Z denotes discriminative vectors for each class in the training dataset. In our experiments, we investigate two strategies to obtain R ie . The first one is to yield a random orthogonal matrix [52] to represent R ie , while the second one is to learn a dataset-level representation for each category to construct R ie [39]. For the second strategy, we simplify R ie update strategy for class id k as

where m is an update momentum and set as 0 . 1 by default. Before training, we simply initialize R ie as a zero matrix and R gt ∈ R 1 × Z is calculated as

where GT denotes the ground truth segmentation mask and Mean ( · ) is used to average the extracted pixel representations with the same ground truth label.

Feature Interaction. After the feature enhancement module, we propose to make R sl interact with each other to enrich themselves by using a semantic-level relation matrix M r ∈ R K × K , which is updated by the proposed deletion diagnostics mechanism. The procedure of the feature interaction module can be represented as

where we adopt ⊗ to denote matrix multiplication and R esl ∈ R N e × Z is the enhanced semantic-level representations. T is employed to transform M r to adapt the shape of R sl , and it is formulated as

where S is introduced to control that there are no interactions between two semantic-level representations whose relationship in M r is weak. t = 0 is a threshold utilized to identify those weak relationships. In practice, we set M r, [ i,j ] be negative infinity in the calculation if M r, [ i,j ] is smaller than t . Subsequently, we re-arrange R esl into the original pixel representation shape to augment R p ,

where k idx is the corresponding matrix index for class id k in R sl . R a ∈ R Z × H 8 × W 8 are the feature representations initialized as a zero matrix, and we fill it with the representations in R esl according to the pseudo labels in each pixel position.

Final Prediction. We use R a ∈ R Z × H 8 × W 8 to augment R p for final predictions,

where we use the self-attention mechanism SA [64] to further process the augmented representations to balance the diversity and discriminative of the pixels with the same class. Based on R , the final pixel predictions P o ∈ R K × H × W is produced as

where F O is implemented by a convolution layer, which is utilized to produce pixel class probabilities and U is a bilinear interpolation operation.

During training, the overall objective function for backpropagation algorithm [45, 58] is defined as

where L c and L o are the cross entropy losses between P c and GT , and P o and GT , respectively. α is a hyperparameter for balancing L c and L o . We empirically set it as 0 . 4 in our experiments.

3.2 Deletion Diagnostics

Deletion diagnostics is proposed to update the relation matrix M r , which is initialized as an identity matrix. As shown in Figure 2, we randomly delete one semantic-level representation in R sl with a deletion distribution Prob and obtain R ′ sl . The deletion probability Prob i of the i -th semantic class representation is related to the number of times ( i.e. , count i ) that this category has been deleted and can be represented as (termed balanced deletion in Figure 2)

We calculate the pixel predictions P ′ o by re-conducting Eq (6) - (10) with R ′ sl as input and further calculate the pixel-wise cross-entropy loss l and l ′ for P o and P ′ o , respectively. After that, we extract the loss values from l with respect to semantic class id j as follows,

We also conduct the same operation on l ′ to obtain l ′ j .

Given the extracted two sets of loss values l j and l ′ j , we can model the relationship between the deleted semantic class id i -th and one reserved class id j in R sl . Specifically, we calculate the mean and variance differences between l j and l ′ j ,

where l j and l ′ j denote the mean of l j and l ′ j , respectively. Eq (14) indicates that the built semanticlevel relations are supposed to help pixel recognition from the perspective of the mean and variance of the loss variation. To utilize such modeled relationship, we maintain two relation matrices M r mean and M r var during training and update them as

where m mean and m var are the momentum for two matrices and they are both empirically set as 0 . 1 . By re-conducting Eq (6)-(8) with M r mean and M r var as the relation matrix, we can obtain R a,mean and R a,var , respectively. Finally, we combine them to obtain R a in Eq (9): R a = R a,mean ⊕ R a,var .

3.3 Discussion of IDRNet

The procedure of deletion diagnostics looks a lot like Dropout [61], while they are totally different mechanisms. Dropout is a mechanism to prevent deep neural networks from overfitting, while our approach introduces deletion diagnostics to help model pixel relations. In addition, our work also relates closely to some recent segmentation approaches, which extract semantic-level representations to augment pixel representations, e.g. , OCRNet [79] and ISNet [40]. Whereas, these methods only use feature similarities to make connections between pixels, which is distinct from our paradigm as it is intervention-driven and without masses of predetermined priors.

Our paradigm simplifies pixel relations modeling as object relations modeling. Benefiting from this simplification, deletion diagnostics enables the network to directly utilize its objective function to examine whether one class i can help recognize another class j . The key idea behind this mechanism is that loss value of class j should increase after deleting class i if class i can facilitate the prediction

of class j , and the increased value can be adopted to measure the importance of class i to j . Therefore, there is no potential predetermined priors bias in the process of deletion diagnostics. As a comparison, the geometric transformations of multi-scale-driven context schemes are manually set, which tends to aggregate some ineffective information and lacks of generalization. Similarity-driven context schemes are assumed to aggregate semantically similar pixel representations, which results in ignoring other dissimilar but effective pixel representations for building co-occurrent patterns, e.g. , sky pixels for airplane pixels. In a nutshell, our intervention-driven paradigm is more effective compared to the existing similarity- / multi-scale-driven context modules.

4 Experiments

4.1 Experimental Setup

Benchmark Datasets. Our approach is validated on five popular semantic segmentation benchmark datasets, including ADE20K [87], COCO-Stuff [4], Cityscapes [17], LIP [29] and PASCAL-Context [22]. In detail, ADE20K is one of the most well-known datasets for scene parsing, which contains 150 stuff/object category labels. There are 20K/2K/3K images for training, validation and test set, respectively in the dataset. COCO-Stuff is a challenging dataset which provides rich annotations for 91 thing classes and 91 stuff classes. It consists of 9K/1K images in the training and test sets. Cityscapes has 5K high-resolution annotated urban scene images, with 2,975/500/1,524 images for training/validation/testing. It covers 19 challenging categories, such as traffic sign, vegetation and rider. LIP mainly focuses on single human parsing and contains 50K images with 19 semantic human part classes and 1 background class. Its training, validation and test sets separately involve 30K/10K/10K images. PASCAL-Context involves 59 foreground classes and a background class for scene parsing. The dataset is divided into 4,998 and 5,105 images for training and validation.

Implementation Details. Our algorithm is implemented in PyTorch [54] and SSSegmentation [38]. For backbone networks ( i.e. , encoders), they are all pretrained on ImageNet dataset [59]. Remaining layers ( i.e. , decoders) are initialized by the default initialization strategy in PyTorch. As for the data augmentation, some common data augmentation techniques are utilized, including random horizontal flipping, random cropping, color jitter and random scaling (from 0.5 to 2). The learning rate is scheduled by polynomial annealing policy with factor (1 -iter total _ iter ) 0 . 9 .

Specific to ADE20K, we set learning rate, crop size, batch size and training epochs as 0 . 01 , 512 × 512 , 16 and 130 , respectively. Specific to COCO-Stuff, learning rate, crop size, batch size and training epochs are set as 0 . 001 , 512 × 512 , 16 and 110 , respectively. As for LIP, we set learning rate, crop size, batch size and training epochs as 0 . 01 , 473 × 473 , 32 and 150 , respectively. As for Cityscapes, learning rate, crop size, batch size and training epochs are set as 0 . 01 , 512 × 1024 , 8 and 220 , respectively. Specific to PASCAL-Context, we set learning rate, crop size, batch size and training epochs as 0 . 004 , 480 × 480 , 16 and 260 , respectively.

MCIBI++

FCN

OCRNet

ISNet

IDRNet

Image

GT

RD

200

6000

BD

150

5000

4000

100

3000

2000

OC

L000

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

0.0

2.5

5.0

7.5

10.0

Evaluation Metric. Following conventions [7, 51], mean intersection-over-union (mIoU) is utilized for evaluation.

4.2 Ablation Study

Performance Improvements. As seen in Table 2, it is observed that introducing deletion diagnostics to help pixel relationship building can bring consistent performance improvements on different segmentation approaches ( i.e. , FCN, PSPNet, DeeplabV3 and UPerNet) and benchmark datasets ( i.e. , ADE20K, COCO-Stuff, Cityscapes and LIP). By way of illustration, our method boosts FCN, PSPNet, DeeplabV3 and UPerNet by 2.61%, 1.35%, 1.52%, 1.05% mIoU on LIP dataset, respectively.

Feature Enhancement. Feature enhancement is designed to make pixel representations with different classes more discriminative to help model relationship between semantic-level representations R sl . As indicated in Table 3, we can observe that yielding random orthogonal matrix (IE-Orthogonal) and generating dataset-level representations (IE-DL) for feature enhancement are both promising strategies to boost segmentation performance. Specifically, IE-Orthogonal brings 0 . 17% , 0 . 15% and 1 . 07% mIoU gains for ADE20K, PASCAL-Context and COCO-Stuff, respectively. IE-DL achieves 0 . 79% , 0 . 32% and 0 . 85% mIoU improvements for the three datasets, respectively.

Relation Matrix. We also validate the effectiveness of the proposed relation matrix in Table 3. It is observed that M r mean and M r var both contribute to the performance improvements. In detail, M r mean makes our segmentor outperform baseline models (first row) by 6 . 17% , 7 . 18% and 6 . 13% on ADE20K, PASCAL-Context and COCO-Stuff, respectively. M r var boosts baseline models by 5 . 78% , 7 . 38% and 6 . 30% on the three datasets, respectively. The combination of M r mean and M r var finally yields 43 . 61% , 52 . 97% and 38 . 64% mIoU on the three datasets, respectively.

Deletion Diagnostics. In Table 4, we compare our proposed deletion diagnostics procedure (DDdriven M r ) with back-propagation updating strategy (BP-driven M r ). We can observe that adopting DD-driven M r outperforms using BP-driven M r by 3.26% mIoU on ADE20k. The result suggests that our deletion diagnostics strategy is more effective than the back-propagation updating strategy.

Balanced Deletion. Balanced deletion is utilized to help improve the being-diagnosed probabilities of uncommon categories in training dataset and thereby, deletion diagnostics can be performed evenly. In Table 3, we can observe that introducing the balanced deletion strategy can help improve the segmentation performance by 0 . 71% , 0 . 05% and 1 . 05% for ADE20K, PASCAL-Context and COCO-Stuff, respectively. Also, we count and compare the sampling frequency of each category when using balanced deletion (BD) and random deletion (RD) in Figure 4. We can observe that the sampling frequency of uncommon categories have increased after utilizing BD as we expected.

Complexity Comparison. We also present analysis on complexity of our intervention-driven context module following the settings of OCR [79]. In Table 5, it is observed that our IDRNet requires the least Parameters, FLOPS and GPU Memory, while achieving the best mIoU on the ADE20k dataset. Moreover, our IDRNet is complementary to other context modules. For example, when incorporated with PPM (termed "PPM+IDRNet" in the table), the Parameters, FLOPS, Time and GPU Memory only increase by 0.58 M, 39.78 G, 11.19 ms and 73.65 M, respectively, which shows that our proposed IDRNet is light and efficient during network inference. Note that few parameters

Table 6: Compare segmentation performance in cross domain.

increase in PPM+IDR as we adopt a shared 3 × 3 convolution layer to reduce the feature channels of the backbone outputs.

Qualitative Results. Figure 3 provides qualitative comparison of the proposed approach against FCN [51], OCRNet [79], ISNet [40] and MCIBI++[41] on representative examples in various benchmark datasets. It is observed that IDRNet can output better segmentation results.

4.3 Cross Domain Segmentation Performance.

Table 6 compares the cross-domain segmentation performance. It can be observed that IDRNet can also boost cross-domain segmentation performance, which demonstrates the generalization of our intervention-driven pixel relation modeling paradigm. For example, DeeplabV3+IDRNet trained on day-time domain ( i.e. , Cityscapes) can also bring 3 . 63% and 1 . 94% mIoU gains to Dark Zurich [60] and Nighttime Driving [18] datasets which belong to night-time domain. In addition, after training FCN+IDRNet on single-human parsing scenarios ( i.e. , LIP), it also boosts the segmentation performance on multi-human parsing scenarios ( i.e. , CIHP [28]) by 1 . 39% mIoU.

4.4 Comparison with State-of-the-art Methods

Table 7 compares the quantitative results on four challenging benchmark datasets, including ADE20K val , LIP val , PASCAL-Context val and COCO-Stuff test . As for ADE20K val , it is observed that prior to this paper, Mask2Former [12] with Swin-Large backbone [41] achieves the state-of-the-art performance hitting 57 . 30% mIoU. By incorporating Mask2Former with IDRNet, it yields 58 . 22% mIoU which surpasses other state-of-the-art segmentation methods. Specific to LIP val , we can observe that among the previous SOTA methods, UPerNet+MCIBI++ with Swin-Large backbone obtains the state-of-the-art result, i.e. , 59 . 91% mIoU. Our method, UPerNet+IDRNet with SwinLarge backbone, outperforms it by 1 . 26% and reports the new SOTA result, 61 . 17% mIoU. With regard to PASCAL-Context val , our method, UPerNet+IDRNet with Swin-Large backbone, surpasses all the competitors, i.e. , 0 . 50% over UPerNet-MCIBI++, 5 . 51% over Segmenter, 8 . 68% over SETR, 9 . 71% over OCRNet, to name a few. On the subject of COCO-Stuff test , we can observe that our UPerNet+IDRNet with Swin-Large backbone reaches 50 . 50% mIoU which is 0 . 23% higher than previous state-of-the-art method UPerNet+MCIBI++.

4.5 Extend Deletion Diagnostics to Object Detection

In this section, we apply deletion diagnostics mechanism to the object detection frameworks to further demonstrate the effectiveness and generalization of our method.

Implementation Details. Our experiments are conducted on MS COCO dataset [48]. All algorithm implementations are based on PyTorch [54] and MMDetection [6], and we follow the default settings in MMDetection for model training and testing.

Experimental Results. As illustrated in Table 8, the proposed deletion diagnostics mechanism is integrated into two popular object detection frameworks, i.e. , Faster R-CNN [56] and Mask R-CNN [32]. We can observe that intervention-driven relation building for objects brings 0 . 8% and 1 . 5% AP improvements for the two detection frameworks, respectively.

5 Conclusion

This work studies how to utilize deletion diagnostics to model pixel relationships. The key contribution consists of two parts: (1) we simplify pixel relation modeling as object relation modeling and (2) we introduce deletion diagnostics to make the networks build object relationships by perceiving changes in the outputs of the objective functions. Compared to similarity-/multi-scale-driven context modeling paradigms, there are fewer human priors in our context modeling paradigm. Experimental results demonstrate the effectiveness of our proposed paradigm qualitatively and quantitatively.

References

H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

M. Berman, A. R. Triki, and M. B. Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4413-4421, 2018.

G. Bertasius, J. Shi, and L. Torresani. Semantic segmentation with boundary neural fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3602-3610, 2016.

H. Caesar, J. Uijlings, and V . Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In Proceedings of the IEEE/CVF international conference on computer vision workshops , pages 0-0, 2019.

K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 801-818, 2018.

W. Chen, X. Gong, X. Liu, Q. Zhang, Y. Li, and Z. Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv preprint arXiv:1912.10917 , 2019.

X. Chen and A. Gupta. Spatial memory for context reasoning in object detection. In Proceedings of the IEEE international conference on computer vision , pages 4086-4096, 2017.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022.

B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:17864-17875, 2021.

M. J. Choi, A. Torralba, and A. S. Willsky. A tree-based context model for object recognition. IEEE transactions on pattern analysis and machine intelligence , 34(2):240-252, 2011.

X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems , 34:9355-9366, 2021.

R. D. Cook. Detection of influential observation in linear regression. Technometrics , 19(1):15-18, 1977.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

D. Dai and L. Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) , pages 3819-3824. IEEE, 2018.

J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 764-773, 2017.

H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, and G. Wang. Boundary-aware feature propagation for scene segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6819-6829, 2019.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010.

P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence , 32(9):16271645, 2009.

C. Galleguillos and S. Belongie. Context based object categorization: A critical survey. Computer vision and image understanding , 114(6):712-722, 2010.

C. Galleguillos, A. Rabinovich, and S. Belongie. Object categorization using co-occurrence, location and appearance. In 2008 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8. IEEE, 2008.

S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence , 43(2):652-662, 2019.

R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015.

K. Gong, X. Liang, Y. Li, Y. Chen, M. Yang, and L. Lin. Instance-level human parsing via part grouping network. In Proceedings of the European conference on computer vision (ECCV) , pages 770-785, 2018.

K. Gong, X. Liang, D. Zhang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 932-940, 2017.

M.-H. Guo, C.-Z. Lu, Q. Hou, Z. Liu, M.-M. Cheng, and S.-M. Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08575 , 2022.

J. He, Z. Deng, L. Zhou, Y. Wang, and Y. Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7519-7528, 2019.

K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3588-3597, 2018.

X. Hu, K. Yang, L. Fei, and K. Wang. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 1440-1444. IEEE, 2019.

L. Huang, Y. Yuan, J. Guo, C. Zhang, X. Chen, and J. Wang. Interlaced sparse self-attention for semantic segmentation. arXiv preprint arXiv:1907.12273 , 2019.

Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 603-612, 2019.

Z. Jin. Sssegmenation: An open source supervised semantic segmentation toolbox based on pytorch. arXiv preprint arXiv:2305.17091 , 2023.

Z. Jin, T. Gong, D. Yu, Q. Chu, J. Wang, C. Wang, and J. Shao. Mining contextual information beyond image for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7231-7241, 2021.

Z. Jin, B. Liu, Q. Chu, and N. Yu. Isnet: Integrate image-level and semantic-level context for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7189-7198, 2021.

Z. Jin, D. Yu, Z. Yuan, and L. Yu. Mcibi++: Soft mining contextual information beyond image for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

T.-W. Ke, J.-J. Hwang, Z. Liu, and S. X. Yu. Adaptive affinity fields for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 587-602, 2018.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature , 521(7553):436-444, 2015.

Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school , volume 1, pages 21-28, 1988.

X. Li, Z. Zhong, J. Wu, Y. Yang, Z. Lin, and H. Liu. Expectation-maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9167-9176, 2019.

Y. Li, L. Song, Y. Chen, Z. Li, X. Zhang, X. Wang, and J. Sun. Learning dynamic routing for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8553-8562, 2020.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, and L. Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 82-92, 2019.

Z. Liu, Y. Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

F. Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050 , 2006.

R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

Z. Peng, L. Dong, H. Bao, Q. Ye, and F. Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366 , 2022.

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

T. Ruan, T. Liu, Z. Huang, Y. Wei, S. Wei, and Y. Zhao. Devil in the details: Towards accurate single and multiple human parsing. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4814-4821, 2019.

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. nature , 323(6088):533-536, 1986.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

C. Sakaridis, D. Dai, and L. V . Gool. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7374-7383, 2019.

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7262-7272, 2021.

A. Torralba, K. P. Murphy, W. T. Freeman, and M. A. Rubin. Context-based vision system for place and object recognition. In Computer Vision, IEEE International Conference on , volume 2, pages 273-273. IEEE Computer Society, 2003.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

C. Wang, Y. Zhang, M. Cui, P. Ren, Y. Yang, X. Xie, X.-S. Hua, H. Bao, and W. Xu. Active boundary loss for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2397-2405, 2022.

J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence , 43(10):3349-3364, 2020.

W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.05778 , 2022.

W. Wang, T. Zhou, F. Yu, J. Dai, E. Konukoglu, and L. Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7303-7313, 2021.

X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7794-7803, 2018.

T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV) , pages 418-434, 2018.

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077-12090, 2021.

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

H. Xu, C. Jiang, X. Liang, and Z. Li. Spatial-aware graph relation network for large-scale object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 92989307, 2019.

M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang. Denseaspp for semantic segmentation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3684-3692, 2018.

M. Yin, Z. Yao, Y. Cao, X. Li, Z. Zhang, S. Lin, and H. Hu. Disentangled non-local neural networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16 , pages 191-207. Springer, 2020.

C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, and N. Sang. Context prior for scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12416-12425, 2020.

F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 , 2015.

W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10819-10829, 2022.

Y. Yuan, X. Chen, and J. Wang. Object-contextual representations for semantic segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 , pages 173-190. Springer, 2020.

Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, and J. Wang. Ocnet: Object context for semantic segmentation. International Journal of Computer Vision , 129(8):2375-2398, 2021.

Y. Yuan, J. Xie, X. Chen, and J. Wang. Segfix: Model-agnostic boundary refinement for segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 , pages 489-506. Springer, 2020.

W. Zhang, J. Pang, K. Chen, and C. C. Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems , 34:10326-10338, 2021.

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

S. Zhao, Y. Wang, Z. Yang, and D. Cai. Region mutual information loss for semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

X. Zhao, R. Vemulapalli, P. A. Mansfield, B. Gong, B. Green, L. Shapira, and Y. Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10623-10633, 2021.

S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6881-6890, 2021.

B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633-641, 2017.

X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9308-9316, 2019.

Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 593-602, 2019.

Peer reviews:
comments: SUMMARY:
This study focuses on the utilization of deletion diagnostics for modeling pixel relationships. The main contribution of this work can be divided into two parts. Firstly, the simplification of pixel relation modeling as object relation modeling. Secondly, the introduction of deletion diagnostics to enable networks to build object relationships by detecting changes in the outputs of objective functions. Experimental results qualitatively and quantitatively demonstrate the effectiveness of the proposed paradigm.

STRENGTHS:
* The use of Deletion Diagnostics to update the relation matrix is a novel approach that considers the problem of category context in semantic segmentation from a different perspective than previous multi-scale driven and similarity-driven methods.

* Thorough ablation experiments were conducted to verify the impact of different designs of IDRNet on segmentation performance.

* The writing is clear and easily understandable for readers.

WEAKNESSES:
* The experiments of Mask2Former+IDRNet only report results on one dataset, ADE20K. As Mask2Former is a general segmentation framework, it is recommended that the authors also report the performance of the proposed model on the other three datasets for a comprehensive comparison.

* Additional experiments based on Mask2Former could be added to Table 2 to verify the effectiveness of IDRNet on a regular backbone.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
There seems no discussion about limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents IDRNet, a context aggregation scheme for semantic segmentation networks which is driven by intervention in the object query stage. The work is a follow up on OCR (object-contextual representations), but adds to the previous work by an intervention-based strategy to update, augment, or delete tokens. The updates to these tokens are based on semantic relations.\n\nSTRENGTHS:\nPresentation:\nThe paper is clear in its presentation of the concept, and the result. Tables and figures are self-explanatory and it is easy to follow the though process of the authors. The reviewer appreciates the effort spent in explaining the concept of this work.\n\nResults: \nAs a scheme which is built to add to pixel representations, the results are quite impressive on five different datasets. The authors have also provided ablation studies detailing the effect of each separate block in the proposed algorithm.\n\nVisulalizations:\nThe reviewer also appreciates the detail the authors have provided on viewing semantic relationships, which was the major motivation of their approach. Hence, it is safe to say that the motivation of the work and results go hand in hand.\n\nWEAKNESSES:\nOriginality:\nThe concept of updating (deleting, augmenting, enhancing, \"diagnosing\") embeddings based on object relations might seem novel when you think of them in a convolution-based setting. However, I am afraid this concept is not original in transformer networks. Works such as Mask2Former do present queries which describe objects and the network learns (based on bipartite matching) to associate queries with the concept of object. There are also several other works [1], [2], [3] which prune, refine or update these tokens based on semantic relationships. \n\nComplexity:\nAs the method adds additional compute to the network, the authors mention how much compute is added and also, more importantly, is Baseline+IDRNet better than simply scaling up the baseline to match this compute.\n\n[1] Revisiting Token Pruning for Object Detection and Instance Segmentation\n[2] Sparse Tokens for Dense Prediction - The Medical Image Segmentation Case\n[3] DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive\nSegmentation Transformer\n\nQUESTIONS:\n1. Please could describe how this method is different from other works performing token pruning\n2. Please could you provide some analysis on complexity/runtime related to other context aggregation schemes such as OCR\n3. Please could you provide some limitations of the approach\n\nLIMITATIONS:\nThe authors have not addressed limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a context relation modeling method, which can be incorporated with existing semantic segmentation model. Instead of modeling pixel-level relation, it models class-level semantic relation by weighted averaging features at each location and deletion diagnostics. The proposed module is flexible and tested with different network backbone and segmentation architectures, which observes consistent improvements on several widely used datasets. Furthermore, the proposed context relation module can be applied in object detection, that stronger results on MS COCO are shown based on Faster-RCNN.\n\nSTRENGTHS:\n+ This paper presents a context relation module, which can be combined with previous semantic segmentation models. It is used to enhance the learned features and perform features interaction with global co-cocurrence. Besides, in order to overcome the heavy computation issue, this work proposes to use semantic relation to replace pixel relation.\n\n+ This paper conduct extensive experimental results, including 4 popular datasets. Besides, this work is combined with many previous segmentation models. For example, even though PSPNet can aggregate context information, the semantic relation module can further improve its performance.\n\n+ Comparison with previous state-of-the-art demonstrates the effectiveness of the proposed module.\n\n+ This work is also compatible with object detection framework. It should be also feasible to apply the proposed module in semantic instance segmentation.\n\nWEAKNESSES:\n- Too many symbols in section 3. For some details, I cannot follow.\nIn Eq 3, what is the output dimension for R_{sl} ?\nIn Eq 6, why is R_{esl} a Ne xZ matrix, as M_r is KxK ?\nIn Eq 8, what are the values in R_a if Y_c != k ?\nIn Eq 9, what does the \\bigoplus operation represent ?\n\n- Figure 5 is hard to see the graph connection, for example, the rider class. For sky, it is connected to truck, building, road, which does not seem to make sense.\n\n- In Table 5, why not show the performance for Mask2Former+IDRNet on LIP/COCO-Stuff/PASCAL-Context datasets?\n\n- Suggest to list all the comparing methods.\n\nQUESTIONS:\n1. Actually, you do not have to use deletion diagnostics to update M.  What is the motivation of deletion diagnostics? What kind of issues does the proposed deletion aim to address?\n\nLIMITATIONS:\nSee weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo further enhance the segmentation performance, the authors utilize deletion diagnostics to model pixel relationships. Specifically, to address the computational cost of pixel relation modelling, they simplify it as object relation modelling. Then, they introduce deletion diagnostics to facilitate the network in building object relationships. The experimental results qualitatively and quantitatively demonstrate the effectiveness of their proposed paradigm.\n\nSTRENGTHS:\nTo improve the contextual information aggregation，the authors propose a novel Intervention-Driven Relation Network (IDRNet), which leverages a deletion diagnostics procedure to guide the modelling of contextual relations among different pixels. This method brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets.\n\nWEAKNESSES:\n1. The paper lacks the presentation of additional time-consuming and memory-consuming aspects associated with the Intervention-Driven context module.\n2. The paper lacks interpretability regarding how the Intervention-Driven context module improves contextual information aggregation.\n3. Some minor issues: confusion in the writing, some symbols used in equations or throughout the paper are not adequately explained.\n\nQUESTIONS:\n1. There is confusion in the writing. For example, in the paper, IDRNet refers to an additional module that enhances contextual information aggregation. However, in Table 1, IDRNet is described as a simple FCN segmentation framework equipped with IDRNet. This inconsistency can cause confusion for readers.\n2. Without using the Intervention-Driven context module, would combining the Multi-Scale-Driven and Similarity-Driven modules yield better results?\n3. What is the extent of the additional time-consuming and memory-consuming aspects associated with the intervention-Driven context module?\n4. In equation (8), the meaning of $R_a$ is not explained.\"\n5. In equation (9), the meaning of $\\oplus$ is not explained. Is it element-wise addition or channel-wise merging?\n\nLIMITATIONS:\nThere is no limitation claimed in the paper. For example, the cost of the Intervention-Driven context module, particularly in terms of training time and storage, remains unknown.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}