IDRNet: Intervention-Driven Relation Network for Semantic Segmentation
Authors: Zhenchao Jin
Xiaowei Hu
Lingting Zhu
Luchuan Song
Li Yuan
Lequan Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: semantic segmentation
relation modeling
object detection

Abstract: Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, \emph{e.g.}, multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or ineffective contextual information aggregation due to reliance on large amounts of predetermined priors. To alleviate the issues, we propose a novel \textbf{I}ntervention-\textbf{D}riven \textbf{R}elation \textbf{Net}work (\textbf{IDRNet}), which leverages a deletion diagnostics procedure to guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, a deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and the extracted relations are utilized to guide the semantic-level representations to interact with each other.  Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments are conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-driven context scheme brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes.

IDRNet: Intervention-Driven Relation Network for Semantic Segmentation
Zhenchao Jin', Xiaowei Hu', Lingting Zhu', Luchuan Song', Li Yuan Lequan Yu'
'The University of Hong Kong Shanghai AI Laboratory
SPeking University University of Rochester
(blux96@connect., 1tzhu998comnect., 1qyu@)hku.
huxiaovei@pjlab. org.cn yuanli coeniieceeppuu.co edu.
Lsongl10ur rochester.. roccheter edu
Abstract
Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, e.g., multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or inef- fective contextual information aggregation due reliance on large amounts of pre- determined priors. To alleviate the issues, we propose a novel Intervention-Deee Driven Relation Network (IDRNet), which leverages deletion diagnostics procedure guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and extracted relations are utilized to guide the semantic-level representations interact with each other. Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-drive context scheme brings consistent performance improvements tion frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stufi PASCAL--Cntext, LIP, and Cityscapes. Code is available at https: //t::://gtttt thub. com/ Segnentation httprooonoo.L//ssseeentsaa.a.. sssogmentatinnnnn
sate-offthe the-art segmenta-
Table 1: Performance comparison between existing context schemes and our IDRNet.
Multi-Scale-Driven
Similarity-Drive
Intervention-riveer
Benchmark
ASPP
PPM [3]
UPerNer 70
Non-Local (69
Anx [89
79
IDRNer
IDRNer+
ADE20K 87
43.19
42.64
43.02
42.15
41.75
42.47
43.61
44.4
Cityscapes [17
79.62
79.05
79.08
78.34
78.36
79.40
79.91
80.81
Introduction
Semantic image segmentation, which aims at assigning semantic category to each pixel, is one of the most fundamental research areas in computer vision. Recent years have witnessed great progress in semantic segmentation using deep convolutional neural networks 44 and transformers [64]. The
state-of-the-art semantic segmentation approaches [ I2
30 39
40
50 67 79 86) mostly follow
the encoder-decoder paradigm [51] where the decoder is defined by replacing the fully-connected layers in classification tasks with some convolution layers.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

pixel representations
prediction
input image
11
PiK
Group
relation matrix +
TKK
balanced delete
updote
semantic-level representations
semantic-level representations
diagnostic
Figure I: Diagram of our intervention-driven relation network. Deletion diagnostics is leveraged to build relations between semantic- representations. With the built relation matrix and semantic- level representations, pixel representations can be augmented for pixel prediction.
Inspired by co-occurrent visual patterns 22 63 83 a popular class of boosting the mecddrr
architecture can be regarded as incorporating various contextual modules [8 40 41 669 (7 [ into decoders. Consequently, the segmentation performance improvements are conferred by enabling each pixel to perceive information from other pixels in the input image. For instance, ASPP (8] and PPM 83 exploited dilated convolutions or pyramid pooling layers to aggregate pixel representations at different spatial positions. OCRNet [79] proposed to augment pixel representations with object-level representations. MCIBI+ 39 41 differentiated the beyond-ing Seyond-image contextual information from the within- contextual information and thus improved pixel representations from the perspective of the whole dataset. To summarize, the existing contextual modules can be grouped into two batches,
79 39 context schemes. Despite these methods achieved astonishing results, both schemes suffer from inadequate or ineffective contextual information aggregation as pixel relations are modeled with masses of predetermined priors. Specific to similarity-driver methods, contextual modules are designed to make pixels aggregate semantically similar contextual representations, which is inadequate for building the co-occurrent visual patterns. Also, some rigorous empirical analysis [5] 75 demonstrates that the contexts of pixels modeled by similarity-driver operations are indistinguishable for different query positions within an input image. As for multi-scale-dries
ie., mulli-scale-driven 8 70 83 and similarity-driver [69
approaches, pixel contexts are assumed to be fixed with certain geometric structures, which may compel pixels to capture some uninformative feature representations. Besides, previous studies [19 67 [8 indicate that this fixed context assumption prevents the generalization of one-scenario-traine segmentors to other scenarios with unknown geometric transformations.
Different from conventional pipelines of building contextual modules, this paper investigates a more effective paradigm build relationships among pixels. The proposed intervention-drived paradigm utilizes deletion diagnostics [16] to guide building pixel relations and thus aggregate more meaningful contextual information. In practice, there are two main challenges constructing such a paradigm: (I) it is time-consuming to perform deletion diagnostics on all pixels and (2) it is memory-consuming to maintain a pixel relation matrix as one image contains plenty of pixels. Motivated by the observation that pixel relationships are closely connected to object relationships since pixel semantic label is determined by which object region the pixel belongs to, we propose simplify pixel-level relations modeling as object-level relations modeling to address the above-mentioned challenges.
Figure D depicts the diagram of our proposed intervention-drival relation network (IDRNet). We first group pixel representations into semantic-level representations according to predicted pseudo labels. Then, relation matrix is constructed to make the semantic-level representations interact with each other and thus enhance themselves, where the relation matrix is updated by our deletion diagnostics mechanism. Finally, original pixel representations are augmented with the interacted representations and used to produce pixel labels. Compared with simply performing pixel-level deletion diagnostics, the GPU memory and computational complexity of our method can be significantly reduced. For example, on Cityscapes [17], we only need to maintain relation matrix with size 19 x 19 rather than a relation matrix with an input image size of 1024 512 during training. As the proposed deletion diagnostics mechanism is designed for semantic-level representations, it is also worth noting that our paradigm is general and can be easily extended broader applications other vision tasks,
e.g., object detection (27) and instance segmentation 222

---

T compares the segmentation performance of various context modeling paradigms on Cityscapes and ADE20K benchmark datasets under fair settings (e.g.. backbone networks and training schedules). As observed, simple FCN segmentation framework equipped with our intervention-drive module (termed IDRNer) can achieve 43.61% % and 79.91% mloU on the datasets, respectively, which outperforms both dominant amilarity-drriv and further demonstrate the superiority of our method, we also integrate the proposed context scheme into UPerNet segmentation framework (termed IDRNer+), ), and it is observed that our paradigm with ResNet-50 backbone yields more impressive results on both datasets under single-scale testing.
context
context aggregation schemes. To
In principle, our paradigm is fundamentally different from and would complement most similarity- driven and multi-scale scalei-eaale-rive contextual modules. The main contribution can be summarized as,
To our best knowledge, for the first time, this paper presents an intervention-drive paradigm, i.e. deletion diagnostics mechanism, to help model pixel relations.
This paper indicates that, in semantic segmentation task, pixel-level relation modeling can be simplified as building semantic-leve relations.
Our intervention-driven context modeling paradigm is conscientiously designed. It can be seamlessly integrated into various state-of-the-an semantic segmentation frameworks and achieve consistent performance improvements on various benchmark datasets, including ADE20K, Cityscapes, COCO-Sturt Stuff,, LIP and PASCAL-Context.
We expect that IDRNet can provide insights for the future development of semantic segmentation and other related vision tasks.
2 Related Work
2.1 Semantic Segmentation
Semantic segmentation is fundamental computer vision task, which is an indispensable component for many applications such as autopilot and medical diagnosis. Modern semantic image segmentation methods (in deep learning era) are believed to be derived from FCN [51] which formulates semantic segmentation as an encoder-decoder architecture. After that, numerous efforts have been devoted to improving the pipeline. For example, some tailored encoders like Res2Net (26) and HRNet [66] In addition,
are designed for semantic segmentation to replace classification networks 33 72
automating network engineering 110 47
49
is also one of the most efficient ways to boost the
encoder structure. Since transformer
64 attains growing attention, many recent works [1] 21
30
also introduce transformer structure to help the encoder extract more effective pixel representations. Furthermore, there are also plenty of studies focusing on utilizing CO- occurrent patterns to boost the decoder performance, e.g., modeling long- rangre dependency
50 55
62 67
78 86
37
[40 69, 75 76) and enlarging the receptive field 7 8 74, 77
83 Besides, some researchers also can strengthen context cues. Apart from the above, some studies also choose to break new ground, e.g., introducing contrastive learning mechanism
find that introducing context-aware optimization objectives
2
44
668 formulating the instance, panoptic and semantic segmentation tasks in leveraging boundary information 3 20
a unified framework
12 13
57
65
81. to name a few.
This paper studies how to leverage co-occurrent patterns. The key differences are that (1) we transform the pixel relation modeling problem to the semantic-level relation modeling problem and (2) for the first time, we propose introduce deletion diagnostics mechanism help model pixel relations.
2.2 Pixel Relation Modeling
As there exist co-occurrent patterns (i.e., how likely two semantic classes can exist in a same image) in semantic segmentation, pixel relation modeling becomes a crucial component of recent state-of-the-ant
semantic segmentation algorithms [5] 36 40 70 69 75 77 79 83]. For example, non-local network [69 built non-local blocks for capturing long-range dependencies. GCNet [5] unified the non-local and squeeze- equeeze-excitation operations into general framework for aggregating more efficacious global contextual information. DNL 775 presented the disentangled non-local block to help learn more discriminative pixel representations. ISA [3 was motivated to improve the efficiency of the self- attention operations and proposed to factorize one dense affinity matrix as the product of two sparse affinity matrices. Besides, OCRNet [79] proposed to emphasize utilizing the contextual information in the corresponding object class. ISNet (10 further improved OCRNet by incorporating image-level and semantic-level contextual information.
3

---

(stuiiurity draip-rriee modile)
Only profarm during tratning
Optional sperations
Group
inser Interact
Assige
Befine poodier
Ru
The
Bolanced deletion!
Mr
Diognontic
Pin
Yett
Refine
insart and intemnet
R
Figure 2: Illustration of our intervention-drivn relation network (IDRNet). We first extract pixel representations RS using a backbone network FB, e.g., ResNet 33 or SwinTransffrmer [50]. Then, Rp is grouped into semantic-level representations Ret based we leverage a relation matrix M, to make Rgi interact with each other and further obtain enhanced semantic-level representations Rest. Finally, R, is augmented by Rest and utilized for final pixel predictions Pa- After each training iteration, the deletion diagnostics conducted to update M,,
coarse pixel prediction P.- Next,
Succinetly, these contextual modules utilize feature similarities fixed geometric structures model pixel relations, which injects an unavoidable human priors bias. Distinctively, this paper introduces deletion diagnostics mechanism guide pixel relation modeling.
2.3 Object Relation Modeling
In object detection, co-occurrence first applied DPM 23 to refine object scores. After that, several works [14] [4]] [33] were proposed to take object position and size into account for better building object relations in pre-deep learning era. In deep learning cra, object relation modeling also of the most popular techniques augment object representations. For instance, relation networks 34 extended original attention module to model both object similarities and relative geometry as the relations between objects. Motivated by relation networks, SGRN 77) took advantage of graph convolutional networks 43 to better bridge object regions by defining them as graph nodes. SMN | proposed adopt memory mechanism to perform object-object context reasoning.
Considering that each semantic-level representation our paradigm plays similar role to an object region in object detection, our intervention-drin relation modeling paradigm can also be extended to object detection tasks to help object relation modeling.
3 Methodology
3.1 Overview of IDRNet
iiluuttillsstrttess our proposed intervention-driven relation network (IDRNet). The initial pixel-wise representations are grouped into semantic-level representations facilitate relationship modeling. In detail, the semantic-level representations are first processed with feature enhancement module and a feature interaction module with the guidance of a relation matrix M,, where the relation matrix is learned with the proposed deletion diagnostics paradigm. We subsequently augment the initial pixel representations with the processed semantic-level representations for final predictions.
Semantic-level Representation Generation. Following conventional FCN-based architecture [5]] first adopt backbone network FB to extract the pixel representations R,p x " from an input image I E |x3 where H and W are height and width of the image and Z is the number of feature channels. To facilitate the seamless integration our paradigm with existing segmentation frameworks, we formulate the pseudo-label generation process follows,
P. = (R,)), Y. argmma((P.)
where C, denotes an identity function (instantiated as FCN (TT) or previous similarity- multi-scale- driven context module (instantiated OCRNet [79 and PSPNet (83 and Fo is convolution layer

---

to project pixel representations into probability space. Pe E RRKXX is the predicted probabilities of K classes and Ye E IR $ $ is the pseudo label.
As is computationally impracticable to direct model pixel-level relationships with deletion diagnos- ties, we propose to simplify pixel-level relations modeling as semantic-leve relations modeling, i.e. we group Ry into semantic-level representations Rat by aggregating the corresponding input pixel representations with the same pseudo label,
Rat = {Fir (( d Yejs,,1 - =k)) (0,
where Rot E R2 and Ne is the number of classes existed in I. FM is weighted summation operation to merge pixel representations with the corresponding probabilities in Pe as the weights.
Feature Enhancement. We further introduce feature enhancement module to improve the discrimi- nation of Rgl to help augment pixel representations Ry. Specifically, we modify Eq (2) as
Fax (Rp. s, J] ① [ie,[s..]] +) Ya j] k)) E [O.
Rgt
where @ denotes concatenation operation and Rie E RKx2 denotes discriminative vectors for each class the training dataset. In our experiments, we investigate two strategies to obtain Ric- The first one is to yield random orthogonal matrix [52) to represent Ric, while the second one is to learn a dataset-level representation for each category to construct Rie [39]. For the second strategy, we simplify Rie update strategy for class id k as
m) + Rgt x m, where m an update momentum and set as 0.1 by default. Before training, we simply initialize R as zero matrix and Ryt E IR. xxx is calculated as
Rie,[R.-]] [R,, (1-m)
(4)
Rit Mean({Rp.t e, 5l , = kJ), & E [0, [0,
(5)
where GT denotes the ground truth segmentation mask and Mcan(.) is used to average the extracted pixel representations with the same ground truth label.
Feature Interaction. After the feature enhancement module, we propose to make Rg interact with each other to enrich themselves by using a semantic-level relation matrix M, IR KxK, which is updated by the proposed deletion diagnostics mechanism. The procedure of the feature interaction module can be represented as
Rest Softmax(T(MM döm = 1) @ Rai:
where we adopt @ to denote matrix multiplication and Rest E R.N, x is the enhanced semantic-level representations. T is employed to transform M, to adapt the shape of Rai and is formulated as
M, T(M)) = (S.),,, ú,j E {class ids in Ra));
where introduced to control that there are no interactions between two semantic-level repre- sentations whose relationship in MM, is weak. t = 0 is a threshold utilized to identify those weak relationships. In practice, we set M,. be negative infinity the calculation if M,.[(.Ji [i] is smaller than t. Subsequently. we re-arrange R,st into the original pixel representation shape to augment Rp.
Ra.(-.is]] Rest,[ if Ya((-)] k. k {class ids in Rat ).
(8)
where kide the corresponding matrix index for class id le in Rat R. E 2x W are the feature representations initialized as a zero matrix, and we fill it with the representations in Rest according to the pseudo labels in each pixel position.
Final Prediction. We use Rs e I x
W to augment R, for final predictions,
R
Ra Ce R = SA(R).
where we use the self-attention mechanism SA [64] to further process the augmented representations to balance the diversity and discriminative of the pixels with the same class. Based on R, the final pixel predictions P. E B x HxW is produced as
P. = Usx (Fo()))
(10)
5

---

where Fo is implemented by convolution layer, which is utilized to produce pixel class probabilities and U is a bilinear interpolation operation.
During training, the overall objective function for backpropagation algorithm 45
58
defined as
C = + Lo.
(11)
where L. and L., are the cross entropy losses between P. and GT, and P. and GT, respectively. is a hyperparameter for balancing L. and L.. We empirically set 0.4 in our experiments.
3.2 Deletion Diagnostics
Deletion diagnostics is proposed to update the relation matrix Mr. which is initialized as identity matrix. As shown in Figure 2 we randomly delete one semantic-level representation in Rst with a deletion distribution Prob and obtain R²z. The deletion probability Prob; of the i-th semantic class representation is related to the number of times (i.e., counl;) that this category has been deleted and can be represented (termed balanced deletion in Figure
Prob
count,
i E {class ids in R
(12)
count,
We calculate the pixel predictions P.d by re-conducting Eq 66 (IO) with Ra as input and further calculate the pixel-wise cross-entropy loss I and I' for P. and P. respectively. After that, we extract the loss values from with respect semantic class id j as follows,
1j ui]
W [T = =j}.
(13)
We also conduct the same operation on l' to obtain 1z-
Given the extracted two sets of loss values 1 and deleted semantic class id i-th and one reserved class i we can model the relationship between the and variance differences between I and 1',
j in Rug. Specifically, we calculate the mean
ΣV - E4
Jij mean
K x H x W
(14)
Σ(G,A) - -1)-- - - -1,2
J.j vor
K XxHxW
where I, and 7, denote the mean of 1ú and 1', respectively, Eq 14 indicates that the built semantic- level relations supposed to help pixel recognition from the perspective of the mean and variance of the loss variation. To utilize such modeled relationship, we maintain two relation matrices My and My during training and update them as
mennn
M Mrmean. [i.]] Mmmean X poj
mean + mmman) Mrmean.(tt.h
(15)
Mr [i.j] = Myar X Tear + - Mear) x MMuer.((t.jl
where Mmean and Maar are the momentum for two matrices and they are both empirically set as 0.1. By re-conducting Eq (6- with M, and Mr..r as the relation matrix, we can obtain Ra,mean
and Ra,var, respectively. Finally, we combine them obtain Ra in Eq 9 Ru Ra,mean Rayvor-
3.3 Discussion IDRNet
The procedure of deletion diagnostics looks a lot like Dropout 61 while they are totally different mechanisms. Dropout is a mechanism to prevent deep neural networks from overfitting, while our approach introduces deletion diagnostics to help model pixel relations. In addition, work also relates closely to some recent segmentation approaches, which extract semantic-level representations to augment pixel representations, e,g., OCRNet [79] and ISNet [40]. Whereas, these methods only use feature similarities to make connections between pixels, which distinct from our paradigm as is intervention-driee and without masses of predetermined priors.
Our paradigm simplifies pixel relations modeling as object relations modeling. Benefiting from this simplification, deletion diagnostics enables the network to directly utilize its objective function to examine whether one class can help recognize another class j. The key idea behind this mechanism is that loss value of class should increase after deleting class if class can facilitate the prediction

---

Table 2: Performance improvements various benchmark datasets with different segmentation frameworks after leveraging our intervention-driven paradigm to model pixel relations.
Method
Backhone
Stiide
ADE20K Urein /ul) COCO-Stuff (rrinn /tes))
Cityscapes (iratis / vul)
LIP (irain / vuf)
FCN [S]]
ResNet-50
8x
31.76
75.10
48.03
FCN-IDR Net (ours)
ResNet-50
8x
43.61 (66.65)
38.64 (16.88)
79.91 (+4.75)
$1.24 (+2.61)
FSPNet [33]
3esNet.00
8x
42.64
37,40
79.05
31,94
IIPNNr+IDRNNr (OMNN)
Re
8x
44.02 (+ 1,38)
39.13 (+7.7)
79.88 (=0.83)
53.29(+1.38))
DeeplahV33 B
50
8x
43.19
37.63
79.62
52.15
DeeplabV-IDDNatt.oomm (ours)
ResNet-50
8x
(+1.56)
39.31 (+1.6)
80.69 (+107)
53.87 (+1.52)
Net 70
RGaNer30
8x
43.02
37.65
79:08
$2.95
UPeNNet-IRReetouu (owrs)
ResNet-50
8x
44.84 (+ (+1.82
39.35 (+1.70)
80.81 (+1.73)
54.00 (+1.05)
Table 3: Ablation study on the design of IDRNet. IE-Orthogonal: feature enhancement with orthogonal matrix.
IE-DL: feature enhancement with dataset-level representations. BD: balanced deletion in deletion diagnostics. RD: random deletion in deletion diagnosties.
IF:Onthooona
IF.D.
M.
M.
BD
RD
ADE20K (mriin / vil)
PASCAI (fradn vol)
COCO-Stuff (reain test)
x
x
36.96
45.48
31,76
x
x
37.79
x
/
/
/
/
x
43,61
52.97
38.64
/
/
42.99
$2,80
38.86
x
38.06
x
/
x
/
x
43.13
52,66
37,89
7
8
7
42.90
52.92
37.59
of class j. and the increased value can be adopted measure the importance of class i to j. Therefore, there is no potential predetermined priors bias in the process of deletion diagnostics. As a comparison, the geometric transformations of multi-scale-driven context schemes are manually set, which tends to aggregate some ineffective information and lacks of generalization. Similarity-drivee context schemes are assumed to aggregate semantically similar pixel representations, which results in ignoring other dissimilar but effective pixel representations for building co-occurrent patterns, e.g., sky pixels for airplane pixels. In a nutshell, our intervention-drive paradigm is more effective compared to the existing similarity- multi-scale-drive context modules.
4 Experiments
4.1 Experimental Setup
Benchmark Datasets. Our approach is validated on five popular semantic segmentation benchmark datasets, including ADE20K 187), COCO- Stuff [4) Cityscapes [17. LIP [29] and PASCAL-Context
222]. In detail, ADE20K is one of the most well-known datasets for scene parsing, which contains 150 stuff/object category labels. There are 20K/2K/3K images for training, validation and test set, respectively in the dataset. COCO for 91 thing classes and 91 stuff classes. It consists of 9K/IK images in the training and test sets. Cityscapes has 5K high-resolution annotated urban scene images, with 2,975/500/1 1,522 images for rraining/valiiatioonnessii and rider. LIP mainly focuses on single human parsing and contains 50K images with 19 semantic human part classes and background class. Its training. validation and test sets separately involve 30K/10K/100 OO/OOK images. PASCAL-Cone -onteett involves 59 foreground classes and a background class for scene parsing. The dataset is divided into 4,998 and 5,105 images for training and validation.
is a challenging dataset which provides rich annotations
It covers 19 challenging categories, such as traffic sign, vegetation
Implementation Details. Our algorithm is implemented in Py Torch [54] and SSSegmentation [8] For backbone networks (i.e., encoders), they are all pretrained on ImageNet dataset [59]. Remaining layers (i.e., decoders) are initialized by the default initialization strategy in py Torch. As for the data augmentation, some common data augmentation techniques are utilized, including random horizontal flipping, random cropping, color jitter and random scaling (from 0.5 to 2). The learning rate is scheduled by polynomial annealing policy with factor (1
éter 0.99
total iter
Specific to ADE20K, we set learning rate, crop size, batch size and training epochs as 0.01, 512 x 512, 16 and 130, respectively. Specific COCO- learning rate, crop size, batch size and training epochs are set as 0.001, 512 x 512, 16 and 110, respectively. As for LIP, we set learning rate, crop size, batch size and training epochs as 0.01, 473 x 473, 32 and 150, respectively. As for Cityscapes, learning rate, crop size, batch size and training epochs are set as 0.01, 512 x 1024, 8 and 220, respectively. Specific to PASCAI-Context,, we set learning rate, crop size, batch size and training epochs 0.004, 480 480, 16 and 260, respectively.
7

---

Image
PCN
OCRNet
MCIBI
NoR
Figure 3: Qualitative comparison with FCN, OCRNet, ISNet and MCIBI+ on Cityscapes dataset.
Update strategy M,
Backbone
mloU ADE20K
FCN
ResNet-50
36.96
FCN + BP-driven M,
ResNet-50
40.35
FCN DD-driven M,
ResNet-50
43.61
Table Performance comparison between utilizing
Figure 4: Compare category sampling frequeney balanced deletion (BD) and random deletion (RD) on using our deletion diagnostics (i.e., DD-driven M,) Cityscapes (left) and LIP (right) benchmark dataset. to update the relation matrix M,,
back propagation algorithm (i.e., BP-driven M, and
Evaluation Metric. Following conventions 7 for evaluation.
51, mean intersection- -ivriiinnooe---iiion (mloU) is utilized
4.2 Ablation Study
Performance Improvements. As seen in Table it is observed that introducing deletion diagnostics to help pixel relationship building can bring consistent performance improvements on different segmentation approaches (i.e., FCN, PSPNet, DeeplabV3 and UPerNet) and benchmark datasets (i.e., ADE20K, COCO-Stuff, Cityscapes and LIP). By way of illustration, our method boosts FCN, PSPNet, D3eplabV3 and UPerNet by 2.61% %, 1.35%, 1.52%, 1.05% mloU on LIP dataset, respectively.
Feature Enhancement. Feature enhancement is designed make pixel representations with different classes more discriminative to help model relationship between semantic-level representations Rug As indicated in Table 3 we can observe that yielding random orthogonal matrix (IE-Orthogonal) and generating dataset- representations (IF-DL) for feature enhancement are both promising strategies boost segmentation performance. Specifically, IE-Orthogonal brings 0.17%. 0.15% and 1.07% mloU gains for ADE20K, PASCAL-Context and COCO-Stuft, respectively. IE-DL achieves 0.79%, 0.32% and 0.85% mIoU improvements for the three datasets, respectively.
Relation Matrix. We also validate the effectiveness of the proposed relation matrix in Table 3 observed that Mrmean and MM... both contribute to the performance improvements. In detail, Mr Mmean makes our segmentor outperform baseline models (first row) by 6. 6.17%, 7.18% and 6. 6.13% on ADE20K, PASCAL-Context and COCO-Stuff, respectively. Mr... boosts baseline models by 5.78%, 7.38% and 6.30% on the three datasets, respectively. The combination of Mr and My... finally yields 43.61 %, 52.97% and 38.64% mloU the three datasets, respectively.
Deletion Diagnostics. In Table we compare our proposed deletion diagnostics procedure (DD- driven M,) with back-propagation updating strategy (BP-driven M,). We can observe that adopting DD-driven M, outperforms using BP-driven M, by 3.26% mloU on ADE20k. The result suggests that our deletion diagnostics strategy is more effective than the back-propagation updating strategy.
Balanced Deletion. Balanced deletion is utilized to help improve the being- -iagnnosed probabilities of uncommon categories in training dataset and thereby, deletion diagnostics can be performed evenly. In 3 we can observe that introducing the balanced deletion strategy can help improve the segmentation performance by 0.71 %, 0.05% and 1.05% for ADE20K, PASCAL-Conter and COCO-Stur Stuff, respectively. Also, we count and compare the sampling frequency of each category when using balanced deletion (BD) and random deletion (RD) in Figure 4 We can observe that the sampling frequency of uncommon categories have increased after utilizing BD as we expected.
Complexity Comparison. We also present analysis complexity of our intervention-drivee context module following the settings of OCR (79] In Table [ it is observed that our IDRNet requires the least Parameters, FLOPS and GPU Memory, while achieving the best mloU on the ADE20k dataset. Moreover, our IDRNet is complementary to other context modules. For example, when incorporated with PPM (termed "PPM+IDRNet" in the table), the Parameters, FLOPS, Time and GPU Memory only increase by 0.58 M, 39.78 G, 11.19 ms and 73.65 M. respectively, which shows that our proposed IDRNet is light and efficient during network inference. Note that few parameters

---

Table 5: Complexity comparison with existing context modules single RTX 3090 Ti GPU. The input feature
map is with size [1 x 2048 128 x 128]. Excepted for mloU column, all numbers are the smaller better.
Context Module
Parameters
FLOPS
Time
GPU Memory
mioU on ADE20K (%)
OCR 79
15.12M
242.48G
16.58ms
617.24M
42.47
ASPP [8
42.21M
674,47G
41,98ms
976.06M
43.19
PPM 83
23.07M
309.45G
21.45ms
960.63M
42.64
UPerNet 770
34.75M
500.76G
36,51ms
1429.18M
43.02
ANN [89
22.42M
369.62G
26.58ms
1445.75M
41.75
CCNer [37
23.92M
397.38G
30.92ms
986.28M
42.48
DNL 75
24.12M
395.25G
51.38ms
2381.04M
43.50
IDRNet (ours)
10.79M
155.890
20.52ms
365.66M
43.61
PPM+IDRNet (ours)
23.65M
349.23G
32.64ms
1034.28M
44.02
Table Compare segmentation performance cross domain.
Urban Scene Parsing (Cityscapes train)
Human Parsing (LIP train)
Method
Dork Zurich val Nighttime Driving
CIHP
FCN [5]]
10.66
17.90
27.20
FCN+IDRNet (ours)
12.55 (+1.89)
21.33 (+3.43)
28.59 (+1,39)
DeeplahV3 [8
10.03
21.91
27.02
DecplabV3+IDRMM (ours)
13.66 (43.63)
23.85 (+1.94)
27.93 (+0.91)
UPerNer [83
11.06
17.67
26.84
UPenNet+IDRNet (ours)
11.42 (+0.36)
21.13 (+3.46)
27.46 (+0.62)
increase in PPM+IDR adopt shared 3 x 3 convolution layer to reduce the feature channels of the backbone outputs.
Qualitative Results. Figure 3 provides qualitative comparison of the proposed approach against FCN [5I], OCRNet (79), ISNet [40] and MCIBI++[4I] on representative examples in various benchmark datasets. It is observed that IDRNet can output better segmentation results.
4.3 Cross Domain Segmentation Performance.
coopprgss the cross-domain segmentation performance. can be observed that IDRNet can also boost cross-domain segmentation performance, which demonstrates the generalization of our intervention-ddiven pixel relation modeling paradigm. For example, DeeDlabV3+IDRNet trained on day-time domain (i.e., Cityscapes) can also bring 3.6% 3.63% and 1.94% mloU gains to Dark Zurich 60 and Nighttime Driving [8]] datasets which belong night-time domain. In addition, after training FCN+IDRNet on single-hun gugle-human parsing scenarios (i.e., LIP), it also boosts the segmentation performance on
parsing scenarios (i.e.. CIHP 28 by 1.39% mloU.
4.4 Comparison with State-of-the-art Methods
Table compares the quantitative results on four challenging benchmark datasets, including ADE20K val, LIP val, prior to this paper. Mask2Former [12] with Swin-Large backbone 41 achieves the oftte--f-the-art art performance hitting 57 57.30% mloU. By incorporating Fosk2Former with IDRNet, it yields 58. 28.2 mloU which surpasses other state-of-the-art segmentation methods. Specific to LIP val, we can observe that among the previous SOTA methods, UPerNe++ with Swin-Large backbone obtains the saate-of-the-at result, i.e., 59.91% mloU. Our method, UPerNet+IDRRNG with Swin- Large backbone, outperforms it by 1.26% and reports the new SOTA result, 61.17% mloU. With regard to PASCAL-Context val, our method, UPerNet=IDRNet with Swin-Large backbone, surpasses all the competitors, i.e., 0.50% over UPerNet-MCIBI++ 5.51% over Segmenter, 8.68% over SETR, 9.71% over OCRNet, to name few. On the subject of COCO-Stute Stuff rest. we can observe that our UPerNet#IDRNet with Swin-Large backbone reaches 50.50% mIoU which is 0.23% higher than previous stateoof-hhe-a method UPerNet+MCIBI+++
val and COCO-Stuff test. As for ADE20K val, it is observed that
4.5 Extend Deletion Diagnosties Object Detection
In this section, apply deletion diagnostics mechanism to the object detection frameworks to further demonstrate the effectiveness and generalization of our method.
Implementation Details. Our experiments are conducted on MS COCO dataset 48] All algorithm implementations are based on PyTorch (54 and MMDetection 6.. and we follow the default settings in MMDetection for model training and testing.

---

Table 7: State-of-the-art results ADE20K, COCO-Stuff, LIP and PASCAL-ContexL dataset.
Medbod
Backbone
Sinde
ADE2KK (mrate /ooin//lll
COCO-Stulf (rrin /ier) PASCAL- ConileCCnnstttt [train/ /val (ruite/vat
FSPNa 883
KeNNa-TO
5x
43.29
47.80
EMANe: 16
ResNec 101
8x
39,90
53.10
DCNNr 800
ResNet-101
8x
45.45
54,72
ACNe: K3
ResNet- 10]
8x
45.90
40.10
54,10
ARCNe 81
ResNet-101
45.38
54,70
OCRNet
ResNet-10
54.80
ISNet
ResNer-101
47.31
UPeNe=-MCiB1
11
ResNet110 10]
47.93
41.84
56.82
56.32
OCKNett
HKNarV2-W-5
4%
43.06
40.50
56.20
26.05
SFTR 50
VrEaaag
16x
50.28
55.85
Segmenter
63
Large
16x
53.63
59,00
UPerNet
10
BEil-Laage
16x
57.00
UPeniat
Swin-l
32x
53.50
UPeNee-MCLB1
Swin
32x
54,52
50.27
64.01
39,91
MaskFormer
13
55.60
Mask2Furmer
Swin Large
SegNeXt- L
MSCAN Large
52x
52. 12.10
47.20
60.30
Sigfoomeer 21
MIT-R5
82X
51.80
46.70
UPeNNet-IDRNN (owng)
Swin-Large
92%
54.67
50.50
64.51
61.17
Forke DDRNet lours)
Swin Large
32x
58.22
29.96
61.65
61.86
Table 8:
Performance improvements two popular object detection frameworks and MS COCO benchmark dataset after leveraging deletion diagnostics to build object relations.
Method
Backbone
Schedule
AP
APs0
AP
AP,
AP
mg
AP;
Faster R-CNN
56
GesNet-SO-FFD
1x
37.4
58.1
40.4
21.2
41.0
48.1
Faster R- I-CNN-IDRNet (ours)
ResNet-50-EPN
1x
(40.8)
59.7
41.3
22.0
42.0
49.2
Mask R-CNN
32
ResNet-80-PPP
38.2
58.8
41.4
21.9
40.9
49.5
Mask R CNN+IDRNNt (ours)
ResNet-50 So-tt-00FFFP FPN
1x
(+1..)))
60.8
43.0
22.6
43.2
52,5
Experimental Results. As illustrated in Table 8 the proposed deletion diagnosties mechanism is integrated into two popular object detection frameworks, i.e., Faster R-CNN [56] and Mask R-CNN
[32. We can observe that intervention-drine relation building for objects brings 0.8% and 1.5% AP improvements for the two detection frameworks, respectively.
Conclusion
This work studies how to utilize deletion diagnostics model pixel relationships. The key contribution consists of two parts: (1) we simplify pixel relation modeling as object relation modeling and (2) introduce deletion diagnostics make the networks build object relationships by perceiving changes in the outputs of the objective functions. Compared to uumilarity/multi-sale----- modeling paradigms, there fewer human priors in context modeling paradigm. Experimental results demonstrate the effectiveness of our proposed paradigm qualitatively and quantitatively.
context
Acknowledgement
The work described in this paper was partially supported by grants from the National Key R&D Pro- gram of China (2022ZD01 18 101),, the National Natural Science Fund (62201483) and the Research Grants Council of the Hong Kong Special Administrative Region, China (T45-401/22-N),
References
H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0m 08254..882.21100. 2021.
[2] Berman, R. Triki, and M. Blaschko. lovász-softmax loss: tractable surrogate for optimization of the intersection-over-uiioo measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4413 4421, 2018.
[3] G. Bertasius, J. Shi, and I.. Torresani. Semantic segmentation with boundary neural fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3602-3610, 2016.
H. Caesar, Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-122 1218, 2018.
[5] Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu. Genet: Non-local networks meet and beyond. In Proceedings of the IEEE/CVF international conference on computer vision workshops. pages 0-0, 2019.
networks
10

---

[6] K. Chen, J. Wang, J. Pang. Y. Cao. Y. Xiong, X. Li. S. Sun, W. Feng, Z. Liu, J. Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint 19tp:/000...1..00000 2019.
[7] L.-C. Chen, G. Papandreou. I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))834844) 2017.
[8] L.-C. Chen, Papandreou. F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 00587, 2017.
[9] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[10] W. Chen, X. Gong, X. Liu, Q. Zhang, Y. Li. and Z. Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv preprint arXiv: 19..1.912..1.... 2019.
[11] X. Chen and A. Gupta. Spatial memory for context reasoning in object detection. In Proceedings of the IEEE international conference on computer vision, pages 4086-4096, 2017.
[12] B. Cheng, Misra, G. Schwing, Kirillov, and Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[13] B. Cheng, A. Schwing. and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advancey in Neural Information Processing Systems, 34:17864- 17875, 2021.
[14] M. J. Choi, A. Torralba, and A. S. Willsky. A tree-based context model for object recognition. IEEE transactions pattern analysis and machine intelligence, 34(2):240 252, 2011.
[15] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen. Twins: Revisiting the design of spatial attention vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366 2021.
[16] R. D. Cook. Detection of influential observation linear regression. Technometrics, 19(1):15-1 --88 1977.
[17] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[18] D. Dai and L. Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3819-3824. IEEE, 2018.
[19] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks, In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[20] H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, and G. Wang. Boundary-aware feature propagation for scene segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6819-6829, 2019.
[21] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 20.00 1929, 2020.
[22] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[23] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence. 32(7): 1645, 2009.
[24] C. Galleguillos and S. Belongie. Context based object categorization: critical survey. Computer vision and image understanding, 11 11(6))712-722, 2010.
[25] C. Galleguillos, A. Rabinovich, and S. Belongie. Object categorization using co-occurrence, location and appearance. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2008.

---

[26] S.-H. Gao. M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr. Res2net: A new multi-scale backbone architecture. IEEE Iransactions on pattern analysis and machine intelligence, 43(2):652-662. 2019.
[27] R. Girshick. Fast r-enn, In Proceedings of the IEEE international conference computer vision, pages 4440-1488, 2015.
[28] K. Gong, X. Liang, Y. Li, Y. Chen, M. Yang, and L. Lin. Instance-level human parsing via part grouping network. Proceedings of the European conference on computer vision (ECCV), pages 770-785, 2018.
[29] K. Gong, X. Liang. D. Zhang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark human parsing. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 932-940, 2017.
[30] M.-H. Guo, C.-Z. Lu, Q. Hou, Liu, M.-M. Cheng, and S.-M. Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08577 2022.
[31] He, Deng, L. Zhou, Wang, and Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7519-7528, 2019.
[32] K. He. G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference compuder vision, pages 2961-2969, 2017.
[33] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[34] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networks for object detection. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 3588-3597, 2018.
[35] X. Hu, K. Yang, I., Fei, and K. Wang. Acnet: Attention based network to exploit complementary features for rghd semantic segmentation. In 2019 IEFE International Conference on Image Processing (ICIP), pages 1440-14400 1444. IEEE, 2019.
[36] L. Huang, Y. Yuan, J. Guo, C. Zhang. X. Chen, and J. Wang. Interlaced sparse self-attention for semantic segmentation, arXiv preprint arXiv: 1907.12277.. 2019.
[37] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 603-612, 2019.
[38] 7. Jin. Sssegmenation: open source supervised semantic segmentation toolbox based pytorch. arXiv preprint arXiv:2305. 17091, 2023.
[39] Z. Jin, T. Gong, Yu, Q Chu, Wang, C. Wang, and J. Shao. Mining contextual information beyond image for semantic segmentation. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7231-7241, 2021.
[40] Z. Jin, B. Liu, Q. Chu, and N. Yu. Isnet: Integrate image-level and semantic-level context for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7189-7198, 2021.
[41] 7. Jin, D. Yu, Z. Yuan, and Yu. Mcibi++: Soft mining contextual information beyond image for semantic segmentation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[42] T.WW. Ke, J.J. Hwang, Z. Liu, and S. X. Yu. Adaptive affinity fields for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 587-602, 2018.
[43] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks, arXiv preprint aivi:: A.xx::6660 (10.666990000 2016.
[44] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553)):336 444, 2015.
[45] Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski. A theoretical framework for back-propagale In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21-28, 1988.
[46] X. Li, Z. Zhong, J. Wu, Y. Yang. Z. Lin, and H. Liu. Expectation Expeetation-mmxximzzaiie maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9167 99769 2019.
12

---

[47] Y. L. Song, Y. Chen, Z. Li, X. Zhang, X. Wang, and J. Sun. Learning dynamic routing for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 3553-8562, 2020.
[48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, Ramanan, P. Dollár, and C. L.. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 133th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740 740-755. Springer, 2014.
[49] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, and L. Fei-Fei. Auto-deeplab: Hicrarchical neural architecture search for semantic image segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 82-92, 2019.
[50] Z. Liu, Lin, Y. Cao, H. Hu, Y. Wei, Zhang, S. Lin, and Guo, Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE international conference on computer vision, pages 10012-10022, 2021.
[51] J. Long, E. Shelhamer, and T. Darrell, Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 3431-3440, 2015.
[52] F. Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math math-ph/00000500 2006.
[53] R. Mottaghi, X. Chen, X. Liu. N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 891-898, 2014.
[54] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. De Vito, Z. Lin, A. Desmaison, L. Antiga, and
A. Lerer. Automatic differentiation pytorch. 2017.
[55] 7.. Peng, I.. Dong, Bao, Q. Ye, and Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06338 2022.
[56] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[57] T. Ruan, T. Liu, Z. Huang, Y. Wei, S. Wei, and Y. Zhao. Devil in the details: Towards accurate single and multiple human parsing. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4814-4821, 2019.
[58] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by pooopppppggting errors. nature, 323(6088):533-555 -366, 1986.
[59] O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, (1:33)2221-2222 2015.
[60] C. Sakaridis, D. Dai, and L. V. Gool. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7374-777 7784, 2019.
[61] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
[62] R. Strudel, R. Garcia, Laptev, and C. Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272, 2021.
[63] A. Torralha, K. P. Murphy, W. T. Freeman, and M. A. Rubin. Context-hased vision system for place and object recognition. In Computer Vision, IEF. International Conference on, volume 2, pages 273-273. IEFE Computer Society, 2003.
[64] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ł. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[65] C. Wang, Y. Zhang. M. Cui, P. Ren, Y. Yang. X. Xie, X.SS Hua, H. Bao, and W. Xu. Active boundary loss for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2397-2202222 2475, 2022.
13

---

[66] J. Wang, K. Sun. T. Cheng. B. Jiang, C. Deng, Y. Zhao, D. Liu. Y. Mu, M. Tan, X. Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3349-33
2020.
[67] W. Wang. J. Dai, Z. Chen, Z. Huang. Z. Li, X. Zhu, X. Hu, T. I.. Lu, H. Li, et al. Interniaage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint urXiv:221..0577. 2022.
[68] W. Wang, T. Zhou, F. Y. J. Dai, E. Konukoglu, and L. Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7303-7313, 2021.
[69] X. Wang. R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[70] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for seene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-4344 2018.
[71] E. Xie, W. Wang. Z. Yu. A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 14:17077-00090, 2021.
[72] S. Xie, R. Girshick, P. Dollár, 7. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 1492-1500, 2017.
[73] H. Xu, C. Jiang, X. Liang, and Z. Li. Spatial-aware graph relation network for large-scale object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9298- 9307, 2019.
(74] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang. Denscaspp for semantic segmentation in street seenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3684-3692, 2018
[75] M. Yin, Z. Yao, Y. Cao, X. Li, Z. Zhang, S. Lin, and H. Hu, Disentangled non-local neural networks. In Computer Vision ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 191-201 207. Springer, 2020.
[76] C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, and N. Sang. Context prior for scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12416-12425, 2020.
[77] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 1.1.011511.0712..22 2015.
[78] W. Yu, M. Luo, Zhou, Si, Y. Zhou, X. Wang. J. Feng, and S. Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819 19-1999,,, 2022.
179] Y. Yuan, Chen, and J. Wang. Object-contextual representations for semantic segmentation. In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 173-190. Springer, 2020.
[80] Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, and Wang. Ocnet: Object context for semantic segmentation. International Journal of Computer Vision, 129(8))2375-2398 2021.
[81] Y. Yuan, J. Xie, X. Chen, and J. Wang. Segtix: Model-agnostic boundary refinement for segmentation. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XII 16, pages 489-506. Springer, 2020.
[82] W. Zhang. J. Pang. K. Chen, and C. C. Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34:10326-10338, 2021.
[83] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2881-2890. 2017.
[84] S. Zhao, Y. Wang, Z. Yang, and D. Cai. Region mutual information loss for semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[85] X. Zhao, R. Vemulapalli, P. A. Manstield, B. Gong, B. Green, L. Shapira, and Y. Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16623-16333 2021.
14

---

[86] S. Zheng, J. Lu, H. Zhao. X. Zhu, Z. Luo, Y. Wang. Y. Fu, J. Feng. T. Xiang, P. H. Torr, et al. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881-6890. 2021.
[87] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Tomalba. Scene parsing through ade20k dataset. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 633-64 641, 2017.
[88] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: More deformable, better results. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 9308 9316, 2019.
[89] Z. Zhu, M. Xu, S. Bai, T. Huang. and X. Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 593-602, 2019.
15


Reviews:
[{"review_id": "tLyeKFZj4V", "reviewer": "Reviewer_PhSv", "summary": "To further enhance the segmentation performance, the authors utilize deletion diagnostics to model pixel relationships. Specifically, to address the computational cost of pixel relation modelling, they simplify it as object relation modelling. Then, they introduce deletion diagnostics to facilitate the network in building object relationships. The experimental results qualitatively and quantitatively demonstrate the effectiveness of their proposed paradigm.", "questions": "1. There is confusion in the writing. For example, in the paper, IDRNet refers to an additional module that enhances contextual information aggregation. However, in Table 1, IDRNet is described as a simple FCN segmentation framework equipped with IDRNet. This inconsistency can cause confusion for readers.\n2. Without using the Intervention-Driven context module, would combining the Multi-Scale-Driven and Similarity-Driven modules yield better results?\n3. What is the extent of the additional time-consuming and memory-consuming aspects associated with the intervention-Driven context module?\n4. In equation (8), the meaning of $R_a$ is not explained.\"\n5. In equation (9), the meaning of $\\oplus$ is not explained. Is it element-wise addition or channel-wise merging?", "limitations": "There is no limitation claimed in the paper. For example, the cost of the Intervention-Driven context module, particularly in terms of training time and storage, remains unknown.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "To improve the contextual information aggregation，the authors propose a novel Intervention-Driven Relation Network (IDRNet), which leverages a deletion diagnostics procedure to guide the modelling of contextual relations among different pixels. This method brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets.", "weaknesses": "1. The paper lacks the presentation of additional time-consuming and memory-consuming aspects associated with the Intervention-Driven context module.\n2. The paper lacks interpretability regarding how the Intervention-Driven context module improves contextual information aggregation.\n3. Some minor issues: confusion in the writing, some symbols used in equations or throughout the paper are not adequately explained. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the comments.  We hope that our responses could thoroughly address your concerns. \n\n> **Q1: Confusion of IDRNet.**\n\nWe are sorry for the confused representation of IDRNet. In the revision, we will utilize ***IDRNet*** to indicate the whole segmentation framework and ***IDR*** to represent the intervention-driven context module to clear up the confusion. For example, “a simple FCN segmentation framework equipped with IDRNet (termed IDRNet)” will be revised as “a simple FCN segmentation framework equipped with intervention-driven context module (termed IDRNet)”.\n\n> **Q2: Lacks presentation of time-consuming and memory-consuming aspects of intervention-driven context module.**\n\nThanks for your comment. Following the inference settings in OCRNet $^{[1]}$, we perform complexity comparison with other context modules on a single RTX 3090 Ti GPU and the results during network inference are reported in the below table. We will also add this table in the revision.\n\n| Context Module | Params | FLOPS |  Time    | Memory  |  mIoU on ADE20k (%)  |\n| :---:             |  :---:   |  :---:    |  :---:   | :---:    |  :---: |\n| OCR       |  15.12M  |  242.48G  |  16.58ms | 617.24M  |  42.47 |\n| ASPP      |  42.21M  |  674.47G  |  41.98ms | 976.06M  |  43.19 |\n| PPM       |  23.07M  |  309.45G  |  21.45ms | 960.63M  |  42.64 |\n| UperNet   |  34.75M  |  500.76G  |  36.51ms | 1429.18M |  43.02 |\n| ANN       |  22.42M  |  369.62G  |  26.58ms | 1445.75M |  41.75 |\n| CCNet     |  23.92M  |  397.38G  |  30.92ms | 986.28M  |  42.48 |\n| DNL       |  24.12M  |  395.25G  |  51.38ms | 2381.04M |  43.50 |\n| IDR (*ours*)      |  10.79M  |  155.89G  |  20.52ms | 365.66M  |  43.61 |\n| PPM+IDR (*ours*)  |  23.65M  |  349.23G  |  32.64ms | 1034.28M |  44.02 |\n\nIt is observed that IDR module requires the least Params, FLOPS and GPU Memory,  while achieving the best mIoU on the ADE20k dataset compared with other context modules. Moreover, our IDR module is complementary to other context modules. For example, when incorporated with PPM (***PPM+IDR***), the Params, FLOPS, Time and Memory only increase by $0.58$ M, $39.78$ G, $11.19$ ms and $73.65$ M, respectively, which shows that our proposed IDR is light and efficient during network inference. Note that few parameters increase in ***PPM+IDR*** as we adopt a shared $3 \\times 3$ convolution to reduce the feature channels of the backbone outputs.\n\n[1] Yuan et al. \"Object-contextual representations for semantic segmentation.\" ECCV 2020.  \n\n> **Q3: Results of combining the multi-scale-driven and similarity-driven module.**\n\nFollowing your suggestion, we add the results of different combinations of multi-scale-driven, similarity-driven and intervention-driven context modules.\n\n| Context Modules                                 |    mIoU on ADE20k  |\n| :---:                                                     |  :---:  |\n| multi-scale-driven (UperNet)                              |  43.02  |\n| similarity-driven (OCRNet)                                |  42.47  |\n| intervention-driven (ours)                                |  43.61  |\n| multi-scale-driven (UperNet) + similarity-driven (OCRNet) |  41.57  |\n| multi-scale-driven (UperNet) + intervention-driven (ours) |  44.84  |\n| similarity-driven (OCRNet) + intervention-driven (ours)   |  44.50  |\n\nIt is observed that intervention-driven context module is complementary to multi-scale-driven and similarity-driven context modules, *i.e.*, brings 1.82% and 2.03% mIoU improvements to multi-scale-driven and similarity-driven module, respectively. On the contrary, the performance of multi-scale-driven (UperNet) + similarity-driven (OCRNet) is a little poor compared with their original performance (41.57% *v.s.* 43.02% and 42.47%). The reason may be that such a combination is difficult to train or may easily lead to overfitting.\n\n> **Q4: How the Intervention-Driven context module improves contextual information aggregation.**\n\nThe motivation of intervention-driven context module is to enable the network to _**directly utilize its objective function to examine whether one class $i$ can help recognize another class $j$**_. Specifically, if the removed contextual pixels of class $i$ is helpful for recognizing the pixels of class $j$, the loss values of class $j$ should increase and such increased value can be used to reflect the relation between class $i$ and $j$ in $\\mathcal{M}$. Consequently, class $j$ can aggregate more effective contextual information from class $i$ according to the estimated relation value to facilitate network optimization and improve segmentation accuracy. \n\nThe key difference between our module and previous context modules is that we involve ***less potentially predetermined priors bias*** during pixel context modeling. Particularly, the geometric transformations of multi-scale-driven context schemes are manually set, which tends to aggregate some ineffective information and lacks of generalization. Similarity-driven context schemes are assumed to aggregate semantically similar pixel representations, which results in ignoring other dissimilar but effective pixel representations for building co-occurrent patterns, *e.g.*, sky pixels for airplane pixels. \n\n> **Q5: Some minor issues.**\n\n- $R_a \\in \\mathbb{R}^{Z \\times \\frac{H}{8} \\times \\frac{W}{8}}$ is the feature representations by re-arranging $R_{esl}$. Specifically, we initialize $R_a$ as a zero matrix and fill it with the representations in $R_{esl}$ according to the pseudo labels in each pixel position. For example, if the labels in the input image $I$ are {$0, 1, 5, 7$}, $R_{esl}$ will be {$R_{esl}^{cls=0}, R_{esl}^{cls=1}, R_{esl}^{cls=5}, R_{esl}^{cls=7}$} in Eq (6). And if class id of $R_{a, [\\*, 0, 0]}$ is 5, we will set $R_{a, [\\*, 0, 0]} = R_{esl}^{cls=5}$ in Eq (8).\n\n- $\\oplus$ indicates a concatenation operation as explained after Eq (3). In the revision, we will further explain it after Eq (9) to easy understanding."}, {"review_id": "wH1daLQxvA", "reviewer": "Reviewer_q3X9", "summary": "This paper presents a context relation modeling method, which can be incorporated with existing semantic segmentation model. Instead of modeling pixel-level relation, it models class-level semantic relation by weighted averaging features at each location and deletion diagnostics. The proposed module is flexible and tested with different network backbone and segmentation architectures, which observes consistent improvements on several widely used datasets. Furthermore, the proposed context relation module can be applied in object detection, that stronger results on MS COCO are shown based on Faster-RCNN.", "questions": "1. Actually, you do not have to use deletion diagnostics to update M.  What is the motivation of deletion diagnostics? What kind of issues does the proposed deletion aim to address?\n", "limitations": "See weakness section.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "+ This paper presents a context relation module, which can be combined with previous semantic segmentation models. It is used to enhance the learned features and perform features interaction with global co-cocurrence. Besides, in order to overcome the heavy computation issue, this work proposes to use semantic relation to replace pixel relation.\n\n+ This paper conduct extensive experimental results, including 4 popular datasets. Besides, this work is combined with many previous segmentation models. For example, even though PSPNet can aggregate context information, the semantic relation module can further improve its performance.\n\n+ Comparison with previous state-of-the-art demonstrates the effectiveness of the proposed module.\n\n+ This work is also compatible with object detection framework. It should be also feasible to apply the proposed module in semantic instance segmentation.", "weaknesses": "- Too many symbols in section 3. For some details, I cannot follow.\nIn Eq 3, what is the output dimension for R_{sl} ?\nIn Eq 6, why is R_{esl} a Ne xZ matrix, as M_r is KxK ?\nIn Eq 8, what are the values in R_a if Y_c != k ?\nIn Eq 9, what does the \\bigoplus operation represent ?\n\n- Figure 5 is hard to see the graph connection, for example, the rider class. For sky, it is connected to truck, building, road, which does not seem to make sense.\n\n- In Table 5, why not show the performance for Mask2Former+IDRNet on LIP/COCO-Stuff/PASCAL-Context datasets?\n\n- Suggest to list all the comparing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your insightful reviews and positive comments on our methodology design, extensive evaluation, and framework generalizability. Our responses are shown below.\n\n\n> **Q1: Explanation of symbols in Section 3.**\n\nWe are glad to explain the meaning of these symbols and will add these explanations and also list all the comparing methods following your suggestation in the revised version.\n\n- The output dimension for $R_{sl}$ in Eq (3) is $\\mathbb{R}^{N_e \\times 2Z}$, where $N_e$ is the number of classes existed in the input image and $Z$ is the number of channels of $R_p$ (or $R_{ie}$). When introducing the ***Feature Enhancement*** module, we use a $1 \\times 1$ convolution layer to make $R_{sl}$ be with the shape of $N_e \\times Z$ after Eq (3) to avoid introducing too many extra parameters in IDRNet.\n\n- In Eq (6), we first leverage $\\mathcal{T}$ (see Eq (7) for more details) to transform $\\mathcal{M}_r \\in \\mathbb{R}^{K \\times K} $ into $\\mathcal{\\hat{M}}_r \\in \\mathbb{R}^{N_e \\times N_e}$ for matching the shape of  $R\\_{sl}$ and thereby, $\\hat{\\mathcal{M}}\\_{r}$ only contains the relation values of the classes existed in the current input image. As a result, the shape of $R\\_{esl}$ is $N\\_{e} \\times Z$.\n\n- $R_a \\in \\mathbb{R}^{Z \\times \\frac{H}{8} \\times \\frac{W}{8}}$ is the feature representations by re-arranging $R_{esl}$. Specifically, we initialize $R_a$ as a zero matrix and fill it with the representations in $R_{esl}$ according to the pseudo labels in each pixel position. For example, if the labels in the input image $I$ are {$0, 1, 5, 7$}, $R_{esl}$ will be {$\\{R_{esl}^{cls=0}, R_{esl}^{cls=1}, R_{esl}^{cls=5}, R_{esl}^{cls=7}\\}$} in Eq (6). And if class id of $R_{a, [\\*, 0, 0]}$ is 5, we will set $R_{a, [\\*, 0, 0]} = R_{esl}^{cls=5}$ in Eq (8). Thus, $Y_{c, [i,j]}$ will always be equal to one of the class ids existed in $R_{esl}$ and it is impossible for the case of $Y_{c, [i,j]} \\neq k$.\n\n- $\\oplus$ indicates a concatenation operation as explained after Eq (3). In the revised version, we will further explain it after Eq (9) to easy understanding.\n\n> **Q2: Figure 5 is hard to see the graph connection.**\n\nIn Figure 5, we show the relationships graph without adding the corresponding connection weights, which may cause some confusion. Actually, \"sky is connected to truck, building and road\" is reasonable in the Cityscape datasets since there are some images containing both \"truck, building and road\" and \"sky\". However, the connection weight of \"truck\" and \"sky\" in $\\mathcal{M}$ is ***much smaller*** than the one of \"truck\" and \"road\". Therefore, the pixel representations of \"truck\" will aggregate much more information from the \"road\" rather than \"sky\". In the revised version, we will mark relation weights between the classes and modify the graph to show only the top k connections to better visualize the relationships between the classes.\n\n> **Q3: Missing results for Mask2Former+IDRNet on LIP/COCO-Stuff/Pascal Context.**\n\nIn Table 5, as the performance of UPerNet+IDRNet has already achieved the state-of-the-art performance on LIP/COCO-Stuff/Pascal Context, we did not show the performance of Mask2Former+IDRNet on these datasets. Following your suggestion, we have conducted the experiments for Mask2Former+IDR on LIP, COCO-Stuff and Pascal Context datasets and the results are shown below, which will be added in the revised version.\n\n| Method             |   Backbone   | ADE20k  | LIP    | COCO-Stuff  | Pascal Context |\n| :---:              |   :---:      | :---:   | :---:  | :---:       | :---:          |\n| Mask2Former        |   Swin-Large | 57.30   | 60.37  | 48.08       | 60.67          |\n| Mask2Former+IDR |   Swin-Large | 58.22   | 61.86  | 49.96       | 61.65          |\n\n> **Q4: Motivation and benefit of deletion diagnostics.**\n\nThanks for your comment. First, there are indeed different ways to update $\\mathcal{M}$, such as back-propagation. In the below table, we compare our proposed deletion diagnostics procedure with back-propagation updating strategy. We can observe that deletion diagnostics strategy outperforms back-propagation by 3.26% mIoU on ADE20k. The result suggests that our deletion diagnostics strategy is more effective than other strategies for updating $\\mathcal{M}$.\n\n| Method                                         |  mIoU on ADE20k  |\n| :---:                                          |  :---:           |\n| FCN baseline                                   |  36.96           |\n| FCN + IDR (back-propagation)                   |  40.35           |\n| FCN + IDR (deletion diagnostics)               |  43.61           |\n\nSecond, the motivation of utilizing deletion diagnostics is to enable the network to directly utilize its objective function to examine whether one class $i$ can help recognize another class $j$. The key difference between our method and previous context modules is that we ***involve less potentially predetermined priors bias*** in the process of deletion diagnostics. Specifically, the geometric transformations of multi-scale-driven context schemes are manually set, which tends to aggregate some ineffective information and lacks of generalization.  Moreover, similarity-driven context schemes are assumed to aggregate semantically similar pixel representations, which results in ignoring other dissimilar but effective pixel representations for building co-occurrent patterns, e.g., sky pixels for airplane pixels. \n\nIn a nutshell, our intervention-driven paradigm is ***more effective and have better generalization abilities*** compared to the existing similarity- / multi-scale-driven context modules. In the revised version, we will further explain this point to more clearly motivate the proposed deletion diagnostics procedure."}, {"review_id": "ZUyN8wS72x", "reviewer": "Reviewer_dJpB", "summary": "This work presents IDRNet, a context aggregation scheme for semantic segmentation networks which is driven by intervention in the object query stage. The work is a follow up on OCR (object-contextual representations), but adds to the previous work by an intervention-based strategy to update, augment, or delete tokens. The updates to these tokens are based on semantic relations.", "questions": "1. Please could describe how this method is different from other works performing token pruning\n2. Please could you provide some analysis on complexity/runtime related to other context aggregation schemes such as OCR\n3. Please could you provide some limitations of the approach", "limitations": "The authors have not addressed limitations of their method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Presentation:\nThe paper is clear in its presentation of the concept, and the result. Tables and figures are self-explanatory and it is easy to follow the though process of the authors. The reviewer appreciates the effort spent in explaining the concept of this work.\n\nResults: \nAs a scheme which is built to add to pixel representations, the results are quite impressive on five different datasets. The authors have also provided ablation studies detailing the effect of each separate block in the proposed algorithm.\n\nVisulalizations:\nThe reviewer also appreciates the detail the authors have provided on viewing semantic relationships, which was the major motivation of their approach. Hence, it is safe to say that the motivation of the work and results go hand in hand.", "weaknesses": "Originality:\nThe concept of updating (deleting, augmenting, enhancing, \"diagnosing\") embeddings based on object relations might seem novel when you think of them in a convolution-based setting. However, I am afraid this concept is not original in transformer networks. Works such as Mask2Former do present queries which describe objects and the network learns (based on bipartite matching) to associate queries with the concept of object. There are also several other works [1], [2], [3] which prune, refine or update these tokens based on semantic relationships. \n\nComplexity:\nAs the method adds additional compute to the network, the authors mention how much compute is added and also, more importantly, is Baseline+IDRNet better than simply scaling up the baseline to match this compute.\n\n[1] Revisiting Token Pruning for Object Detection and Instance Segmentation\n[2] Sparse Tokens for Dense Prediction - The Medical Image Segmentation Case\n[3] DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive\nSegmentation Transformer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the comments. We hope that our responses could thoroughly address your concerns. \n\n> **Q1: How this method is different from other works performing token pruning.**\n\nWe are glad to clarify the differences between our work and those token pruning works. \n\n- The objectives are different. The mentioned works aim to identify which are the important representations / tokens for network pruning, while our method aims to augment the image representations via utilizing and learning the relationships among different semantic representations. \n\n- The definition of representations / tokens are different. The tokens in [1] (released in June 2023) and [2] are flattened 2D image patches, which may ***contain the information of multiple, singe, or no objects***. [3] is an interactive segmentation algorithm and the queries are dynamically generated using image features and user clicks. The object queries in [4] are a set of learnable embeddings. As a comparison, the semantic-level representations in IDRNet are yielded by gathering the pixel representations with the same label and each semantic-level representation ***only contains the information of the corresponding category***.\n\n- The principles to build the relationships between representations / tokens are different.  [1]-[4] leverage the transformer and back-propagation to learn the ***statistical correlation*** between tokens / queries, while our IDRNet introduces the ***intervention-based*** deletion diagnostics procedure to update the relation matrix for the semantic-level representations. We make attempts to only adopt back-propagation to update the relation matrix in IDRNet, while the the mIoU on ADE20k drops from 43.61% to 40.35%, verifying the effectiveness of our intervention-based method.\n\n- The approach to learn the relationship are different. [1] utilizes a gating module (a 2-layer perceptron + Gumbel Softmax) to facilitate acceleration of transformer encoder and the learnable parameters in the gating module are supervised by a dynamic pruning ratio loss. Differently, the deletion diagnostics in IDRNet is to perform pixel relationship modeling and the relation matrix is updated by the changes of the objective function value, *e.g.*, the cross entropy loss.\n\n- Our intervention-driven strategy is orthogonal to the segmentation backbone design and ***can also be intergrated into transformer-based backbones***. Particularly, we integrate IDR with Mask2Former [4]. We can observe that IDR brings 0.92%, 1.49%, 1.88% and 0.98% mIoU improvements on different benchmark datasets, suggesting that IDR is complementary to Mask2Former.\n\n| Method             |   Backbone   | ADE20k  | LIP    | COCO-Stuff  | Pascal Context |\n| :---:              |   :---:      | :---:   | :---:  | :---:       | :---:          |\n| Mask2Former        |   Swin-Large | 57.30   | 60.37  | 48.08       | 60.67          |\n| Mask2Former+IDR |   Swin-Large | 58.22   | 61.86  | 49.96       | 61.65          |\n\nIn summary, our method is ***inherently different*** from [1]-[4].\n\n[1] Liu et al. Revisiting Token Pruning for Object Detection and Instance Segmentation. arXiv 2023.  \n[2] Zhou et al. Sparse Tokens for Dense Prediction-The Medical Image Segmentation Case. 2022.  \n[3] Rana et al. DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer. arXiv 2023.  \n[4] Cheng et al. Masked-attention mask transformer for universal image segmentation. CVPR 2022.\n\n> **Q2: Provide some analysis on complexity/runtime related to other context aggregation schemes such as OCR.**\n\nWe present analysis on complexity/runtime of our intervention-driven context module following the settings of OCR. Specifically, we compare the Params, FLOPS, Time and Memory of our IDRNet with previous context modules in the below table.\n\n| Context Module | Params | FLOPS |  Time    | Memory  |  mIoU on ADE20k (%)  |\n| :---:             |  :---:   |  :---:    |  :---:   | :---:    |  :---: |\n| OCR       |  15.12M  |  242.48G  |  16.58ms | 617.24M  |  42.47 |\n| ASPP      |  42.21M  |  674.47G  |  41.98ms | 976.06M  |  43.19 |\n| PPM       |  23.07M  |  309.45G  |  21.45ms | 960.63M  |  42.64 |\n| UperNet   |  34.75M  |  500.76G  |  36.51ms | 1429.18M |  43.02 |\n| ANN       |  22.42M  |  369.62G  |  26.58ms | 1445.75M |  41.75 |\n| CCNet     |  23.92M  |  397.38G  |  30.92ms | 986.28M  |  42.48 |\n| DNL       |  24.12M  |  395.25G  |  51.38ms | 2381.04M |  43.50 |\n| IDR (*ours*)      |  10.79M  |  155.89G  |  20.52ms | 365.66M  |  43.61 |\n| PPM+IDR (*ours*)  |  23.65M  |  349.23G  |  32.64ms | 1034.28M |  44.02 |\n\nIt is observed that IDR module requires the least Params, FLOPS and GPU Memory,  while achieving the best mIoU on the ADE20k dataset. Moreover, our IDR module is complementary to other context modules. For example, when incorporated with PPM (***PPM+IDR***), the Params, FLOPS, Time and Memory only increase by 0.58 M, 39.78 G, 11.19 ms and 73.65 M, respectively, which shows that our proposed IDR is light and efficient during network inference. Note that few parameters increase in ***PPM+IDR*** as we adopt a shared $3 \\times 3$ convolution to reduce the feature channels of the backbone outputs.\n\n> **Q3: Provide some limitations of the approach.**\n\nWe briefly discuss some limitation and future works of our approach. \n\n- Some modules in IDRNet needs to perform forward function twice to update $\\mathcal{M}$ during training, Thus, IDRNet would require more training cost compared to the previous context modules.\n- Due to the limited resource, we did not explore the effectiveness of IDRNet on the large-scale vision foundation models like InternImage $^{[1]}$.\n- In the future, we will investigate the effectiveness of our IDRNet on the domain-specific problem, like medical image segmentation.\n\n[1] Wang et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions, CVPR 2023."}, {"review_id": "zmQv8J6Y1a", "reviewer": "Reviewer_3dVM", "summary": "This study focuses on the utilization of deletion diagnostics for modeling pixel relationships. The main contribution of this work can be divided into two parts. Firstly, the simplification of pixel relation modeling as object relation modeling. Secondly, the introduction of deletion diagnostics to enable networks to build object relationships by detecting changes in the outputs of objective functions. Experimental results qualitatively and quantitatively demonstrate the effectiveness of the proposed paradigm.", "questions": "See Weaknesses.", "limitations": "There seems no discussion about limitations in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The use of Deletion Diagnostics to update the relation matrix is a novel approach that considers the problem of category context in semantic segmentation from a different perspective than previous multi-scale driven and similarity-driven methods.\n\n* Thorough ablation experiments were conducted to verify the impact of different designs of IDRNet on segmentation performance.\n\n* The writing is clear and easily understandable for readers.", "weaknesses": "* The experiments of Mask2Former+IDRNet only report results on one dataset, ADE20K. As Mask2Former is a general segmentation framework, it is recommended that the authors also report the performance of the proposed model on the other three datasets for a comprehensive comparison.\n\n* Additional experiments based on Mask2Former could be added to Table 2 to verify the effectiveness of IDRNet on a regular backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your insightful reviews and positive comments on our novel methodology, the extensive evaluation, and clear writting. Our responses are shown below.\n\n\n> **Q1: Missing results for Mask2Former+IDRNet on LIP/COCO-Stuff/Pascal Context.**\n\nFollowing your suggestion, we also conduct the experiments for Mask2Former+IDRNet on LIP, COCO-Stuff and Pascal Context datasets and the results are shown in the below table. We will add these experiments to Table 5 in the revised version.\n\n| Method             |   Backbone   | ADE20k  | LIP    | COCO-Stuff  | Pascal Context |\n| :---:              |   :---:      | :---:   | :---:  | :---:       | :---:          |\n| Mask2Former        |   Swin-Large | 57.30   | 60.37  | 48.08       | 60.67          |\n| Mask2Former+IDR |   Swin-Large | 58.22   | 61.86  | 49.96       | 61.65          |\n\nIt is observed that Mask2Former+IDRNet also outperforms Mask2Former for 1.49%, 1.88% and 0.98% mIoU on LIP, COCO-Stuff and Pascal Context datasets, respectively, which further show the effectiveness of our method.\n\n\n> **Q2: Additional experiments based on Mask2Former with a regular backbone could be added to Table2.**\n\nThanks for your suggestions. Additional experiments based on Mask2Former with a regular ResNet-50 backbone are conducted and reported in the below table. We will also add these experiments to Table 2 in the revised version.\n\n| Method             |   Backbone   | ADE20k  | LIP    | COCO-Stuff  | Cityscapes     |\n| :---:              |   :---:      | :---:   | :---:  | :---:       | :---:          |\n| Mask2Former        |   ResNet-50  | 47.20   | 51.23  | 37.22       | 79.40          |\n| Mask2Former+IDR |   ResNet-50  | 48.49   | 53.22  | 38.45       | 80.79          |\n\nIt is observed that our Mask2Former+IDRNet consistently outperforms Mask2Former by more than 1% mIoU on various benchmark datasets with a regular ResNet-50 backbone."}]

Author rebuttal:

Meta-review: This paper received varying scores, with three reviewers recommending acceptance and one reviewer suggesting rejection. The authors effectively addressed the reviewers' comments during the rebuttal phase. This paper presents a novel approach aimed at enhancing contextual information aggregation in the field of image segmentation.  This paper introduces the deletion diagnostics mechanism to guide the modeling of contextual relationships among various visual pixels. The authors rigorously examine the impact of different contextual design choices through a series of ablation experiments, which the AC believe is inspiring to the segmentation community. The evaluations across five different datasets are notably impressive and solid. Furthermore, the authors effectively addressed the reviewers' comments during the rebuttal phase and the majority of reviewers have expressed a positive opinion of this paper. Considering the overall assessment and the improvements made by the authors, the AC has decided to accept this paper, despite its averaged rating is lower than the acceptance bar.
Decision: Accept (poster)