Exploiting Semantic Relations for Glass Surface Detection
Authors: Jiaying Lin
Yuen Hei Yeung
Rynson W. H. Lau
Venue: NeurIPS 2022 Accept (2022)
Keywords: Glass surface detection
semantic relation learning

Abstract: Glass surfaces are omnipresent in our daily lives and often go unnoticed by the majority of us. While humans are generally able to infer their locations and thus avoid collisions, it can be difficult for current object detection systems to handle them due to the transparent nature of glass surfaces. Previous methods approached the problem by extracting global context information to obtain priors such as object boundaries and reflections. However, their performances cannot be guaranteed when these deterministic features are not available. We observe that humans often reason through the semantic context of the environment, which offers insights into the categories of and proximity between entities that are expected to appear in the surrounding. For example, the odds of co-occurrence of glass windows with walls and curtains are generally higher than that with other objects such as cars and trees, which have relatively less semantic relevance. Based on this observation, we propose a model ('GlassSemNet') that integrates the contextual relationship of the scenes for glass surface detection with two novel modules: (1) Scene Aware Activation (SAA) Module to adaptively filter critical channels with respect to spatial and semantic features, and (2) Context Correlation Attention (CCA) Module to progressively learn the contextual correlations among objects both spatially and semantically. In addition, we propose a large-scale glass surface detection dataset named {\it Glass Surface Detection - Semantics} ('GSD-S'), which contains 4,519 real-world RGB glass surface images from diverse real-world scenes with detailed annotations for both glass surface detection and semantic segmentation. Experimental results show that our model outperforms contemporary works, especially with 42.6\% MAE improvement on our proposed GSD-S dataset. Code, dataset, and models are available at https://jiaying.link/neurips2022-gsds/

Exploiting Semantic Relations for Glass Surface Detection
Jiaying Lin"
Yuen-Hei Yeung" Rynson W. H. Lau
Department of Computer Science
City University of Hong Kong
(jiayinlin--e, yh. yh..y. cityu. eityy.edu...e....... edu.hk, Rynson. Lauocity eew.ttyy.uee..ee..
Abstract
Glass surfaces are omnipresent in our daily lives and often go unnoticed by the majority of us. While humans are generally able to infer their locations and thus avoid collisions, it be difficult for current object detection systems handle them due to the transparent nature of glass surfaces. Previous methods approached the problem by extracting global context information obtain priors such as object boundaries and reflections. However, their performances cannot be guaranteed when these deterministic features are not available. We observe that humans often reason through the semantic context of the environment, which offers insights into the categories of and proximity between entities that expected to appear in the surrounding. For example, the odds of the co-occurrence of glass windows with walls and curtains are generally higher than that with other objects, such as cars and trees, which have relatively less semantic relevance. Based on this observation, we propose a model named Glass Semantic Network ("GlassSemNet') ) that integrates the contextual relationship of the scenes for glass surface detection with two novel modules: (1) Scene Aware Activation (SAA) Module to adaptively filter critical channels with respect to spatial and semantic features, and (2) Context Correlation Attention (CCA) Module to progressively learn the contextual correlations among objects both spatially and semantically. In addition, we propose a large-scale glass surface detection dataset named Glass Surface Detection Semantics ( 'GSD-S'), which contains 4,519 real-world RGB glass surface images from diverse real-world scenes with detailed annotations for both glass surface delection and semantic segmentation. Experimental results show that our model outperforms state-of-the-ar works, especially with 42.6% MAE improvement available at https: // jäaying. Link/ nwwrips2022-gaaaa gads/
proposed GSD-S dataset. Code, dataset, and models are
Introduction
Glass surfaces, including glass doors, windows, and walls of modern architecture, are becoming prevalent in our daily lives. Due to the ambiguity of the transparency property, the autonomous systems contemporary works typically lack the ability to identify glass surfaces. With such characteristics, the peripheral environment displayed by the glass surface only contains opaque objects and scenes from the surroundings. These result in a myriad of potential dangers caused by the impairment of the existing object detection models in handling glass surfaces, manifested in previous works [1, 2]. Consequently, brings a pressing need for a better glass detection model. Existing methods have explored many characteristics of glass surfaces, including context [3],
'Joint first authors. "Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Image
GDNet [3]
GlassNet [4]
Ours
GT
Figure 1: Existing methods [3, 4] may fail when explicit physical cues (e.g., boundaries and reflec- tions) are unreliable. In the top row, the center region has glass-like boundaries, and existing methods wrongly predict this region glass. In the bottom row, the smaller window the right does not have obvious reflection, it can perplexing to detect. Our model can accurately detect glass surfaces in both situations, which are not distracted by regions with glass-like boundaries through learning semantics. It generalizes well even to glass regions without obvious reflection.
boundary [4], reflection [4] and polarization [5]. Although these methods generally perform well, when the assumptions that these models make do exist, the detection ability of these models is significantly impeded. In addition, these methods only exploit the pixel-level binary labels for glass surfaces, which introduces a bottleneck for the model on knowledge learning, as opposed other models with multi-class learning that encourage knowledge acquisition through a more diverse pool of knowledge.
In this paper, we observe that there is often a correlated occurrence of glass surfaces with some surrounding objects in the scene. For example, glass surfaces of 'windows' tend to co-occur with "curtains" (or "blinds") and 'glass doors' with 'walls'. This prompts us to reconsider the problem from new perspective by incorporating knowledge on top of superficial features. Hence, we propose to focus more on semantic context learning. Studies in cognitive neuroscience [6. 7, 8, 9] demonstrate that surrounding objects can offer an effective derivation of contextual information. In addition, applications of contextual information have had many proven successful impacts. Examples tasks include salient object detection [10], domain adaptation [11], network pruning [12], image style transfer [13], and image-text retrieval [14]. Based on these and our findings, we propose to learn and
incorporate the relationship between glass surfaces and their surrounding semantic context for glass surface detection.
Figure shows that existing methods are often misled by obscure scenes wherein explicit physical cues such as boundaries and reflections are missing. To address these issues, we propose novel model exploiting semantic relations for glass surface detection. GlassSemNet adopts the encoder- decoder architecture. The encoder component is supported by two backbones: I) a SegFormer network [15] for comprehensive spatial context learning, and 2) a ResNet50 network [16] for semantic relationship learning. The semantic backbone DeeepLab V3--ResNttRO)O is first pre-trained with regular segmentation modeling comprehend scene context reference and object relationships in real-world settings. The backbone features from the DeepLabY 33+++ classifier are projected to semantic encodings that are fed into downstream modules to serve as contextual hints for subsequent information extraction. Specifically, two modules are proposed to achieve the ontology learning objectives to assist glass surface detection: I) the Scene Aware Activation (SAA) Module to guide contextual feature modeling, and 2) the Context Correlation Attention (CCA) Module associate spatial context and semantic meanings of objects in the environment. Inspired by SENet [17], the SAA Module consists of two feature selection pathways that respectively assimilate the information concerning object locations and object categorical connotations. The CCA Module adopts the Transformer block [18] conduct attention modeling between the extracted backbone features.
Besides, we notice that although Mei et al. [3] and Lin el al. [4] both propose datasets for glass surface detection, these datasets do not contain semantic data (e.g., semantic labels of different objects around glass surfaces) to model the spatial context and high-level scene context. To further the research on glass surface detection, propose new large-scale challenging semantic-aware

---

glass surface dataset (GSD-S) with ground truth semantic labels, not only limited to the binary masks of glass surfaces. Our dataset consists 4,519 images collected from various scenes. It is larger than those proposed by Mei et al. [3] (3,900 images) and Lin el al. [4] (4,102 images), and can largely facilitate research in this area. Extensive experiments and evaluations on all three datasets were conducted to validate the performance of our model.
Our contributions can be summarized as follows:
We propose strategy to apply semantic relationship modeling to cognitively infer the correlation between glass objects and everyday objects for glass surface detection.
We present two novel deep learning modules to capture long-range spatial and implicit semantic dependencies, with results being substantiated by thorough studies.
We have built large-scale dataset, which has complex and challenging scenes with semantic contexts, It can serve as a benchmark for performance validation on future models.
We have conducted extensive experiments to evaluate our model's 's robustness and show that it outperforms state-of-the-an methods on glass surface detection.
2 Related Work
Transparent Object Detection. Transparent Object Detection aims to identify glass-made objects, specifically with bounded shapes such as glasses and glass bottles; occasionally, the task also accommodates window panels. Existing works approached this task by leveraging the bounded shapes to localize the position of prospective transparent objects. The methods range from simple as adopting an encoder-decoded structure to extract boundary [19, 20, 21] and surface normal [22]. Light polarization utilized to capture the rotation of light waves from Physics perspective [23] and multi-view stereo images to generate depth maps that further outline the object shape information [23]. However, glass panels with flat surfaces usually do not possess the boundary characteristic, which induces an even more challenging obstacle for detection models.
Context-Aware Detection. Context-aware methods tackle the limited receptive field bottlenecks of convolutional kernels. Most works employed auxiliary operations such as dilation [24, 25] and pooling [26] to enlarge the receptive field such that it gains more global contextual information. More specific solutions that are tailored for various types of surface detection also follow this fashion of contextual learning, such for mirror surface [27, 28], and glass surface [3, 4]. Nevertheless, these methods are yet another feature aggregation strategy that leaves behind the implicit reasoning embedded in the network learning and rarely exploits the explicit semantic relationship.
Attention-basee Detection. Attention mechanism from [29] enables the modeling to be more specific and oriented towards meaningful context. Early works used matrix formulation to construct such attention purpose [26, 30. 31J. A boost in performance is nurtured by [32], which actualized the 'transformer' concept in Computer Vision tasks. Nonetheless, it is still in the form of spatial context, which relies on semantic feature extraction from each local patch. [33, 34] proposed novel strategies instead focus semantic context to study the relationships between objects and scenes for semantic meanings, which served as an inspiration for our work to utilize semantic dependency. Subsequent work [35] swiftly merged the 'transformer' concept and semantic category embedding in transparent object detection. However, the model is constrained to model relationships between a few types of transparent objects, e.g., eyeglasses, bowls, freezers, and windows. This method completely ignores and disposes of the potential of semantic meanings between object categories and scenic information; we aim to fill the gap by emphasizing contextual relationships.
Proposed Dataset
We acknowledge that there exist numerous datasets [36, 37] that are dedicated semantic seg- mentation tasks. Since most of them are augmented for everyday objects and thus have a general classification purpose, more refined labelings would be required. Consequently, a rectified dataset that caters explicitly to glass surface detection concerning semantic context is still in need. On the other hand, [3] and [4] are among the earliest teams who pioneered the 'glass detection' studies and contributed large-scale glass datasets. While half of [4]'s GSD dataset was assembled from
3

---

Table 1: Composition of our proposed GSD-S dataset. We collect glass images from four existing RGB image datasets with semantic annotations. Note that these datasets initially lack refined
annotations of ground truth glass surface masks. When constructing our dataset, we re-labeled the
GT masks for the glass surfaces.
Dataset
Whole
Train
Test
SUN RGB-D [38]
1,203
920
283
2D-DD-eee SD-Semantics [39]
600
488
112
Matterport3D [2]
1,206
992
213
COCO-Stuff [37]
1,511
1,511
N/A
Total
4,519
3,911
608
Figure 2: Preview of the GSD-S Dataset. Triplets of (RGB images, semantic maps, overlaid GT mask of glass surfaces).
existing semantic segmentation datasets (c.g. [36, 37J). [3]'s GDD dataset was constructed from manual collection and organization with only ground truth glass masks. We herewith propose new Semantic-Awar Glass Surface Detection (GSD-S) dataset, which is accompanied by polished ground truth 'glass surface' masks along with semantic segmentation ground truths, with the hope that this contributes to further extensions.
Dataset Composition. The GSD-S dataset is scrupulously organized from existing semantic seg- mentation datasets [40] with the corresponding ground truth annotation carefully refined since the glass mask labelings in original versions were in general inconsistent. Examples of ambiguous mask labeling include glass areas being segmented into different categories surrounding objects due to the glass surface transparency; and glass surfaces that were ignored and treated as part of the larger subject, such as cupboard (glass door), car (glass window), table (glass table). We processed 4.519 images, with 3,911 training images and 608 testing images altogether. The split between training and testing sets strictly follows that of the original datasets whenever possible (subset from Matterport3D [40] randomly sampled). Table 1 gives overview of the mentioned distribution. Figure displays the object category distributions 43 classes in our GSD-S dataset. Both training and testing sets follow similar distributions. The area ratio of GSD-S is distinctively small compared to the other datasets due to the consideration that more semantic context can be included, which allows comprehensive scene context relationship modeling.
GOAS
013
(a) Class Distribution
(b) Area Ratio
Figure 3: Dataset Statistics: (a) class distribution, and (b) area ratio.

---

Spstal
Serante
Afrendal
Decode:
Activaed
Pazza
SAA
Module
Ouput
Iuput
Figure 4: Model Architecture. We first feed the input image into two backbones to capture semantic knowledge, and spatial location features separately. Together with the semantic encodings, low-level features first get selectively activated by the SAA Module with respect to the decoupled features. The CCA Module is placed at a higher level learn the relationships between contextual meanings and locations of objects. Features from multiple stages aggregated by the UPerNet decoder to produce the output map, along with the intermediate feature maps for supervision.
Proposed Method
Figure 4 illustrates the proposed model's architecture. The input image is first fed into two backbone networks for spatial and semantic feature extraction. To bridge up upstream and downstream module features, we introduce the semantic encodings [Ssem_encod E R where ne = number of object categories 43) generated by the semantic backbone into the SAA and CCA Modules. The collection of enhanced feature maps output by our two novel modules are processed by the UPerNet [41] decoder network. Specifically, with output features produced by SAA and CCA Modules, the decoder which adopts the Feature Pyramid Network structure was configured in accordance with the original paper: Internal Channel Number {128, 256, 512, 1024), Linear Layer Dimension 512, Pooling Scales 3,,2,3,6) 6}.
4.1 Backhone Networks
The backbone networks complement each other in that we hope to explicitly leverage spatial and semantic features. The spatial-wise attention in the SegFormer backbone offers insight into each object's geographic information along with corresponding proximity. This considers that objects can arise in different regions in the picture under various types circumstances. For example, glass windows of commercial buildings that situate in different corners of the scene. Despite the spatial distance, they have hidden dependencies and should belong to the same category. Meanwhile, glass surfaces appear in the form of 'glass door' in comer and 'glass table' in another. Given the different forms of existence on top of varying physical shapes, we need to devise the conceptual meanings such that the model can better differentiate the semantic categories while correlating the implicit relationships among objects. The SegFormer [15] backbone comes into play with its capacity capture long-range dependencies and correlate spatial features with attention. With its lightweight capacity, the ResNet [25] semantic backbone serves as an auxiliary semantic context aggregator. Concretely, the integration of both paths will enable the differentiation of object representations and correlation on object dependency. According to empirical results (Section 5.3), we segregate the backbone feature layers terms low-level for / = 2,, 2, and high-level fr. for h 4 features respectively for SSA and CCA Modules. The low-level features are used for context and object differentiation as they retain nigh-resolution spatial context and thus are effective for fine- an--grained details. The high-level features embedded with richer and abstract context information are used for correlations on seteegory-speciiie knowledge.
4.2 Scene Aware Activation (SAA) Module
Inspired by [42], the information contained in feature maps from each layer can be further reinforced through selection and activation operations. Compared to [42], which only considers generic convo-
5

---

33
Plss Exst
Sanarik Taraees
ML
Con
Mall
Norr MLP
Maln Llead Amension
(a) SAA Module
(b) CCA Module
Figure 5: The SAA Module takes low-level backbone features activate and discriminate object
meanings. Higher-level features are used for contextual information correlation.
lutional layers, we decouple the enhancement process into respective spatial and semantic paths to suit our contextual learning settings."
Sap RRXXXC - S.p &R
fee E HXW - S.c Rrex
Activation feei (semmencod) (fep x Fep) @ (fee x (foe + (eem))nood))
Given the spatial backbone features fep would be first compressed into fap using Global Channel Pooling ('GCP") operations with respect to channel dimension before getting projected back onto fap- For fee. it would be compressed spatially using Global Spatial Pooling ('GSP') before being flattened (i.e. foe E # xC). It then will be transformed into See. using a 'Multi-Layer Perception" ('MLP") operation, with the number of channels changing from originally HW to nc 43 (number of classes). After that, these transformed features will be integrated with the semantic encodings from the semantic backbone. This insertion of contextual knowledge, indicated by the 'Semantic Encoding' ('SE') stage, colored in pink in Figure 5, fosters the subsequent refinement modules to be aware of the object correlations for better target localization. Note that the resultant (5 + ssem_ ) E Te will be re-converted back into HW x. through MIP and reshaped into the same shape fse E IR xxxxX before the final projection. Lastly, @ indicates the arithmetic fusion operations to integrate the processed spatial and semantic features, as detailed in Figure 5(a).
4.3 Context Correlation Attention (CCA) Module
Witnessing the success of ViT [32], the attention mechanism significantly promotes the modeling efficiency on long-range dependency, enabling objects in any region to be thoroughly analyzed. Existing methods generate the (query, key, value) triplets from single input achieve self-attention. We propose to bifurcate the attention procedure with respect to spatial and semantic features to model the correlation between objects of different categories and their corresponding locations.
Sap E HXWxC + + HW
Sso HXWxC |K E RmoxC E B
Q(K + (sem_encod) (V + fsem_encod)
Attention(Q, K,V; semm_needd softmax
ydg
Similar to the preprocessing performed for SAA Module, spatial and semantic features fap and fae are first converted (Query, Key, Value) triplet through flattening and channel compression. Specifically, the "Key" and 'Value' features will go through transformation[117 (indicated by the self-loop in blue). Semantic encodings are then reinforced into these transformed bottleneck layer features before conducting feature attention modeling.
Formulaic details have been omitted for brevity.

---

Table Evaluation results on GDD and GSD.
Dataset
GDD
GSD
Methods
Venue
IOUT
Fa f
MAEJ
BER,
IOU1
Fa t
MAEJ
BER,
PSPNet [48]
CVPR 2017
0.792
0.875
0.132
11.51
0.703
0.834
0.110
10.66
BDRAR [49]
ECCV 2018
0.800
0.908
0.098
9.87
0.759
0.860
0.081
8.61
BASNet [50]
ICCV 2020
0.808
0.891
0.106
9.37
0.698
0.808
0.106
13.54
MINet [51]
CVPR 2020
0.844
0.919
0.077
7.40
0.773
0.879
0.077
9.54
GateNet [52]
ECCV 2020
0.817
0.931
0.073
8.84
0.689
0.898
0.073
10.12
MirrorNet [27]
ICCV 2019
0.851
0.903
0.083
7.67
0.742
0.828
0.090
10.76
PMD [53]
CVPR 2020
0.870
0.930
0.067
6.17
0.817
0.890
0.061
6.74
GDNet [3]
CVPR 2020
0.876
0.937
0.063
5.62
0.790
0.869
0.069
7.72
GlassNet [4]
CVPR 2021
0.881
0.932
0.059
5.71
0.836
0.901
0.055
6.12
Ours
0.908
0.950
0.045
4.34
0.856
0.920
0.044
5.60
5
Experiments
5.1
Implementations
Specifically, the SegFormer backbone adopted variation B5 of Mix Transformer encoders (MiT- B5) The model coupled with pre-trained weights for the purpose transfer learning. The ResNet backbone is based on PyTorch's DeeplabV3-Res
model [25] pre-trained on COCO train2017 [43] with only 21 categories from Pascal VOC [44]. We further trained the model using GSD-S dataset introduce more diverse set of object categories for better semantic extraction capacity. Note that the semantic backbone after pre- tre-training is fixed and isolated from subsequent training processes for better glass surface detection, lest additional information would distort the learned semantic representations. Kaiming uniform initialization [45] is used before the model was trained NVidia RTX 2080Ti GPU. The input data first uniformly resized the size of 384 x 384 before applying normalization. A joint loss, which a combination of binary cross entropy and Lovász-Softmax loss [46], was used supervise the intermediate feature maps (i.e. layers 2 and 4) and final output. The prediction evaluation is accompanied by Fully Connected Conditional Random Fields (CRF) [47] technique for binarization refinement. The evaluation metries include intersection over union (IoU), Mean Absolute Error (MAE), maximum F- F-measure (Fz), and balance error rate (BER).
5.2 Comparisons
Table 3:
Evaluation results on GSD-S.
Methods
Venue
IOU
Fg f
MAET
BER
PSPNet [48]
CVPR 2017
0.560
0.679
0.093
13.40
D3++LabV3+ [54]
CVPR 2018
0.557
0.671
0.100
13.11
PSANet [55]
ECCV 2018
0.550
0.656
0.104
12.61
DANet [56]
CVPR 2019
0.543
0,673
0.098
14.78
SCA-SOD [57]
ICCV 2021
0.558
0.689
0.087
15.03
SETR [58]
CVPR 2021
0.567
0.679
0.086
13.25
Segmenter [59]
ICCV 2021
0.536
0.645
0.101
14.02
Swin [60]
ICCV 2021
0.596
0.702
0.082
11.34
ViT [61]
ICLR 2021
0.562
0.693
0.087
14.72
SegFormer [15]
NeurIPS 2021
0.547
0.683
0.094
15.15
Twins [62]
NeurIPS 2021
0.590
0.703
0.084
12.43
GDNet [3]
CVPR 2020
0.529
0.642
0.101
18.17
GlassNet [4]
CVPR 2021
0.721
0.821
0.061
10.02
Ours
0.753
0.860
0.035
9.26

---

Image
DANet [56] SETR [58]
Swin [60]
GDNet [3] GlassNet [4]
Ours
GT
Figure 6: Existing methods tend to mis-detect non-glass regions due to some distractions (e.g.. window blinder in the 1s row and open in the 3rd and 4th rows). Our model can accurately determine the glass regions by learning their correlation with the surrounding semantics.
We evaluated GlassSemNet against other latest methods, including PSPNet [48), Deepl Lab [63], PSANet [55]. DANet [56] for generic semantic segmentation, and SCA-SOD [10] for Salient Object Detection; recent avant-garde models that utilize transformer technique such as SETR [58], Segmenter [59], Swin [60], VIT [32], SegFormer [15]. Twins [62]; and glass surface detection models, GDNet [3] and GlassNet [4]. These methods are re-trained on GDD, GSD, and GSD-S, following the default training settings stated in the original papers. Table 2 and Table 3 outlines the quantitative performance on the three glass detection datasets concerning the four evaluation metrics, which shows that GlassSem gives major performance increase compared most models. Compared to the second best model i.e. GlassNet (4), GlassSemNet surpasses with an improvement 4, 4.44%, 4.75% 42.6% and 7.58% respectively for IOU, F,,, MAF, and BER GSD-S. On the other hand, the significant disparity illustrates the challenging nature of our dataset since GSD-S has a relatively small area ratio for glass surfaces, thus giving more room for diversified set of additional objects that offer richer semantic context. While previous methods, e.g. GDNet, are not completely catered for such assorted scenarios, GlassNet stands with its Rich Context Aggregation Module. However, with the assistance of an understanding of intricate scene context, the performance of GlassSemNet further elevated.
Figure shows the qualitative comparisons of our method with the state-of-the-an Existing methods that rely boundary features [3] and contextual contrasts [4] often find the scene confusing with the presence of diverse object categories. contrast, GlassSemNet can handle these complex cases and correctly segment the glass regions. We also conducted additional tests real-world outdoor scenes for robustness in terms of generalization, which proved the effectiveness of our model.
5.3 Ablation Study
Ablation study was conducted to validate the contribution of each module, as detailed in Tables 4-
8. Note that 'BB' refers to the backbone features Sap and Ssc while 'SE' refers to the 'Semantic Encoding' insertion into the backbone features as illustrated in Figure 5.
Split Level and Attention Encoding Formulation. We started by tuning the threshold of low- and high-level feature segregation (Versions A C, Final in Table 4). For instance, we changed the module inputs.
from to
SSA(fiow) where low [1,2] and CCA(fhigh) where high = (3, 4) (Ver. B) SSA(ft where low = [1, 2, 3) and CCA(fnigh) where high = [4] (Ver. C).
A mild degradation is observed in IOU, Fg, and MAE. On the other hand, we compared the two options of embedding vectors (Q, K, V) assignment. The initial trial designates spatial features to be Query, Value) embeddings and semantic backbone features be "Key" embeddings (Ver. A

---

Table 4: Ablation study (Split and QKV).
Version
Split
SSA
CCA (SpatiallSemantic)
IOU1
F3
MAE
BER
A
11234
BB + SE
BB + SE (QVIK)
0.748
0.853
0.0360
9.66
B
12134
BB + SE
BB + SE (QVIK)
0.750
0.850
0.0359
9.39
C
12314
BB + SE
BB + SE (QVIK)
0.749
0.847
0.0366
9.27
D
11234
BB + SE
BB + SE (QIKV)
0.747
0.853
0.0346
9.62
E
12134
BB + SE
BB + SE (QIKV)
0.749
0.852
0.0358
9,41
Final
12314
BB + SE
BB + SE (QIKV)
0.753
0.860
0.0351
9.26
C). The roles are then switched in the subsequent trials (Version D Final), where performance gains are observed. We deduced that after the correlation mapping between Q and K, using semantic backbone features as V allows a more variegated query space than that by spatial features, which only has information in terms of separate patches over the scene.
Semantic Encoding Enhancement. dateggry--spccifi information was asserted iteratively into SSA and CCA Modules (Table 5). Compared other partial insertions (Ver. A C). The positive effect is backed by apparent enhancement when the encodings were applied on both modules (Ver. D).
Table 5: Ablation study (Semantic Encoding).
Version
Split
SSA
CCA (SpatiallSemantic)
IOUt
F3 f
MAEЛ
BER
12314
BB
BB (QIKV)
0.751
0.856
0.0350
9.30
B
12314
BB + SE
BB (QIKV)
0.746
0.849
0.0363
9.60
C
12314
BB
BB + SE (QIKV)
0.747
0.854
0.0361
9.84
Final
12314
BB + SE
BB + SE (QIKV)
0.753
0.860
0.0351
9.26
Proposed Modules. As manifested by the results upon module removals (Table 6), the results
demonstrate that the mutual presence of both SSA and CAA Modules offer certain advancement on all metrics compared to the cases when either one was missing. This confirms the significance of the SSA Module in object characteristic differentiation, well that of the CAA Module context correlation.
Table 6: Ablation study (SAA and CCA).
Version
Split
SSA
CCA (SpatiallSemantic)
IOU1
Fs
MAET
BER
A
12314
X
BB + SE (QIKV)
0.747
0.854
0.0364
9,40
12314
BB + SE
X
0.747
0.855
0.0361
9.88
Final
12314
BB + SE
BB + SE (QIKV)
0.753
0.860
0.0351
9.26
Semantic Backbone. To verify the effectiveness of our GSD-S Dataset, vary the configurations on GSD-S fine-tuning ("train")) and parameter update ("fix") of the semantic backbone network. In general, as can be seen from Table fine-tuning on GSD-S (Ver. and C) does indeed offer an incremental performance upgrade, regardless of the parameter update fixing. However, fixing the gradient resulted in comprehensive boost across all the evaluation metrics. This is in accordance with our hypothesis that semantic priors from pre- pre-traeeeete vee-raaiing can be better preserved from information distortion by downstream tuning and thus provides more accurate and insightful contextual meanings.
Cross-Datase Analysis. To illustrate the effectiveness of our proposed dataset GSD-S for model training, conducted cross-dataset analysis[64 GSD [4] (Table 8) by training out-of- distribution data, i.e. GDD [3] and GSD-S. The result showed that the model trained on GSD-S generalizes better than that GDD.
5.4 Limitation
Our model would have constrained performance under some challenging cireumstances. For instance, in the presence of a "mirror" where there exist high-resolution reflections, scenery along with clear

---

Table Ablation study (Semantic Backbone).
Version
Split
SSA
CCA (SpatiallSemantic)
IOU1
Fa f
MAE
BER,
A
12314
BB + SE
BB + SE (QIKV X train X fix)
0.739
0.847
0.0377
10.06
B
12314
BB + SE
BB + SE (QIKV train v fix)
0.744
0.851
0.0361
9.70
C
12314
BB + SE
BB SE (QIKV V train fix)
0.747
0.855
0.0356
9.52
Final
123/4
BB + SE
BB + SE (QIKV train v fix)
0.753
0.860
0.0351
9.26
Table 8: Ablation study (Cross Dataset).
Train
Test
IOU1
Fg
MAE
BER
GDD
GSD
0.701
0.782
0.129
11.20
Glass-Seg
GSD
0.774
0.882
0.0857
9.44
semantic context is reflected. This leads to wrong prediction of false-positic glass surface presence inside the mirror region (shown in the left group of Figure 7), which is admittedly unsatisfactory. Moreover, small surfaces often difficult to identify, given the limited space of image content occupied. Thus, the diminished attention makes it go unnoticed (illustrated by the glass table in the right group). This challenge applies to most models, including the semantic backbone network and our main model. In the future, it is hoped that we can integrate different detection strategies to alleviate this bottleneck.
Image
Ours
GT
Image
Ours
GT
Figure 7: Limitations. Our method may fail to detect glass surfaces in some very challenging scenes with ambiguous visual semantics caused by mirrors or objects small sizes.
Conclusion
In this paper, we have proposed consider semantic knowledge in combination with spatial infor- mation to better capture the scene context as strategic enhancement for tackling the glass surface detection problem. This comes with a meticulously constructed large-scale dataset with refined ground truth masks for both glass surface detection and semantic segmentation. Thorough experi- mentation demonstrates the capability the SAA Module on object characteristic differentiation and the effectiveness of the CCA Module on context correlation. Our experiments show that the proposed model sets new benchmark records on all existing glass detection datasets, including GDD [3], GSD [4], and our GSD-S.
Ackmowledgements This work is partially supported by two SRG grants from City University of Hong Kong (Ref: 7005674 and 7005843).
10

---

References
[1] Oliver Zendel, Katrin Honauer, Markus Murschitz, Martin Humenberger, and Gustavo Fernan- dez Dominguez. Analyzing computer vision data the good, the bad and the ugly. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
[2] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[3] Haiyang Mei, Xin Yang, Yang Wang. Yuanyuan Liu, Shengfeng He. Qiang Zhang, Xiaopeng Wei, and Rynson W.H. Lau. Don't it hit me! glass detection world scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] Jiaying Lin, Zebang He, and Rynson W.H. Lau. Rich context aggregation with reflection prior for glass surface detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 13415- 13424, June 2021.
[5] Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung-Hwan Back, Felix Heide, Pieter Peers, Xiaopeng Wei, and Xin Yang. Glass segmentation using intensity and spectral polarization cues. In /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[6] Daniel Kaiser, Timo Stein, and Marius V Peelen. Object grouping based real-world regularities facilitates pereeption by reducing competitive interactions in visual cortex. Proceedings of the National Academy of Sciences, 1217:11117711222 2014.
[7] Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 56617-629, 2004.
[8] Moshe Bar and Elissa Aminoff. Cortical analysis of visual context. Newron, 38:347-58, 2003.
[9] Aude Oliva and Antonio Torralba. The role of context in object recognition. Trends in Cognitive Sciences, 11(12):520- 527, 2007.
[10] Avishek Siris, Jianbo Jiao, Gary K.I., Tam, Xianghua Xie, and Rynson W.H. Lau. Scene context-aware salient object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 4156-4166, October 2021.
[11] Jinyu Yang, Weizhi An, Chaochao Yan, Peilin Zhao, and Junzhou Huang. Context-aware domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision (WACV), pages 514-524, January 2021.
[12] Wei He, Meiqing Wu, Mingfu Liang, and Siew-Kei Lam. Cap: Context-aware pruning for semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 960-969, January 2021.
[13] Yi-Sheng Liao and Chun- Roon Huang. Semantic context-aware image style transfer. IEEE Transactions Image Processing, 11:1911-1923, 2022.
[14] Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z. Li. Context-awand attention network image-text retrieval. Proceedings of the IEFF//VV Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[15] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo, Segformer: Simple and efficient design for semantic segmentation with transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 22077-20990 Curran Associates, Inc., 2021.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016.
[17] Jie Hu, Li Shen, and Gang Sun. boneeezz-and-xxiiaatiie networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[18] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg. S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sysiems, volume 30. Curran Associates, Inc., 2017.

---

[19] Guanying Chen. Kai Han, and Kwan- Yee K. Wong. Tom-net: Learning transparent object matting from single image. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[20] Enze Xie, Wenjia Wang. Wenhai Wang. Mingyu Ding. Chunhua Shen, and Ping Luo. Segmenting transparent objects in the wild. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-20 28, 2020, Proceedings. Part XIII. page 696-711, Berlin, Heidelberg. 2020. Springer- Verlag.
[21] Hao Hc, Xiangtai Li, Guangliang Cheng. Jianping Shi, Yunhai Tong, Gaofeng Meng, Véronique Prinet, and LuBin Weng. Enhanced boundary leamning for glass-like object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15859-15868, October 2021.
[22] Zhenggin Li, Yu- fu-Ying Yeh, and Manmohan Chandraker. Through the looking glass: Neural 3d reconstruc- tion of transparent shapes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[23] Agastya Kalra, Vage Taamazyan, Supreeth Krishna Rao, Kartik Venkataraman, Ramesh Raskar, and Achuta Kadambi. Deep polarization eues for transparent object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[24] Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang. Xiaogang Wang. Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[25] Liang- -hich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05558 2017.
[26] Junjun He, Zhongying Deng. Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pateenn Recognition (CVPR), June 2019.
[27] Xin Yang. Haiyang Mei, Ke Xu, Xiaopeng Wei, Baocai Yin, and Rynson W.H. Lau. Where is my mirror? In ICCV 2019.
[28] Jiaying Lin, Guodong Wang, and Rynson W. H. Lau. Progressive mirror detection. In CVPR 2020.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
[30] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[31] Changqian Yu. Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[33] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph E. Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Where do transformers really belong vision models? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 599-609, October 2021.
[34] Mingyuan Liu, Dan Schonfeld, and Wei Tang. Exploit visual dependency relations for semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9726 9735, June 2021.
[35] Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo., Segmenting transparent object the wild with transformer, 2021.
[36] Bolci Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJV, 127(33302--31, 2019.
[37] Holger Caesar. Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff' classes in context. In CVPR 2018
12

---

[38] I. Armeni, A. Sax. A. R. Zamir, and S. Savarese. Joint 2D-3D-Semanti Data for Indoor Scene Under- standing, ArXiv e-prints, February 2017.
[39] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567-576, 2015.
[40] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nicßner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D D data in indoor environments. In 3DV 2017.
[41] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418 434, 2018.
[42] Sanghyun Woo, Jongchan Park. Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pages 3-19, 2018.
[43] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014.
[44] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2), 2010.
[45] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In CVPR 2017.
[46] Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. The lovász-softmas loss: A tractable surrogate for the optimization of the
measure neural networks. In CVPR, 2018.
[47] Philipp Kräähenbühl and Vladlen Koltun. Efficient inference in fully connected CRFs with gaussian edge potentials. In NewrlPS, pages 109-117, 2011.
[48] Hengshuang Zhao Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[49] Lei Zhu, Zijun Deng, Xiaowei Hu, Chi- Biig Fu, Xuemiao Xu, Jing Qin, and Pheng- Ana Heng. Bidirec- tional feature pyramid network with recurrent attention residual modules for shadow detection, In ECCV, 2018.
[50] Xuebin Qin, Zichen Zhang, Chenyang Huang. Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary- aware salient object detection. In CVPR, 2019.
[51] Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan La. Multi-scale interactive network for salient object detection. In CVPR, June 2020.
[52] Xiaoqi Zhao, Youwci Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang. Suppress and balance: A simple gated network for salient object detection. In ECCV, 2020.
[53] Jiaying Lin, Guodong Wang, and Rynson W.H. Lau. Progressive mirror detection. In CVPR, 2020.
[54] Jiang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[55] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: woint-wise spatial attention network for scene parsing. In ECCV, 2018.
[56] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019.
[57] Avishek Siris, Jianbo Jiao, Gary Tam, Xianghua Xie, and Rynson Lau. Scene context-aware salient object detection. ICCV 2021.
[58] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. In CVPR, 2021.
13

---

[59] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7262-7272, 2021.
[60] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[61] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
[62] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS 2021, 2021.
[63] Liang-Chich Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017.
[64] Antonio Torralba and Alexci A. Efros. Unbiased look at dataset bias. In CVPR 20/1, pages 1521-1528, 2011.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes]
Did you include the license the code and datasets? [No] proprictary.
The code and the data
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Yes
(b) Did you describe the limitations of your work? [Yes] Yes
(c) Did you discuss any potential negative societal impacts of your work? [N/A] N/A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A] N/A
(b) Did you include complete proofs of all theoretical results? [N/A] N/A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Yes
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] N/A
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Yes
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes
(b) Did you mention the license of the assets? [Yes] Yes
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
N/A
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Yes
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] N/A
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] Yes
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] N/A
15


Reviews:
[{"review_id": "tRh2yO3h2DZ", "reviewer": "Reviewer_yyRs", "summary": "The paper addresses the issue of segmenting glass surfaces in an image. The novel contribution is to use semantic context more explicitly in the overall network system design, as often there are no visual features with which to identify glass. The authors also contribute an extensive new data set with carefully labeled glass areas that includes many images from standard data segmentation data sets. The new network design outperforms the state of the art methods.", "questions": "Did the authors find any situations where the context fooled the network into thinking glass existed when it did not?", "limitations": "The authors note the limitation created by mirrors or mirror-like surfaces. There are no examples shown where context fools the system into thinking glass is present when it is not (e.g. an open window).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The network incorporates both a spatial and a semantic (area+category) based backbone, followed by two modules that combine inputs from both backbones using different architectures. The ablation study shows that both modules are important for performance.\n\nThe ablation study seems comprehensive, and it includes how best to assign the query/value/key concepts in the context module.\n\nThe network design outperforms existing state-of-the-art networks on both prior data sets and the new contributed data set.\n\nThe authors are providing a new and useful data set for glass detection.\n\nFigure 3 has space for more annotation of the components. All component blocks should have a label, and the reader should not have to go back to the text to figure out what the components are (for example, if Upernet is used for the decoder, that should be clearly labeled in figure 3.\n\nThe paper needs editing for minor details.  For example, in the caption for figure 3\n\n\"The CCA Module is positioned at higher level in the encoder component to inference the relationships between contextual meanings and locations of objects. Features from multiple stage are aggregated by decoder to produce output map.\" -> \"The CCA Module is positioned at a higher level in the encoder component to infer the relationships between contextual meanings and locations of objects. Features from multiple stages are aggregated by the decoder to produce the output map.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fOVH4A1a1eU", "reviewer": "Reviewer_pjyT", "summary": "The authors observe that the glass object usually cooccurs with some specific objects. Based on the observation, the work proposes a method consisting of two main components CCA and SAA to feature selection in spatial and semantic dimension and efficiently model long range dependency.  Extensive experiments are conducted on their newly collected data to demonstrate the effectiveness of the proposed method and its superiority to the state-of-the-art methods.   ", "questions": "Please address the weaknesses mentioned above. My major concern is that the observation is specifically with glass object, thus the proposed method does not deal with challenges caused by the characteristics of glass object. ", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The performance of the proposed method is promising.\n2. The collected data is useful to facilitate the research in this direction.\n\nWeaknesses\n1. The observation is common and lacks insight, thus making the work less inspiring. In addition, the observation of some object is closely related with some other objects is not only existed in glass object. For example, sea is usually closed with sand.\n2. The proposed CCA and SAA modules are not specifically designed for glass object, it is more like a general way to model the correlation among objects. In this case, the methods should be evaluated in more general segmentation task, where the observed relationship is present as well. \n3. Line 215, the authors mention that \"We then further fine-tuned the model using our GSD-S dataset\". Did all compared models do this fine-tuning? How dose it affect the model performance?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LHF9tdqwzbf", "reviewer": "Reviewer_KsvV", "summary": "\nThe paper tackles the issue of glass surface reflection from a single optical image. The authors observe that glasses are difficult to detect as isolated objects (they do not have specific radiometric or geometric properties). Consequently, the authors propose to account for the semantic context around objects, with the aim to ease the task. Concretly, the authors introduce a network architecture that encompass semantic information and low level features. A new dataset of glass is also created. Results are illustrated on a classic benchmark (Glass Detection Dataset, Mei and al. 2020), and the authors' dataset. \n", "questions": "1- I am little confused about the use of SegFormer and DeepLab, which are according to the authors, \"aggregate spatial-wise object location features\"  for the former, and give \"intrinsic representations of each object category\" for the latter. However, these two networks, to my knowledge, are two semantic segmentation networks. I might misunderstand the differences of usage of these two networks, it would be worth to clarify. \n\n2- The training procedure is important indeed, and a few additional words on this matter would be interesting.  The authors write: \"We then further fine-tuned the model using our GSD-S dataset to introduce a more diverse set of object categories for better semantic extraction capacity. Note that the semantic backbone after fine-tuning is fixed and isolated from subsequent training for glass surface detection\". I do not understand why a fine tuning then a subsequent training is necessary? What is learnt at each stage and for what purpose? \n\n3- Equations 2, describe in a very classical way the attention frameawork. What is the dimensionality of f_sp, f_se, Q,K,V. ", "limitations": "No concern with societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main idea of the authors, ie, accounting for the semantic context surrounding glass surface so as to ease the detection, is certainly interesting. Thought it has been proposed by others for other applications  (in particular for human detection),  it is particularly interesting in the current setting. To push further the idea, the problem could be formalized as establishing pair-wise dependencies between semantic objects. \n\nThe implementation of the idea makes sense to me, thought the technical description of the approach appears to me sometimes somehow vague. I wonder if this architecture could be re-used for other problems or applications, ie how much it is specific to this sole problem  at hand  or could it be useful beyond glass detection?\n\nResults on the two datasets are encouraging. Especially the detection visualizations (figure 5, 6) show very consistent glass contours. Quantitative results are less assertive (but it is ok). \n\nThe weak point of the paper is probably the sometimes vague technical description of the approach and probably insufficient discussion about technical choices.  For example in eq.(1) ,  specifying the dimensional of the input and output tensors would  be useful. A short description of the UperNet network used for final segmentation would also have been useful. A clearer description of the learning (finetuning) procedure would also be welcome. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "CihC4aFTY-L", "reviewer": "Reviewer_XdVk", "summary": "This paper proposed a new method for glass surface detection whose key insight is to incorporate semantic context information of the environment. Two new modules are proposed: (1) a Scene Aware Activation (SAA) Module that adaptively filters critical channels for spatial and semantic features and (2) a Context Correlation Attention (CCA) Module that learns the contextual correlations among objects both spatially and semantically. The paper also proposed a large-scale glass surface detection dataset named GSD-S containing detailed semantic annotations. Experiments show that the proposed method achieves state-of-the-art performance on multiple datasets including the proposed GSD-S dataset.", "questions": "- I think more analysis to justify the effectiveness of the proposed dataset would be helpful. How can one know if the proposed method is more effective? Do methods trained on the proposed dataset perform better on in-the-wild out-of-distribution data than on existing datasets? \n- Does the proposed dataset contain mostly indoor scenes? How will the method perform on images of glasses outdoor? \n- Although the glass co-occurs a lot with walls, etc, the contents that are seen through the glass can be more random and diverse. How robust is the method to different contents inside the glass area?  \n- The first sentence in the caption of Table 1 says \"GSD dataset\". Should it instead be \"GSD-S dataset\"? \n- It is unclear which curve belongs to the proposed GSD-S dataset in Figure 2 (b) as \"GSD-S\" does not exist in the legend (I guess it is \"Glass-Seseg\"?). It is also not clear what GDD and GSD stand for without citations and captions. Suggest making Figure 2 (b) clearer.  \n\nMinor:\n- Last sentence in Fig.3 caption \"multiple stage\" -> \"multiple stages'.\n- Seems that Fig 4. (a) \"Attended Features\" should instead be \"Activated Features\".\n\n\n\n", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea of exploring the semantic relationship between glass and surrounding objects to facilitate glass detection is interesting and insightful. \n+ The paper created a larger dataset for glass detection with refined glass masks and semantic annotations, which will be useful for future research on this topic.\n+ The paper conducted extensive experiments which are convincing. The method is compared to various kinds of baselines on multiple datasets, and the proposed method achieves very good results consistently. Very detailed ablation studies also justify the design choices well.\n\nWeaknesses:\n- I'm not an expert in this area, but I think the technical novelty of the proposed method is a little limited. The method builds upon many existing frameworks, and combines and adapts them to the glass detection problem, which makes the technical novelty incremental, although I think the insights behind the design are good. \n- The method section contains too many details which are overwhelming to read. For example, in Sec 4.2, Eq (1) and Fig 4 (a) give a very detailed description of each individual operation, which kind of goes into the weeds. I think it would be better to cut some unimportant details or move them to the section of implementation details. \n\nGiven the interesting idea, the good performance, and the extensive evaluation, I currently vote for borderline acceptance despite that the technical novelty is a bit limited. I would like to see reviews from other reviewers and rebuttals from the authors to make further decisions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept