Under the Shadow: Exploiting Opacity Variation for Fine-grained Shadow Detection
Authors: Xiaotian Qiao
Ke Xu
Xianglong Yang
Ruijie Dong
Xiaofang Xia
Jiangtao Cui
Venue: NeurIPS 2025 poster ()
Keywords: Shadow Detection
Shadow Removal
Scene Understanding
Primary area: applications
TL;DR: We make the first attempt to investigate fine-grained shadow detection by exploiting opacity variations, enhancing downstream applications like shadow detection, removal, and editing.

Abstract: Shadow characteristics are of great importance for scene understanding.
    Existing works mainly consider shadow regions as binary masks, often leading to imprecise detection results and suboptimal performance for scene understanding.
    We demonstrate that such an assumption oversimplifies light-object interactions in the scene, as the scene details under either hard or soft shadows remain visible to a certain degree.
    Based on this insight, we aim to reformulate the shadow detection paradigm from the opacity perspective, and introduce a new fine-grained shadow detection method.
    In particular, given an input image, we first propose a shadow opacity augmentation module to generate realistic images with varied shadow opacities.
    We then introduce a shadow feature separation module to learn the shadow position and opacity representations separately, followed by an opacity mask prediction module that fuses these representations and predicts fine-grained shadow detection results.
    In addition, we construct a new dataset with opacity-annotated shadow masks across varied scenarios.
    Extensive experiments demonstrate that our method outperforms the baselines qualitatively and quantitatively, enhancing a wide range of applications, including shadow removal, shadow editing, and 3D reconstruction.

Under the Shadow: Exploiting Opacity Variation for Fine-grained Shadow Detection
Xiaotian Qiao 1.2, Ke Xul, Xianglong Yang?, Ruijie Dong', Xiaofang Xia Jiangtao Cui
School of Computer Science and Technology. Xidian University, China Guangghou Institute of Technology, Xidian University, China
Abstract
Shadow characteristics are of great importance for scene understanding. Existing works mainly consider shadow regions as binary masks, often leading to imprecise detection results and suboptimal performance strate that such assumption oversimplifies light-object interactions in the scene, as the scene details under either hard or soft shadows remain visible certain degree. Based on this insight, aim to reformulate the shadow detection paradigm from the opacity perspective, and introduce a new fine-grained shadow detection method. In particular, given input image, we first propose shadow opacity augmentation module to generate realistic images with varied shadow opacities. We then introduce shadow feature separation module to learn the shadow position and opacity representations separately, followed by an opacity mask prediction module that fuses these representations and predicts fine-grained shadow detection results. In addition, we construct a new dataset with opacity-annotatee shadow masks across varied scenarios. Extensive experiments demonstrate that method outperforms the baselines qualitatively and quantitatively, enhancing a wide range of applications, including shadow removal, shadow editing, and 3D reconstruction.
scene understanding. We demon-
Introduction
Shadow is natural phenomenon when objects obstruct light sources either fully partially, resulting in intensity and color variations across certain regions. These shadow characteristics play vital roles for scene understanding. The presence of shadows can degrade image quality and impede numerous vision and graphics tasks, including image editing [48), object detection [33], and visual tracking. Furthermore, shadow analysis allows to interpret semantics and geometry of scene (3I], e.g., object shapes, light source positions, and object relationships. However, accurately detecting shadow characteristies (e.g., boundary, opacity, etc.) the scene can be very challenging. Shadows exhibit diverse characteristics due to scene factors like lighting conditions, surface properties, and occluder configurations, resulting in various shadow types, from sharp to diffuse.
While advanced shadow detection methods have been proposed, they often regard shadows as binary masks assume uniform opacity within shadow regions 2
We argue that such binary or simplified lighting paradigm oversimplifies light-object interactions, as the scene details under either hard or soft shadows retain varying degrees of visibility. As shown in Figure [ existing methods [54] [55] fail to capture the nuanced opacity variations and suffer from position and shape estimation artifacts, undermining downstream applications like shadow removal and editing that rely on precise shadow characteristics. A fine-grained formulation of the shadow regions remains underexplored, which vital for scene understanding.
42
Joint corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Image
BDRAR
FDRNet
Ours
1.0
Removal Detection
0.0
Figure I: Fine-ge aree-grrindd shadow detection. Given an input image (Ist row), existing methods [7 55
[54
predict binary shadow masks (Ist, 2nd and 3rd columns), yielding inaccurate shadow detection results. contrast, we propose to detect fine-grainet shadow characteristies (4th column) that capture subtle opacity variations, with benefits for the downstream applications like shadow removal [2
(2nd row).
We observe that scene details under both umbra (fully occluded) and penumbra (partially occluded) regions retain varying degrees of visibility. The variation of shadow opacity caused by factors like occluder distance or light source intensity, could offer useful scene contextual cues. Take the input image in Figure D as example. The subtle gradation of the shadow boundary suggests occluder distance, while umbra darkness and transparency reflect illumination intensity.
Inspired by the above observation, we propose a learning-based framework to predict a continuous map that indicates the position, boundary, and opacity variation of shadow regions. Our framework contains three main modules. First, given the input shadow image, we propose a Shadow Opacity Augmentation (SOA) module generate multiple images with varied shadow opacities. Each augmented image and the original image are grouped as an image pair. Second, for each image pair, we propose a Shadow Feature Separation (SFS) module disentangle position and opacity representations of the shadows separately. Third, we propose an Opacity Mask Prediction (OMP) module that fuses the learned representations and predicts continuous shadow map as the output.
To support detailed analysis of the proposed paradigm, we further construct a new Fine-grained Shadow Detection (i.e., FSD) dataset. In particular, it contains 2,653 images with opacity- shadow masks across diverse scenarios, including different light source intensities and numbers, indoor and outdoor scenes. We conduct extensive qualitative and quantitative experiments on public datasets and the proposed dataset. Results show that our approach outperforms baselines in capturing ine-grained shadow characteristics, enhancing downstream scene understanding applications.
To sum up, we make the first attempt to investigate fine- reee-gaiied shadow detection by exploiting opacity variations that are ignored by existing shadow detection and removal methods. We propose new shadow detection method by explicitly capturing shadow position and opacity characteristics, dataset with shadow opacity annotations across varied scenarios. Experimental
and construct results show that our method offers a promising shadow detection pipeline that can be applied across various applications, including shadow removal, shadow editing, and 3D reconstruction.
2 Related Work
Shadow Detection. Early research works rely on physical assumptions or heuristic cues. For instance, Lin et al. [29] adopted physical models assuming illumination invariance, while others analyzed chromatic variations in shadow regions using chromaticity 8 9 25), gradient [8] 25 [3]]
13 53 Huang al. 20 trained a shadow detector by feeding edge features into a support vector machine. Guo et al. [14] computed illumination features of segmented regions and constructed graph-based classifier by leveraging inter-region relationships. All these methods depend on handcrafted features, making it difficult to detect shadows accurately in complex scenes.
and intensity cues [39
Deep learning has significantly improved the shadow detection performance 24 Qu et al. [35]
combined fully connected networks with patch-CNN refinement. Nguyen et al. 34 stabilized training using adversarial networks, and Hu et al. [17 19 54) leveraged spatial contexts. Wang et

---

Shadow Feature Separation
Opacity Mask Prediction
Backbone
Position Fjer
Head
Input Image
1-u
Concatenation
Output
Opacity Fige
Shadow Opacity
A
Shared Weights
Augmentation
Larea
Cops
Apss
Binary Mask
Binary Mask
Eyrad
Gradient Map
Gradient Map
OpacityFan Paps
Backbone
Lrecun
Opacity Mask
Opacity Mask
Position Paor
Figure 2: The overall pipeline of our fine- gree-granned shadow detection.
al. 43 [ and Zheng et al. [51 improved accuracy via adversarial networks and distraction modeling,
respectively. Chen et al. []] incorporated edge-assisted detection. Recent works [49 36 further
addressed annotation subjectivity and enhanced features via multi-color-space training.
However, these methods tend to produce binary masks, igmoring fine- fine-grained shadow opacity. While Wang al. [45] also noted the limitations of the binary shadow mask, they only generate soft masks implicitly as intermediate results for shadow removal. In contrast, our approach models the shadow opacity variations explicitly, benefiting various downstream scene understanding applications.
Shadow Matte Estimation. Shadow matte estimation aims model the appearance of shadows in natural scenes. Chuang et al. proposed matte shadows and defined a shadow compositing formula. Wu et al. [46) optimized the shadow compositing formulation and proposed method to eliminate shadows using the shadow matte. Lu et al. [32] introduced generic framework capable of matting objects with associated shadows. Lin al. [28 further improved this to effectively matte objects and
their corresponding shadows in 3D scenes.
However, shadow mattes involve additional colors and textures, making it difficult to be utilized directly for downstream applications. In contrast, our opacity- aware shadow mask formulation allows for more flexible and controllable shadow removal and editing results.
Approach
Given an input shadow image L, our goal to predict a continuous opacity mask M, representing the shadow region. The shadow formulation is defined follows:
I. = I M.) + Ma
where I is the shadow- free image. S represents the shadow color, which could be a constant value. Ma represents the continuous shadow opacity, ranging from 0 (fully transparent) to (fuully opaque).
Figure shows the overall pipeline of our method, which contains a shadow opacity augmentation (SOA) module, a shadow feature separation (SFS) module, and an opacity mask prediction (OMP) module. We first apply opacity augmentation by adjusting the opacity of shadow regions in the input image. We then pass original and augmented images through the weights-shared backbone obtain position and opacity features. The position features Fpos and Fy should the same, while the opacity features Fopa and F should reflect the augmentation difference. Finally, both position and opacity features fused through shadow detection head output the continuous opacity map.

---

3.1 Shadow Opacity Augmentation (SOA) Module
trivial solution is to use an auto-encoder architecture to predict the opacity shadow mask directly. However, it performs poorly in handling diverse and complex shadow scenarios. To fully leverage the shadow opacity characteristics, we perform shadow augmentation on the input images by randomly augmenting the opacity of shadow regions.
Specifically, given an imput image 1, and its opacity shadow mask Ma we modify the opacity of the shadow regions by randomly sampling the parameter generate an augmented shadow image f.. Note that the SOA module only used for training, where I, and i, are grouped an image pair.
3.2 Shadow Feature Separation (SFS) Module
Given each image pair that indicates the same scene with different shadow opacities as input, we extract the position and opacity features separately. For the original image 7,, we introduce a backbone to extract the position feature Fpos and the opacity feature Fopa- Similarly, for the augmented shadow image 7. we use the weight-shared backbone to extract Fpos and Fopo correspondingly.
Since the shadow positions in the image pair remain unchanged, the position features Fpos and Fros should be identical theoretically. Mcanwhile, the opacity features Fopo and Fopo should exhibit the differences caused by the opacity augmentation parameter
3.3 Opacity Mask Prediction (OMP) Module
Given the set of the position feature Foss and the opacity feature Fopos we fuse these representations and predict the continuous shadow opacity map as the fine-grained shadow detection result. In particular, we first fuse the position feature Fpos and the opacity feature Fopa as follows:
Fg = ====Poos + +1 (1) 11)
We then employ cumulative learning strategy (52) to optimize the fused shadow feature Fs through weight selection strategy. For training epoch T and the total training epochs Tmazs p is defined as:
T
p =
T
where v is hyperparameter controlling the decay rate of μ during the training phase. Finally, the optimized shadow features are passed through detection head predict the opacity mask.
3.4 Training
For the SFS module, pos to constrain the consistency of the position features:
Lpss MAE(F pas, Pyps):
(4)
where MAE is mean-absolute-erro loss function. The shadow opacity features Fopa and Fope should be capable predicting the transparency augmentation parameter B. To achieve this, we subtract Fopo from Fopa to predict 3:
Coga MLP (Fopa FFoaa) 31,
where MLP contains a global average pooling layer and fully connected layers.
For the OMP module, to mitigate the computational difficulty arising from the varying proportions of shadow and non- non-shadow regions in each image, we employ Larea calculate the difference for shadow and non non-shadow regions and multiply by their weights:
E, L.s
Appred
A
[4
+ L,
(Apted
A451)
14
0
Carea
(6)
E,145
0]
E,14 + 0

---

1.0
Figure 3: Example images with opacity mask annotations our FSD dataset.
where A.tre represents the predicted value of the i-th pixel, and A represents the true value of the i-th pixel. The definition of Lg is:
00.5 (pred - gt )² if pred - gt <
(pred gt = (preed gt - 0.5 otherwise.
To achieve smooth prediction results that close the real shadow values, also gradient loss Lgred as follows:
Sgnd (II pred gt || + V y,edd vgt II),
where Va and V. represent the gradients in the x and y directions, respectively. We further utilize Lrecon to address the overall discrepancy as follows:
N
Lrecon
Σ
(pred - gt)².
(9)
N
In summary, we train model with the following losses:
Lall + Aupa + >uueea Curea + Agrad Egrad + Precon
(10)
where Xpos Aopa Narea Agrad, and Arecon are the weighting parameters.
Dataset
number of shadow datasets B5
Opacity Mask
Binary Mask
Object Association
Shadow Free
43 [8] have been proposed in the
Dataset
community, advancing the progress of
shadow detection and removal tasks. However, as shown in Table these datasets suffer from shortcomings like lacking scenes with weak light- ing, non-point light sources, or multi- source soft shadows. Most impor- tantly, there no publicly available shadow datasets considering opacity characteristics.
SRD 35
X
SBU 41
X
/
X
X
ISTD 43
X
/
X
/
USR 30
X
/
SOBA 44
J
DESOBA
[6
X
/
RdSOBA
38
X
7
V
/
FSD (ours)
V
Table A taxonomy of shadow datasets.
To increase the diversity of the images, we use Blender to construct a new synthetic fine-grained shadow detection (i.e., FSD) dataset with the following guidelines:
Object. We select common and diverse object categories from daily life (e.g., person, animals, shoes), and place to 5 objects with randomized positions in each scene.
Scene. The shadow images were captured from different camera angles across various indoor and outdoor scenarios, with all cameras configured 50mm focal length, automatic sensor settings, and randomized positions.
Light Source. To enhance diversity, the number, position, and intensity light sources are considered, with their positions and brightness levels being randomly sampled.

---

Input Image
BDRAR
DSC
FDRNet
Ours (binary)
Ours (opacity)
GT
Figure 4: Qualitative comparisons of shadow detection on the ISTD 43 and FSD datasets. Given input image (Ist column), we show the baseline results (2nd to 4th columns), our binary mask(5th column), our opacity mask (6th column), and ground truth (7th column), respectively.
In summary, FSD comprises 2,653 scenes with different object, scene, and light source properties. Each scene contains varied camera positions, light intensities, object numbers, and categories, resulting in 7,701 object-shadow pairs. Some examples with varying shadow opacities are shown in Figure Please refer the supplementary material for additional details.
Experiments
5.1 Experimental Setup
Implementation Details. All experiments are conducted on a single NVIDIA RTX 4090 GPU. We use EfficientNet-B3 [37] as the backbone network and ImageNet (3] for weights initialization. Our model is trained using the Adam optimizer for 20 epochs, with an initial learning rate of 5e-4, dynamically adjusted through
exponential decay strategy (decay rate of 0.7). During training, the input images are resized to 400 x 400 with a batch size of 12. The random horizontal flipping operation is further used for data augmentation. The weight parameters Xpos Aopa Aorear Agred. and Areoon are set to 1, 1, 10, 10, and 10, respectively.
Datasets. We conduct experiments both the proposed FSD dataset and the existing ISTD dataset, which contains 1,330 training images and 540 test images. Furthermore, we utilize two shadow removal datasets, i.e., ISTD+ (26 and SRD B3), for downstream applications. The ground truth shadow masks in SRD are obtained by utilizing a connectivity-basee optimization strategy.
Compared Methods. As this is the first work for fine-grained shadow detection that considers opacity, to make fair comparisons, we convert our output to binary shadow masks, and compare our method with existing shadow detection methods, including SDCM (56), FDRNet (55) MTMT
DSD (51, DSC [1]], BDRAR (54), ST-CGAN 43), scGAN [34], Macked-CNN 41.

---

Dataset
Method
BER
Shadow
Non Shad.
stacked-CNN
[41
8.60
7.69
9.23
ScGAN 34
4.70
3.22
6.18
ST-CGAN 43
3.85
2.14
5.55
BDRAR |54
2.69
0.50
4.87
ISTD
DSC 17
3.42
3.85
3.00
DSD
2.17
1.36
2.98
MTMT 1
1.72
1.36
2.08
FDRNet (55
1.55
1.22
1.88
SDCM [56
1.40
1.02
1.78
Ours
1.32
0.96
1.67
DSC 17
4.46
4.99
3,93
BDRAR 54
4.43
7.74
1.12
FSD
ST-CGAN
45
4.19
4.63
3.76
FDRNet
[55]
3.01
2.58
3.43
Ours
1.79
1.94
1.64
Table 2: Quantitative comparison of the proposed method with baselines the ISTD datasets. The best and second-best performances are marked bold and underlined.
4 and FSD
Evaluation Metries. We evaluate the quality the predicted fine-grained shadow mask from different aspects. We use the Balanced Error Rate (BER) metric to measure the position and boundary of the shadows. addition, to evaluate accuracy predicted shadow opacity, we propose a new metric termed Weighted Shadow Error (WSE). Since non-shadow areas typically dominate most images, commonly used metrics like RMSE may overlook subtle opacity variations within shadow regions. As result, WSE incorporates weighted strategy that emphasizes shadow region accuracy by computing a region-aware RMSE, thereby offering a more balanced assessment fine-grained shadow detection performance.
MN (4%
WSE
N
RRSE R.M.
(11)
EN (4 N
where At represents the alpha value of shadow for each pixel. RMSEat and RMSF mede are the root mean square error of the shadow and non-shadow regions, respectively.
5.2 Results
and FSD datasets are presented in Figure Different from existing methods that predict only binary masks for shadow regions, our approach can further capture the degree of shadow degradation in the scene accurately. Moreover, by explicitly modeling shadow opacity, our method substantially reduces both false positives and false negatives. Take the second row an example. Our method delivers fine-gra fine-grained detection results around the shadow boundaries of the tree leaves, demonstrating clear advantages in predicting precise shadow characteristics.
Qualitative Evaluation. Visual comparisons the ISTD 45
Quantitative Evaluation. We compare the performance of our method with state-of-the-ar shadow detection methods in Table As the existing shadow datasets do contain ground truth opacity annotations, we extract the brightness difference between shadowed and shadow-fres images as the ground truth value for the opacity shadow mask. Our method achieves the best or second-best BER scores among all compared methods, significantly validating the effectiveness of the shadow opacity guidance. Note that our method achieves greater performance gain on the FSD dataset than on the ISTD dataset. The primary reason is that our FSD dataset includes diverse range of challenging cases featuring soft shadow boundaries and varied opacity levels. Thus, these results highlight the strength of our method in handling diverse shadow characteristics in complex scenarios.

---

Method
BER
RMSE
WSE 4
Method
BER 4 RMSEJ
WSEL
AutoEncoder
4.01
0.0919
10.49
w/o Larad and Loga
3.27
w/o Fopa
1.97
w/o Lannd and Larea
2.34
0.0725
6.81
w/o Fpos
2.82
0.0637
5.93
w/o Lyra
2.03
0.0691
6.74
w/o SOA
2.16
0.0744
6.02
w/o Laves
1.86
0.0682
6.03
Ours (full)
1.78
0.0699
5.78
Ours (full)
1.78
0.0669
5.78
Table 3: Ablation study of different components.
Table 4: Ablation study of different losses.
5.3
Ablation Study
We evaluate the impact of different components, losses, and training strategies of our pipeline on the FSD dataset. Further analysis on different datasets are provided in the supplementary material.
Component Analysis. We first introduce naive baseline (i.e., AutoEncoder) with simple encoder and decoder architecture. We then con- duct two ablations for the SFS module. Specif- ically, we remove the position feature branch (i.e., w/o Foos) and the opacity feature branch (i.e., w/o Fopa) individually. We further remove the SOA module (i.c., w/o SOA) to examine the effect of the shadow opacity augmentation strat- egy. As shown in Table 3 these components play essential roles in resolving ambiguities in ine-grained shadow characteristics.
Method
BER 4
RMSE 4
WSE $
w/o cum.
2.18
0.1505
10.83
17 = 0.3
1.93
0.0635
6.21
77 = 0.4
1.83
0.0984
6.16
1) 0.6
1.80
0.0675
6.17
7) = 0.5
1.78
0.0669
5.78
Table 5: Ablation study of the training strategy.
Loss Analysis. The effects of different losses are examined in Table We first remove Legaat and fopa to study the importance of shadow opacity guidance. We then remove Larad and Larea separately or totally. The results show that compared using frcon alone, the integration of Lgads Laren- and Lopa enhance the fine-grained shadow detection performance significantly.
Training Analysis. We also ablate the cumulative learning strategy in Table 5 We retrain the model without using the cumulative learning strategy (i.e., w/o cum.), and use different values to control the decay rate of μ. The results show that 7 = 0.5 achieves the best performance across all metrics.
All
Shadow
Nom Non-Shadow
Methods
Input Masks
PSNRT
SSIM
MAE]
PSNRT
SSIM
MAE,
PSNR
SSIM
MAEL
DC-ShadowNNet
25.03
0.926
7.77
31.06
0.976
12.62
27.03
0.961
6.82
DeS3 [3
NA
31.38
0.958
3.94
36.49
0.989
6.56
34,70
0.972
3,40
BMNet ST
FDRNet 55
32.41
0.962
3.40
36.59
0.989
6.65
35.96
0.978
2,83
HomoFormer
47
FDRNet
35
32.41
0.953
3.51
38.84
0.991
34.58
0.966
3.17
Shudow Diffusion
FDRNet
55
34,08
0.968
3.12
40.12
0.992
5.15
36.66
0.978
2.74
Diffusiin
12
Ours
34.06
0.969
3.11
39.49
0.992
5.28
37.22
0.981
2.68
DC-ShadowNNat
22
NA
31.53
0.955
4.65
34.00
0.975
7.70
35.53
0.981
3.65
DeS3
34.11
0,968
3.56
37.91
0,986
37.45
0.984
3,03
AAM-helps-shadow
50
NA
30.72
0.952
4.79
33.94
0.979
7.44
33.85
0.981
3.74
Shadow-Former
11
DHAN 3
32.46
0.957
4.28
35.55
0.982
6.14
36.82
0.983
3.54
SRD
DHAN
DHAN
30,51
0,949
5.67
33.67
0.978
34.79
0.979
4.80
BMNe
S7
DHAN 3
31.69
0.956
4.46
35.05
0.981
6.61
36.02
0.982
3.61
Inpaint4Shadow
P7
DHAN 3
33.27
0.967
3.81
36.73
0.985
5.70
36. 70
0.985
3.27
Homoformer
17
DHAN
35.37
0.972
3.33
38.81
0.987
4.25
39.45
0.988
2.85
ShadowDDiffusio
DHAN
34.73
0.970
3.63
38.72
0.987
4.98
37.78
0.985
3,44
Diffuuson
FDRNet
55
34.23
0.972
3.50
37.31
0.985
5.04
38.61
0.986
2,90
Shadow Diffusion
Ours
34.84
0.974
3.27
42.06
0.995
3.32
36.45
0.986
3.28
Table 6: Quantitative comparisons with the shadow removal methods on ISTD+ [26] and SRD
33
datasets. NA indicates that mask input is required. The best and the second results highlighted in bold and underlined, respectively.

---

Input
Binary Mask
Opacity Mask
GT
Figure 5: Shadow removal results. Given the input image (Ist row), show shadow removal results from ShadowDiffushe ground truth (4th row).
12
with binary masks (2nd row), our opacity masks (3rd row), and the
Input
Object-Shadoo Assoication
Background Editing
Foreground Editing
object mask
opacity shadow mask
Figure 6: Shadow editing results. Given the input image (1st column), extract the opacity shadow mask with the corresponding object (2nd column). Multiple editing operations can be further conducted, and corresponding fine-grained shadow characteristies will be changed adaptively. For example, replacing the background (3rd column) or changing the foreground object (4th column).
5.4 Applications
Benefiting from model for fine-gras fine-grained shadow detection, explore three downstream applica- tions here. Please refer to the supplementary material for additional details and results.
Shadow Removal. The opacity shadow mask provides rich information by explicitly encoding pixel-wise attenuation coefficients, enabling precise characterization of shadow degradation levels on the background. To validate its practical utility, we reformulate the training pipeline of Sif fusion [1) by replacing binary shadow mask input with our continuous opacity map. Such an adaptation can dynamically adjust denoising strength based on shadow opacity and preserve natural illumination transitions at shadow boundaries through opacity-guided attention modulation.
Quantitative comparisons against existing shadow removal methods are presented in Table The results show that the shadow removal model trained with the opacity shadow mask achieves substantial improvements. Note that our method obtains greater performance gains shadow regions than on non-shadow regions. The primary reason is that our method specifically designed handle shadow opacity, while non-shadow regions dominate most images, masking shadow-specific improvements. For example, on the SRD dataset (35) that contains more complex scenes and soft shadows, PSNR value for shadow regions shows substantial enhancement, validating the effectiveness of the opacity guidance strategy. Qualitative comparisons in 5 highlight our advantages in shadow removal visual quality, particularly for shadows with light-color blurred boundaries.

---

Input
Ours
Input
Ours
Figure 7: Failure cases. Our model may fail on images with complex shadow interactions.
Shadow Editing. Shadows often complicate image editing by hindering consistent manipulation of objects and their associated shading. However, by employing the proposed opacity shadow mask, we can simplify such a process by enabling synchronized adjustments of shadows and objects. As shown in Figure the opacity shadow mask be treated detachable shadow layer and edited alongside the object. Specifically, given an input image, we utilize the proposed model to extract an opacity shadow mask with the corresponding object. Multiple editing operations can be further conducted, and corresponding fine-grained shadow characteristics will be changed adaptively. For example, we can replace the background in the input image with new one, or modify the position and size of the corresponding object in the image.
3D reconstruction. Fine-grained shadow opacity variations (e.g.. penumbra width and transparency gradients) provide critical cues about scene geometry, such as occluder distance and light source intensity, which are highly valuable for 3D reconstruction. Following the shadow-driven neural rendering framework [40]. we apply our continuous opacity maps, rather than binary masks from FDRNet (55) used before, for the 3D reconstruction application. Our method achieved RMSE values of 0.00957 and 0.00531 for the Bunny and CUBE scenes, respectively, significantly lower than FDRNet (55)'s 's 0.01074 and 0.00777. These results demonstrate that leveraging mee-grained shadow properties can enhance geometric reconstruction accuracy, particularly under texture-less complex lighting conditions.
Discussion. The transparency information embedded in the opacity shadow mask provides ad- ditional benefits for scene understanding. First, it enables precise modeling of shadow-object interactions by quantifying the opacity gradient at shadow boundaries, which can guide existing shadow generation models to synthesize more realistic soft shadows with adaptive opacity. The pixel-wise opacity map serves as a physical prior for illumination decomposition, allowing joint optimization of shadow removal and scene relighting tasks. In dynamic scenes with moving objects, the temporal consistency of opacity values can be leveraged enhance video shadow stabilization algorithms, reducing flickering artifacts during editing.
Although model works well in various shadow scenarios, it may fail in some challenging cases. As illustrated in F it would be difficult for our model to predict accurate shadow opacity when multiple shadow interactions exist in complex scenarios. A potential solution this problem is to adopt the instance shadow strategy to distinguish specific object-shadow associations. As future work, we would like explore more shadow characteristies boost the scene understanding ability.
Conclusion
In this paper, we make the first attempt to investigate fine-grained shadow detection by exploiting shadow opacity characteristics in the scene, To this end, we propose learning- based model that explicitly captures shadow position and opacity variations. In addition, we construct the first fine- grained shadow detection (FSD) dataset with opacity annotations across varied scenarios. Extensive qualitative and quantitative results show that our model can predict fine-grained shadow characteristics, achieving superior performance over the baselines and enhancing wide range of applications, including shadow removal, shadow editing, and 3D reconstruction.
10

---

Acknowledgments
This work was partially supported by Guangdong Basic and Applied Basic Research Foun- dation (No.2022A1515110740) National Natural Science Foundation of China NN..6230335, No.62372360, No.62372352 CCF-ALIMAMA TECH Kangaroo Fund (NO.CCF-ALIMAM) OF 2025007), Youth Elite Scientist Sponsorship Program by China Association for Science and Tech- nology (No. YESS20220610), and Innovation Capability Support Program of Shaanxi Province (No.20242C-KIXX--021))
References
[1] Zhihao Chen, Lei Zhu, Liang Wan, Song Wang, Wei Feng, and Pheng-Ann Heng. multi-task mean teacher for superviue shadow detection. In CVPR, 2020.
[2] Yung-Yu Yu Chuang, Dan B Goldman, Brian Curless, David H Salesin, and Richard Szeliski. Shadow matting and compositing. In SIGGRAPHI, 2003.
[3] Xiaodong Cun, Chi-Man Pun, and Cheng Shi. Towards ghost-frre shadow removal via dual hierarchical aggregation network and shadow matting gan. In AAAI, 2020.
[4] Lennart Demes. ambienteg ecO textures, hdris and models, July 2024. URL https://
ambientcg.com com/
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[6] Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao. Argan: Attentive recurrent generative adversarial network for shadow detection and removal. In ICCV,
[7] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items, 2022. URL https://///xi org/ ab///2///2222... 11918
[8] Graham D Finlayson, Steven D Hordley, Cheng Lu, and Mark S Drew. On the removal of shadows from images. TPAMI, 28( (1),5699,,,, 2005.
[9] Graham D Finlayson, Mark S Drew, and Cheng Lu. Entropy minimization for shadow removal. IJCV, 2009.
[10] Maciej Gryka, Michael Terry, and Gabriel J Brostow. Learning remove soft shadows. ACM TOG, 2015.
[11] Langing Guo, Siyu Huang, Ding Liu, Hao Cheng, and Bihan Wen. Shadowformer: Global context helps image shadow removal. In AAAI, 2023.
[12] Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang, Hanspeter Pfister, and Bihan Wen. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. In CVPR, 2023.
[13] Ruiqi Guo, Qieyun Dai, and Derek Hoiem. Single-image shadow detection and removal using paired regions. In CVPR, 2011.
[14] Ruiqi Guo, Qieyun Dai, and Derek Hoiem. Paired regions for shadow detection and removal. IEEE TPAMI, 2012.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.
[16] Yan Hong, Li Niu, and Jianfu Zhang. Shadow generation for composite image in real-world scenes. In AAAI, 2022.
[17] Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, and Pheng- Heng. Direction-aware spatial context features for shadow detection and removal. IEEE TPAMI, 2019.
11

---

[18] Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, and Pheng-Am Ann Heng, Mask-shadowwann Learning to remove shadows from unpaired data. In ICCV, 2019.
[19] Xiaowei Hu, Tianyu Wang. Chi- Shi-Wing Fu, Yitong Jiang, Qiong Wang, and Pheng- Ann Heng. Revisiting shadow detection: A new benchmark dataset for complex world. IEEE TIP, 2021.
[20] Xiang Huang, Gang Hua, Jack Tumblin, and Lance Williams. What characterizes a shadow boundary under the sun and sky? In ICCV, 2011.
[21] Sketchfab Inc. Sketchfab the best 3d viewer on the web, July 2024. //sketchfabccom/
URL https
[22] Yeying Jin, Aashish Sharma, and Robby Tan. De-shadownet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network. In ICCV, 2021.
[23] Yeying Jin, Wei Ye, Wenhan Yang. Yuan Yuan, and Robby T Tan. Des3: Adaptive attention- driven self and soft shadow removal using vit similarity. In AAAI, 2024.
[24] Salman H Khan, Mohammed Bennamoun, Ferdous Sohel, and Roberto Togneri. Automatic shadow detection and removal from a single image. IEEE TPAMI, 2015.
[25] Jean-François Lalonde, Alexei A Efros, and Srinivasa G Narasimhan. Detecting ground shadows in outdoor consumer photographs. In ECCV, 2010.
[26] Hieu Le and Dimitris Samaras. Shadow removal via shadow image decomposition. In ICCV, 2019.
[27] Xiaoguang Li, Qing Guo, Rabab Abdelfattah, Di Lin, Wei Feng, Ivor Tsang, and Song Wang. Leveraging inpainting for single-image shadow removal. In ICCV, 2023.
[28] Geng Lin, Chen Gao, Jia-Bin Huang, Changil Kim, Yipeng Wang. Matthias Zwicker, and Ayush Saraf. Omnimatterf: Robust omnimatte with 3d background modeling. In ICCV, 2023.
[29] Yun-Hsuan Lin, Wen-Chin Chen, and Yung- Yu Chuang. Bedsr-net: A deep shadow removal network from a single document image. In CVPR, 2020.
[30] Daquan Liu, Chengjiang Long, Hongpan Zhang, Hamning Yu, Xinzhi Dong, and Chunxia Xiao. Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In CVPR, 2020.
[31] Ruoshi Liu, Sachit Menon, Chengzhi Mao, Dennis Park, Simon Stent, and Carl Vondrick. What you can reconstruct from shadow. In CVPR, 2023.
[32] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, and Michael Rubinstein. Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
[33] Ivana Mikic, Pamela Cosman, Greg T Kogut, and Mohan M Trivedi. Moving shadow and object detection in traffic scenes. In ICPR, 2000.
[34] Vu Vu Nguyen, Tomas F Yago Vicente, Maozheng Zhao, Minh Hoai, and Dimitris Samaras. Shadow detection with conditional generative adversarial networks. In ICCV, 2017.
[35] Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson Lau, Deshadownet: A multi-context embedding deep network for shadow removal. In CVPR, 2017.
[36] Jiayu Sun, Ke Xu, Youwei Pang, Lihe Zhang, Huchuan Lu, Gerhard Hancke, and Rynson Lau. Adaptive illumination mapping for shadow detection in raw images. In ICCV, 2023.
[37] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[38] Xinhao Tao, Junyan Cao. Yan Hong, and Li Niu. Shadow generation with decomposed mask prediction and attentive shadow filling. In AAAI, 2024.
[39] Jiandong Tian, Xiaojun Qi, Liangqiong Qu, and Yandong Tang. New spectrum ratio properties and features for shadow detection. Pattern Recognition, 51, 2016.
12

---

[40] Kushagra Tiwary, Tzofi Klinghoffer, and Ramesh Raskar. Towards learning neural representa- tions from shadows. In ECCV, 2022.
[41] Tomás Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras. Large-scale training of shadow detectors with noisily-annotated shadow examples. In ECCV, 2016.
[42] Hongqiu Wang, Wei Wang, Haipeng Zhou, Huihui Xu, Shaozhi Wu, and Lei Zhu. Language- driven interactive shadow detection. In ACM MM, 2024.
[43] Jifeng Wang, Xiang Li, and Jian Yang. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In CVPR, 2018.
[44] Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng-Am Heng, and Chi-Wing Fu, Instance shadow detection. In CVPR, 2020.
[45] Xinrui Wang, Lanqing Guo, Xiyu Wang, Siyu Huang, and Bihan Wen. Softshadow: Leveraging penumbra-aware soft masks for shadow removal. In CVPR, 2024.
[46] Tai-Pang Wu, Chi-Keung Tang, Michael S Brown, and Heung- Yeung Shum. Natural shadow matting. ACM TOG, 2007.
[47] Jie Xiao, Xueyang Fu, Yurui Zhu, Dong Li, Jie Huang, Kai Zhu, and Zheng-Jun Zha. Homo- former: Homogenized transformer for image shadow removal. In CVPR, 2024.
[48] Jiamin Xu, Zelong Li, Yuxin Zheng, Chenyu Huang, Renshu Gu, Weiwei Xu, and Gang Xu. Omnisr: Shadow removal under direct and indirect lighting. In AAAI, 2025.
[49] Han Yang, Tianyu Wang, Xiaowei Hu, and Chi-Wing Fu, Silt: Shadow-aware iterative label tuning for learning to detect shadows from noisy labels. In ICCV, 2023.
[50] Xiaofeng Zhang, Chaochen Gu, and Shanying Zhu. sam-helps-shadow: when segment anything model meet shadow removal. arXiv:2306.066 13, 2023.
[51] Quanlong Zheng, Xiaotian Qiao, Ying Cao, and Rynson Lau. Distraction-aware shadow detection, CVPR, 2019.
[52] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral- branch network with cumulative learning for long-tailed visual recognition. In CVPR, 2020.
[53] Jiejie Zhu, Kegan GG Samuel, Syed Z Masood, and Marshall F Tappen. Learning to recognize shadows monochromatic natural images. CVPR, 2010.
[54] Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng. Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In ECCV, 2018
[55] Lei Zhu, Ke Xu, Zhanghan Ke, and Rynson Lau. Mitigating intensity bias in shadow detection via feature decomposition and reweighting. In ICCV, 2021.
[56] Yurui Zhu, Xueyang Fu, Chengzhi Cao. Xi Wang. Qibin Sun, and Zheng-Jun Zha. Single image shadow detection via complementary mechanism. In ACM MM, 2022.
[57] Yurui Zhu, Jie Huang. Xueyang Fu, Feng Zhao, Qibin Sun, and Zheng-Jun Zha. Bijective mapping network for shadow removal. In CVPR, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide detailed and accurate description of the paper's contributions and scope in both the abstract and the concluding paragraph of the references.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are mentioned in the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the network architectures and hyperparameters required for the experiments in both the paper and supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will release the dataset and code to the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceec guwde//dde/eeee
ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of all the training and test details, including backbones, hyperparameters, and optimizers.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
Our experiment does not contain statistical analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the hardware specifications required for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics. It addresses potential societal impacts, avoids harmful applications, and ensures fairness and transparency in methodologies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: No datasets with potential privacy and security risks appeared in the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-party assets (code, data, models) are properly credited with their original sources cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This study does not involve any crowdsourced experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
20

---

Appendix
This Appendix provides extended technical details and experimental analyses to support the main paper. We elaborate on the construction the FSD dataset across diverse scenarios: indoor/outdoor environments, single/multiple light sources, shadow overlaps, and varying illumination intensities. We describe the physics-based rendering pipeline in Blender Cycles, leveraging path tracing simulate realistic shadow interactions governed by radiometric principles. We elaborate in detail on the calculation process of the ground truth values for the opacity shadow mask and supplement the qualitative experimental results of shadow detection. Additionally, we further analyze and present the qualitative experimental results of shadow removal using the opacity shadow mask.
A
FSD Dataset
A.1 Additional details
When constructing the dataset, we considered a variety of scenarios encompassing indoor and outdoor environments, varying from single multiple light sources, weak strong lighting conditions, and situations with overlapping shadows. The dataset comprises shadow-fres images, instance masks, binary shadow masks, self-shadow masks, and opacity shadow masks.
To prevent interference within the dataset, we manually exclude cases where model consists of multiple separate parts. Human models are selected from free models on Sketchfab [1] based on quality, resulting manual selection of 59 models. In outdoor settings, Ambient CG 4 and Poly Haven. To ensure data quality, scenes with extremely weak lighting or existing shadows excluded, leading final selection of 26 scenes. Below are examples of scenes included in the dataset.
For indoor object models, we source 1030 models from Google Scanned Objects 77.
incorporate HDRI textures from
Instance Mask
binary mask
self shadow
opacity mask
Origin image
Shadow free
Instance Mask
binary mask
self shadow
opacity mask
Origin image
Shadow free Figure 8: Indoor, single light
Indoor, Single Light Scenario. In indoor settings, a cubic space serves as the base for constructing the scenes. total of 428 floor textures from Poly Haven are utilized adorn the indoor space. Subsequently, 1-5 objects are randomly positioned near the world coordinate origin, with their sizes randomly adjusted to diversify the dataset. The placement strategy for objects this single-light indoor scenario (refer to Figure. involves distributing the xyxy coordinates within a narrow strip area to simulate common real-world object arrangements. For this scenario, single light source (such as parallel light or SUN in Blender) is employed to replicate indoor reflection effects, aligning with typical indoor scene configurations. The variability in light sources encompasses factors like intensity, blur level (indicating the softness of shadows), light direction, position, and other parameters. Regarding the camera settings in both indoor and outdoor scenes, the position and angle of the camera randomized, excluding specifications such as focal length and sensor size.
21

---

binary mask (1) binary mask (2)
opacity mask (I)opacity mask (2)
Origin image
Shadow free
binary mask (1) binary mask (2)
opacity mask (1) opacity mask (2)
Origin image
Shadow free Figure 9: Indoor, multiple lights
Indoor, multiple lights. In more intricate scenarios (refer to Figure, 9). the presence of multiple light sources may give rise to various shadow phenomena. This particular scenario has received limited attention in prior research. To enrich the dataset's scope, have included this scenario this setting, objects are placed with completely random xyxy
to expand its comprehensiveness. coordinates. As for lighting, have implemented strategy randomly generate lighting effects, encompassing point light (POINT in Blender), sunlight (parallel light, SUN), spotlight (SPOT), and area light (AREA). The selection of light types and quantities is randomized, and the attributes of the lights are determined in similar random fashion as previously described.
Shadow overlaps. When multiple light sources are utilized, shadow intersections occur (see Figure. 9). phenomenon that cannot adequately elucidated by a binary shadow mask alone. Our dataset intentionally includes scenarios featuring shadow intersections. The distinction between this and multiple-lig scenarios lies in the placement strategy of objects and lights. Specifically, two objects positioned to form specific angle, after which two point light sources are placed in the directions of these objects (from the origin to each object) to generate intersecting shadows.
Instance Mask
binary mask
self shadow
opacity mask
Origin image
Shadow free
HOURE
HOUSER
CARDDS
Instance Mask
binary mask
self shadow
opacity mask
Origin image
Shadow free Figure 10: Outdoor, weak light
Outdoor, weak light. For all outdoor scenes, we create a hemisphere and utilize HDRI textures to establish the environment, thereby integrating lighting details from the textures organically. In instances of low lighting intensity (refer to Figure. [0), shadows exhibit a blurred and ghosted appearance, rendering binary shadow masks less efficient for shadow elimination. Given that the lighting details are predetermined by the texture, the only elements subject to randomness are camera angles and the placement of objects.
22

---

Instance Mask
binary mask
self shadow opacity mask
Origin image
Shadow free
Instance Mask
binary mask
self shadow
opacity mask
Origin image
Shadow free Figure 11: Outdoor, strong light
Outdoor, strong light. In outdoor environments characterized by intense lighting conditions (see Figure. [T). representative of real-world settings, shadows tend to be distinct and deep. Our dataset comprises a substantial quantity of such scenes. The primary contrast between these two outdoor setups lies in the HDRI textures employed.
Once the scene set up. we employ keyframe animation to produce the initial image, a shadow-fre rendition, and all desired annotations for each object. Blender sequentially generates the animation frame by frame, which later subjected to post-processing. Due to inherent noise in the original opacity shadow mask from Blender Cycles, we begin by applying noise reduction techniques. Subsequently, we derive the binary shadow mask based on opacity shadow mask. Following the processing of all samples, compile JSON file document datasets in the format specified by SOBA 44). Comprehensive details regarding the scene setup will be released.
B
Additional Shadow Detection Results
B.1 Shadow Rendering Process
Blender Cycles uses the path tracing algorithm, which is based the following equation:
Lo wo) (p. / (p, (.) wi)n widdu
(12)
It uses radiometry to describe lighting more accurately, resulting in more realistic images. The term L,(p, w;)n wi describes the radiance of the light coming from solid angle w; the environment, contributing to the radiance at point p. Thus, Jg L, wi ) widw; represents the contribution of all light rays the hemisphere 22, i.e. all light the environment. The term f. determines how much light will be reflected. Although with numerous BSDF, BRDF models available, they only determine the material of the object. The determinant of shadows object's is self-emission.
term L; .he term L. represents the
The ground truth opacity shadow mask agr is calculated from the shadow image S and the shadow- free image F. The specific steps are as follows: first, convert Sand F to the YCbCr color space and extract the Y channel, then compute the ratio of Ys to Yp.apply a low-pass filter remove noise, and finally obtain dgg through thresholding.
Ys YF
agt max(t, f
(13)
Here, Ys and YF represent the Y channels of the shadow image and the shadow-free image in the YCbCr color space, respectively. denotes low-pass filtering operation with a 0.5, and the threshold, which is set to 0.1 in this paper.
23

---

Shadow Image
Shadow Free
Opacity
Figure 12: The figure demonstrates the shadow image, the shadow-free image, and the extracted opacity shadow mask. Compared to the binary mask, the opacity shadow mask not only indicates the location of the shadows but also reveals the intensity of the shadows and the degree of background.
Input Image
BDRAR
DSC
FDRNet
Ours (binary)
Ours (opacity)
GT
Figure 13:
The figure illustrates qualitative comparison results between and the state-of-the-an methods on the SRD dataset [35), our method exhibits significant advantages in visual quality.
Our method outperforms the ofte-of-the-art approaches across multiple examples,our method demonstrates exceptional perfor- mance in fine-grained shadow detection. For instance, in rows 2 and 4, our method can predict the transparency changes of shadows, while in rows and 5, it achieves more precise detection of soft shadow boundaries. These results further validate the reliability of the shadow detection approach guided by shadow opacity.
Figure 13 presents additional experimental results on the SRD dataset 35
B.2 Ablation Study
Qualitative comparison. The ablation results Figure 14 validate our design: the opacity mask enables robust shadow positioning and transparency estimation, complemented by Lynd for regulating gradient smoothness and Carea for boosting accuracy in extreme scenarios.
24

---

Input
GT binary mask
GT alpha mask
w/o Leyred Lopa
Lirrea
w/o grad
Lurea
Ours (full model)
Figure 14: Qualitative comparison results of different ablated versions of our model.
Quantitative Evaluation. We addi- tionally conducted an ablation study the ISTD
Method
BER +
Shadow
Non Shad..
of our loss functions dataset [43]. As it lacks ground truth annotations for shadow opacity, employed the Balanced Error Rate (BER) metric for evaluation. The re- sults, presented in the table below, demonstrate that full model out- performs all ablated versions, validating the effectiveness of our design choices.
w/o
Legrat and feon
1.73
0.92
2.53
Legrat and Larea
1.55
2.21
0.88
w/o Ligral
1.50
2.25
0.75
w/o Carea
1.46
2.15
0.77
Ours (full)
1.32
1.67
0.96
Table 7: Ablation study on ISTD.
C Additional Shadow Removal Results
Input
Binary Mask
Opacity Mask
GT
Figure 15: We compared the results of using binary masks and opacity masks for shadow removal in training the ShadowDiffusioni model on SRD Dataset (26). The results indicate that using opacity masks instead of binary masks for annotation leads to better visual effects.
In our shadow removal experiments, we improved upon the oftte-of-thearr ShadowDiffusion method. Specifically, we replaced the binary shadow mask with the detection results from our method for training, while keeping other hyperparameters original ShadowDiffusion The experiments were conducted on an RTX 4090 GPU, with the training epochs set to 1000. We employed the Adam optimizer (with momentum parameters (0.9, 0.999)) and initial learning rate of 3 x 10-5, The model weights were initialized using the Kaiming
optimization strategies consistent with
25

---

Input
Opacity Mask Binary Mask
GT
Figure 16: Results of different inputs on the LRSS [10 dataset using SifdowDDiffssion [12
initialization technique [115, and an exponential moving average (EMA) strategy with decay rate of 0.9999 was applied across all experiments.
Figure 15 showcases the performance superiority of the model trained using opacity shadow masks compared to the model trained with binary shadow masks across various test images. The qualitative analysis demonstrates that, under the same training conditions, the utilization of opacity shadow masks enables the model to effectively eliminate shadows while maintaining more detailed information about the underlying scene. Particularly in complex scenarios with blurred shadow boundaries, the model trained with opacity shadow masks shows notably enhanced performance.
To validate the effectiivnees of the opacity shadow mask, we conducted supplementary experiments on the LRSS soft shadow dataset [IO], which contains 46 pairs of shadow and shadow-frec images. The experiments adopted strategy of training on the SRD dataset [26] and evaluating the LRSS dataset [0]. As illustrated Figure. [6] the opacity shadow mask demonstrated significant advantages in restoring soft shadow edges and background textures, thereby fully validating its effectiveness.
26


Reviews:
[{"review_id": "9YxbXECbsl", "reviewer": "Reviewer_Xc6q", "summary": "This paper introduces a novel approach to fine-grained shadow detection by explicitly modeling opacity variations in shadow regions. To train the model, the authors propose the first dataset with pixel-wise shadow opacity annotations across diverse scenarios. Experiments demonstrate that the proposed method can improve the shadow detection accuracy and benefit shadow removal.", "questions": "Since the shadow mask is not the final output, I would like to see more visual comparisons on downstream tasks such as shadow removal. In addition, the paper does not include experiments evaluating the impact of the $L_{opa}$ and $L_{pos}$ losses in the Shadow Feature Separation module.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- 1. This paper is well-written and easy to follow.\n- 2. This paper introduces the first dataset with opacity-annotated shadow masks, filling the gap in the field.\n\nWeakness:\n- 1. Since shadow removal is the primary downstream application of shadow detection, more visual comparisons should be provided between the fine-tuned ShadowDiffusion model using the proposed method and other shadow removal methods using alternative shadow detection approaches.\n- 2. In Table 4, the improvement in the \"All\" region appears relatively minor. An explanation for this would be helpful.\n- 3. In Table 3, the authors should clarify which dataset was used for testing. I hope to know each design's performance on ISTD and FSD dataset.\n - 4. Ablation study for the $L_{opa}$ and $L_{pos}$ should be provided.", "was_revised": true, "final_justification": "The authors' rebuttal has solved most of my concerns, and I will keep my rating as Borderline Accept.", "rebuttal": "Thanks a lot for your time and constructive feedback.\nWe are encouraged by your recognition of our motivation with good presentation and contribution.\nGiven the opportunity, we will incorporate all suggestions with details in the revision.\nBelow, we address all concerns raised.\nWe will be very happy to clarify further concerns (if any).\n\n**Q1: More visual comparisons should be provided between the fine-tuned ShadowDiffusion model using the proposed method and other shadow removal methods using alternative shadow detection approaches.**\n\nA1: As suggested, we conduct more visual comparisons between different methods (ShadowDiffusion + Our Mask, ShadowDiffusion + DHAN, ShadowDiffusion + FDRNet, HomoFormer + FDRNet).\nHowever, since visual comparison is unavailable in the rebuttal, we also conduct a user study to show visual quality comparisons from the human perception perspective.\nWe will show and discuss these visual comparisons in the paper and the supplementary material.\n\nFor the user study, 20 participants are involved, and 10 images are randomly sampled from the SRD dataset.\nEach time, participants are shown an input shadow image with 4 shadow removal results by the above methods.\nParticipants are asked to vote for the best shadow removal result.\n\nIn the end, the results of ShadowDiffusion + Our Mask are ranked the best in 55% of the votes, HomoFormer + FDRNet in 25% of the votes, ShadowDiffusion + FDRNet in 10% of the votes, and ShadowDiffusion + DHAN in 10% of the votes, which confirms the superior visual quality of our approach.\n\n**Q2: In Table 4, the improvement in the \"All\" region appears relatively minor. An explanation for this would be helpful.**\n\nA2: We appreciate the valuable suggestion and will add detailed discussions in the revised paper. \n\nThe results in the \"All\" region are obtained by combining shadow and non-shadow areas, where non-shadow regions dominate in most images, reducing the visibility of shadow-specific improvements.\nAs our method is specifically designed to handle the shadow opacity, we can see that our method achieves higher performance improvement for the shadow region in Table 4.\n\nAdditionally, our opacity masks achieve consistent performance (best or second best) across datasets (ISTD, SRD), unlike existing methods like ShadowDiffusion or HomoFormer, in which the performance varies a lot due to dataset-specific shadow characteristics. The user study results mentioned in Q1 further demonstrate the effectiveness of our method in improving shadow removal visual quality. \n\n**Q3: In Table 3, the authors should clarify which dataset was used for testing. I hope to know each design's performance on ISTD and FSD dataset.**\n\nA3: Sorry for the confusion.\nWe use the FSD dataset for testing in Table 3.\nSince ISTD does not provide the ground truth for the shadow opacity, we use the BER metric for evaluation.\nAs shown in the table below, our full model outperforms ablated versions, validating the effectiveness of our design choices.\n\n| Method                                | BER ↓ | Shadow ↓ | Non Shad. ↓ |\n|---------------------------------------|-------|----------|-------------|\n| w/o opacity mask                      | 1.73  | **0.92**     | 2.53        |\n| w/o $L_{\\text{grad}}$ and $L_{\\text{area}}$ | 1.55  | 2.21     | 0.88        |\n| w/o $L_{\\text{grad}}$       | 1.50  | 2.25     | **0.75**        |\n| w/o $L_{\\text{area}}$       | 1.46  | 2.15     | 0.77        |\n| **Ours (full)**                       | **1.32** | 1.67 | 0.96    |\n\n**Q4: Ablation study for the $L_{pos}$ and $L_{opa}$ should be provided.**\n\nA4: As suggested, we conduct additional ablation studies for $L_{pos}$ and $L_{opa}$ on FSD as follows. \nFor the BER metric, the results of w/o $L_{opa}$, w/o $L_{pos}$, and ours (full) are 1.97, 2.82, and 1.78, respectively.\nFor the WSE metric, the results of w/o $L_{pos}$, and ours (full) are 5.93 and 5.78, respectively.\nThese results confirm that both losses are critical for fine-grained shadow detection, with $L_{opa}$ enhancing shadow opacity modeling and $L_{pos}$ ensuring accurate shadow position.\n\n**Q5: I would like to see more visual comparisons on downstream tasks such as shadow removal.**\n\nA5: As mentioned in Q1, we will add more visual comparisons for downstream tasks in the revised paper.\nIn addition, to further demonstrate the usefulness of our opacity masks in downstream applications, we explore another task, i.e., 3D reconstruction, which is suggested by Reviewer DKcE.\n\nWe retrain the 3D reconstruction model [1] with the shadow regions predicted by either FDRNet (binary mask) or our method (continuous opacity mask).\nFollowing the evaluation protocol in [1], we quantitatively analyze the quality of the reconstructed meshes by different methods.\nTake the Bunny scene as an example.\nThe RMSE value with FDRNet between the predicted point cloud and the ground truth point cloud is 0.01206, while the RMSE value with our opacity mask is 0.00928.\nThese results further demonstrate the effectiveness of our method in enhancing scene understanding capabilities.\n\n**References:**\n\n[1] Tiwary et al., Towards learning neural representations from shadows, ECCV 2022."}, {"review_id": "It7VwXCuxK", "reviewer": "Reviewer_DKcE", "summary": "The authors propose a new shadow detection task and corresponding method to solve it - fine-grained shadow detection. Whereas existing methods mainly predict binary shadow masks, this task consists of creating an opacity mask for every pixel in an input image, which allows opacity variation in shadows (due to full vs partial occlusion) to be captured. The authors argue that this finer-grained detection can improve performance for downstream tasks, such as shadow removal and editing. The proposed method leverages a simulated dataset proposed by the authors and trains a network to learn opacity masks from an input image and augmentations of the GT shadow masks (during training). The authors report results on both shadow detection and downstream applications of shadow removal and editing.", "questions": "My main concerns are weaknesses (1) and (2) above, informing my current recommendation.\n\nIn addition, a few other smaller comments:\n- You may consider adding other tasks to line 22 and the introduction - in particular 3D reconstruction from shadows [1] [2], since, at least to me, it seems much more clear that this application would benefit from fine-grained shadow detection (since weaker shadows are a function of partial occlusion, thus reflecting information about the underlying scene geometry). This could partially address my first concern. Getting results for 3D reconstruction from shadows that show more substantive improvements would address my concern further, but I acknowledge this may be challenging in the timeline of the rebuttal.\n- Organization: I’d suggest moving the dataset section to before the experiments\n- Further motivation why the new metric (weighted shadow error) is needed should be added\n- Should equation 5 have an absolute value (i.e. L1 loss)?\n- L188 typo: covert —> convert\n\n[1] Towards Learning Neural Representations from Shadows. https://doi.org/10.1007/978-3-031-19827-4_18\n[2] What You Can Reconstruct from a Shadow. https://doi.org/10.1109/CVPR52729.2023.01636", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The proposed task is novel - and, in general, I believe that modeling more of the underlying physics of light, as attempted in this work, can be beneficial for computer vision pipelines. Moreover, the proposed solution is simple yet effective.\n- Shadow detection results support the efficacy of the method.\n\n**Weaknesses**\n- The results for downstream applications show marginal improvements (Table 4). In addition, providing intuition for why “a fine-grained formulation of shadow regions” is “vital for scene understanding” (L36) would significantly improve the motivation for the paper. This notion is vaguely referred to again on L41 (...“could offer useful scene contextual clues”), but making this more concrete would be helpful. See my notes below - 3D reconstruction from shadows might be one more intuitive example for why fine-grained shadows can be helpful.\n- I'd suggest the authors provide a baseline of the “trivial” solution described in 4.1 to substantiate the claim that using an auto encoder is less effective than the proposed technique. In addition, ablating the impact of the shadow augmentation in training would be helpful since this seems to be one of the main technical contributions.\n- The authors fail to mention in the main text that the proposed dataset is simulated. Moreover, L106 is misleading as it states the dataset is “captured”, when, in fact, from my understanding based on the supplement, it is actually rendered.", "was_revised": true, "final_justification": "While I still have some doubts about the significance of impact on downstream tasks, based on the strong rebuttal, I'd like to raise my score to borderline accept since major concerns about motivation and missing comparisons/ablations have been mitigated.", "rebuttal": "Thanks a lot for your detailed review and insightful suggestions.\nWe are encouraged by your recognition of the novelty of our new problem and the effectiveness of our approach.\nGiven the opportunity, we will incorporate all suggestions with details in the revision.\nBelow, we address all concerns raised.\nWe will be very happy to clarify further concerns (if any).\n\n**Q1: The results for downstream applications show marginal improvements (Table 4).**\n\nA1: The results in the \"All\" region are obtained by combining shadow and non-shadow areas, where non-shadow regions dominate in most images, reducing the visibility of shadow-specific improvements.\nAs our method is specifically designed to handle the shadow opacity, we can see that our method achieves higher performance improvement for the shadow region in Table 4.\n\nAdditionally, our opacity masks achieve consistent performance (best or second best) across datasets (ISTD, SRD), unlike existing methods like ShadowDiffusion or HomoFormer, in which the performance varies a lot due to dataset-specific shadow characteristics.\n\n**Q2: Providing intuition for why “a fine-grained formulation of shadow regions” is “vital for scene understanding” (L36) would significantly improve the motivation for the paper...3D reconstruction from shadows might be one more intuitive example...**\n\nA2: We totally agree with the reviewer that additional applications like 3D reconstruction from shadows can significantly improve our motivation. (see discussions with [2] in L19-L24).\nFine-grained shadow properties, such as penumbra width and opacity variations, provide cues about occluder distance and light source intensity, which are critical for scene geometry understanding.\n\nAs suggested, we conduct additional experiments for the 3D reconstruction task.\nTake the Bunny scene as an example.\nWe retrain the 3D reconstruction model [1] with the shadow regions predicted by either FDRNet (binary mask) or our method (continuous opacity mask).\nFollowing the evaluation protocol in [1], we quantitatively analyze the quality of the reconstructed meshes by different methods.\nThe RMSE value of FDRNet between the predicted point cloud and the ground truth point cloud is 0.01206, while the RMSE value of our method is 0.00928.\nThese results further demonstrate the effectiveness of our method in enhancing scene understanding capabilities.\n\nNote that [1] claims that their method assumes a binary label on shadows and does not consider soft shadows (in the Discussions Section) yet.\nThe initial experiment we conducted above also proves such an idea that utilizing fine-grained shadow properties could enhance the 3D reconstruction performance.\nDue to the limited time of the rebuttal, we will add thorough discussions for 3D reconstruction in the revised introduction and experiment sections.\n\n**Q3: Provide a baseline of the “trivial” solution described in 4.1 to substantiate the claim that using an auto encoder...ablating the impact of the shadow augmentation in training...**\n\nA3: Thanks for the valuable suggestions.\nTwo additional quantitative comparisons on the FSD dataset are added to the table below.\nOne is a baseline with an auto-encoder structure (L128-L129), and the other is an ablated version without using the shadow augmentation strategy.\nWe can see that both strategies, i.e., a trivial solution like an auto-encoder and an ablated version without shadow augmentation, cannot achieve good performance, validating the effectiveness of our model design.\n\n| Method                     | BER (↓) | RMSE (↓) | WSE (↓) |\n|----------------------------|---------|----------|---------|\n| AutoEncoder            | 4.01    | 0.0919   | 10.49   |\n| Ours (w/o Augmentation)    | 2.16    | 0.0744   | 6.02    |\n| **Ours (full)**            | **1.78**| **0.0669**| **5.78**|\n\n\n**Q4: The authors fail to mention in the main text that the proposed dataset is simulated...**\n\nA4: We apologize for our tone and will make it clear that we use a physics-based simulator to ensure precise opacity annotations.\nDue to the page length limitation, we just highlight the main difference between our dataset and the existing dataset in the paper (see Table 1), and provide a detailed dataset construction process in the supplementary (see Sec. A).\nAs we make the first attempt towards the fine-grained shadow detection problem, we believe that constructing a real shadow dataset with varied shadow properties can be an interesting and useful future work.\nOur dataset and code will also be released to facilitate future research in the community.\n\n**Q5: Organization: I’d suggest moving the dataset section to before the experiments.**\n\nA5: As suggested, we will reorganize the paper to place the dataset section before the experiments.\n\n**Q6: Further motivation why the new metric (weighted shadow error) is needed should be added.**\n\nA6: BER and RMSE are commonly used to measure the accuracy of discrete and continuous labels, respectively (see L191-L194).\nAlthough RMSE can also be used for our fine-grained shadow detection problem, it does not distinguish shadow and non-shadow regions in the image.\nAs non-shadow regions dominate in most images, we use the weighted strategy to pay more attention to nuanced shadow properties in shadow regions.\nThus, we propose a new metric, named Weighted Shadow Error (WSE), which is a weighted RMSE value for shadow detection between the shadow and non-shadow regions.\n\nTo further validate whether the proposed WSE metric better aligns with human perception, we conduct a user study below.\nFor a set of images, we consider the users' average ranking order $seq_{user}$ of fine-grained shadow detection quality for these images, and measure the similarity between $seq_{user}$ and the ranking order $seq_{metric}$ derived by the evaluation metric.\nA higher similarity score represents that this metric aligns better with human perception of shadow detection quality.\nWe recruit 20 participants and randomly select 10 images from our testing dataset.\nIn each comparison, 3 detected results by our method are displayed in random order, and the corresponding input and ground truth are also displayed side by side.\nThe participants are asked to give the ranking order based on the shadow detection quality.\nIn the end, the similarity score of WSE is 83% and the similarity score of RMSE is 64%, confirming once again that the new WSE metric aligns better with human perception of shadow detection quality.\n\n**Q7: Should equation 5 have an absolute value? L188 typo: covert to convert.**\n\nA7: Sorry for the typos. We indeed use the absolute value in Eq. 5. We will proofread and correct all typos in the revised paper.\n\n**References:**\n\n[1] Tiwary et al., Towards learning neural representations from shadows, ECCV 2022.\n\n[2] Liu et al., What You Can Reconstruct from a Shadow, ICCV 2023."}, {"review_id": "N5FV6M4auT", "reviewer": "Reviewer_cS6y", "summary": "This paper presents a fine-grained shadow detection method that uses opacity instead of binary masks. It models shadows as having varying degrees of opacity rather than treating them as uniformly opaque regions, better reflecting the physical interactions of light with objects and surfaces. The proposed method consists of three components: a Shadow Opacity Augmentation (SOA) module to simulate varied shadow opacities during training, a Shadow Feature Separation (SFS) module to disentangle shadow position and opacity features, and an Opacity Mask Prediction (OMP) module to generate a continuous shadow opacity map. Furthermore, the paper introduces a new dataset — Fine-grained Shadow Detection (FSD) — containing opacity-annotated masks under varied real-world conditions. Extensive experiments, including comparisons and ablations, show superior performance in shadow detection and downstream tasks (e.g., shadow removal and editing), demonstrating the effectiveness of opacity-aware modeling.", "questions": "Please see Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper is clearly written. The module explanations and illustrations are easy to follow.\n\n* The method is well-motivated, and each module in the pipeline is designed for effective learning. The experiments, including comparisons with state-of-the-art methods, ablation studies, and downstream application demonstrations, are relatively comprehensive.\n\n* Using opacity instead of binary masks for fine-grained shadow representation demonstrates effectiveness in shadow detection research and related tasks such as image editing, relighting, and scene understanding.\n\n* The concept of modeling shadow opacity variations and explicitly learning opacity-aware representations seem novel in the context of shadow detection. The construction of the FSD dataset adds further value, as it addresses an important gap in existing resources.\n\n\nWeaknesses:\n\n* There is a lack of direct and comprehensive comparison with similar continuous mask representations. While the paper compares its approach against state-of-the-art binary mask detectors, it does not explore baselines from related fields such as alpha matting and soft segmentation, which also produce continuous-valued outputs. Additionally, some prior work has explored similar representations for example using soft masks [1]. Including discussions and comparisons with these methods could strengthen the evaluation.\n\n* More experimental results are needed to demonstrate the method's effectiveness. Specifically:\n\n(a) Opacity estimation in complex scenes. While briefly acknowledged, model limitations in handling multiple interacting shadows or varying illumination directions warrant deeper analysis.\n\n(b) Comparisons with recent shadow removal methods. More thorough comparisons and discussions with recent works [1–4] would provide a clearer picture of the method’s strengths and weaknesses in shadow removal and downstream tasks.\n\n* Evaluation limitations. The proposed Weighted Shadow Error (WSE) metric seems promising, but its interpretation and correlation with downstream task quality could be elaborated further. There is limited discussion on how this new metric aligns with human perception or real-world usability.\n\n* Potential dataset bias.  It is unclear how diverse the newly introduced FSD dataset is in terms of scene diversity, lighting conditions (e.g., artificial, natural, mixed), or camera types. Limited diversity in these areas could hinder the generalization ability of models trained on this dataset. It would be helpful to include a characterization map or summary table highlighting key dataset attributes such as: Distribution of indoor vs. outdoor scenes; Proportions of different lighting types (e.g., daylight, incandescent, LED); Variety of camera sensors or sources.\n\n\n[1] Wang et al. SoftShadow: Leveraging Soft Masks for Penumbra-Aware Shadow Removal. CVPR 2025.\n\n[2] Wang et al. MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis. CVPR 2025.\n\n[3] Lu et al. EvenFormer: Dynamic Even Transformer for Real-World Image Restoration. CVPRW 2025.\n\n[4] Fu et al. Multi-level Feature Fusion Network for Shadow Removal Detection. TCSVT 2025.", "was_revised": true, "final_justification": "Most of my concerns have been addressed well in the rebuttal. As I wrote to the authors, the limitations and the additional results should definitely be included in the revised paper.", "rebuttal": "We thank the reviewer for recognizing the presentation, motivation, and the extensive evaluation of our work.\nGiven the opportunity, we will incorporate all suggestions with details in the revision.\nBelow, we address all concerns raised.\nWe will be very happy to clarify further concerns (if any).\n\n**Q1: There is a lack of direct and comprehensive comparison with similar continuous mask representations...explore baselines from related fields such as alpha matting and soft segmentation, which also produce continuous-valued outputs.**\n\nA1: Thanks for the valuable suggestion.\nWe follow the continuous mask creation process from related fields such as alpha matting and soft segmentation, and conduct additional comparisons as follows.\nIn particular, the baseline combines Alpha Matting [5] with the binary shadow mask detection method [6] to produce continuous-valued outputs as the estimated shadow opacity.\nThe BER, RMSE, and WSE results of the above baseline on the FSD dataset are 3.01, 0.1405, and 8.05, respectively, while our method achieves 1.78, 0.0669, and 5.78, respectively.\nThese results demonstrate that, unlike our method, existing general continuous mask representations do not consider the physical properties of shadows, leading to poor performance in estimating shadow opacity.\n\n**Q2: Some prior work has explored similar representations, for example, using soft masks [1]. Including discussions and comparisons with these methods could strengthen the evaluation.**\n\nA2: As stated in L80-L82, we model the shadow opacity variations explicitly by the underlying physics of light for shadow detection, while SoftMask [1] generates soft masks as intermediate results for shadow removal.\nThus, SoftMask [1] cannot be easily adapted to downstream applications like shadow detection and editing.\nWe will add a thorough discussion of these existing works in the revision.\n\n**Q3: Opacity estimation in complex scenes. While briefly acknowledged, model limitations in handling multiple interacting shadows or varying illumination directions warrant deeper analysis.**\n\nA3: Due to the page length limitation, we discuss the limitations briefly in the paper (see L271-L275), and show detailed visual analysis in the supplemental (see Fig. 11 and Sec. D).\nIn scenarios like forests or low-light environments with scattered or overlapping shadows, while the performance of our method’s opacity estimation drops, existing binary shadow representations (without continuous value gradients) often perform even worse due to the lack of smooth transitions.\nOne possible solution is to consider scene illumination properties for shadow opacity estimation in complex scenes. \n\n**Q4: Comparisons with recent shadow removal methods. More thorough comparisons and discussions with recent works [1–4].**\n\nA4: As these works were published recently after the NeurIPS submission deadline, they are not open-sourced yet.\nWe consider these shadow removal methods as concurrent works, and will add detailed analysis in the revision.\n\nAs stated in L80-L82, SoftMask [1] generates soft masks as intermediate results for shadow removal, while we model the shadow opacity variations explicitly by the underlying physics of light for shadow detection.\nThus, it cannot be easily adapted to downstream applications like shadow detection and editing.\n\nMetaShadow [2] focuses on instance-level shadow detection, removal, and synthesis, while we focus on all shadows in the scene.\nInstance-level fine-grained shadow detection may help the downstream applications like image editing, and could be an interesting future work.\n\nEvenFormer [3] is specifically designed for real-world image restoration, and all experiments are based on different scenarios (i.e., NTIRE25-Shadow-Test dataset).\n\nThe goal of Fu et al. [4] is to detect the remaining shadows in the image after the shadow removal process, rather than shadow removal.\n\n**Q5: Evaluation limitations. The proposed Weighted Shadow Error (WSE) metric seems promising...there is limited discussion on how this new metric aligns with human perception or real-world usability.**\n\nA5: BER and RMSE are commonly used to measure the accuracy of discrete and continuous labels, respectively (see L191-L194).\nAlthough RMSE can also be used for our fine-grained shadow detection problem, it does not distinguish shadow and non-shadow regions in the image.\nAs non-shadow regions dominate in most images, we use the weighted strategy to pay more attention to nuanced shadow properties in shadow regions.\nThus, we propose a new metric, named Weighted Shadow Error (WSE), which is a weighted RMSE value for shadow detection between the shadow and non-shadow regions.\n\nTo further validate whether the proposed WSE metric better aligns with human perception, we conduct a user study below.\nFor a set of images, we consider the users' average ranking order $seq_{user}$ of fine-grained shadow detection quality for these images, and measure the similarity between $seq_{user}$ and the ranking order $seq_{metric}$ derived by the evaluation metric.\nA higher similarity score represents that this metric aligns better with human perception of shadow detection quality.\nWe recruit 20 participants and randomly select 10 images from our testing dataset.\nIn each comparison, 3 detected results by our method are displayed in random order, and the corresponding input and ground truth are also displayed side by side.\nThe participants are asked to give the ranking order based on the shadow detection quality.\nIn the end, the similarity score of WSE is 83% and the similarity score of RMSE is 64%, confirming once again that the new WSE metric aligns better with human perception of shadow detection quality.\n\n**Q6: Potential dataset bias. It is unclear how diverse the newly introduced FSD dataset is in terms of scene diversity, lighting conditions, or camera types...**\n\nA6: We construct the FSD dataset with several guidelines, including object, scene, and light source properties (see L102-l112 in the paper).\nThe detailed construction process is illustrated in the supplemental (see L12-L65).\nWe will visualize and highlight the key dataset attributes more clearly in the revision.\n\nIn particular, the FSD dataset ensures diversity through the following guidelines to avoid dataset bias:\n+ Scenes: The number of indoor and outdoor scenes is 1113 and 514, respectively. The number of objects in each scene ranges from 1 to 5. \n+ Lighting: Brightness and positions are randomly sampled.\n+ Camera: All cameras have a 50mm focal length, AUTO sensor setting, and randomized positions.\n\n**References:**\n\n[1] Wang et al., SoftShadow: Leveraging Soft Masks for Penumbra-Aware Shadow Removal. CVPR 2025.\n\n[2] Wang et al., MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis. CVPR 2025.\n\n[3] Lu et al., EvenFormer: Dynamic Even Transformer for Real-World Image Restoration. CVPRW 2025.\n\n[4] Fu et al., Multi-level Feature Fusion Network for Shadow Removal Detection. TCSVT 2025.\n\n[5] Gastal et al., Shared sampling for real‐time alpha matting. CGF 2010.\n\n[6] Zhu, et al., Mitigating intensity bias in shadow detection via feature decomposition and reweighting. ICCV, 2021."}, {"review_id": "UEClnpC85j", "reviewer": "Reviewer_WyPt", "summary": "This paper proposes predicting an opacity map instead of a traditional binary shadow mask for the task of fine-grained shadow detection. To support this, the authors introduce a new dataset, FSD, and design three novel modules. The method achieves comparable or superior performance not only on the proposed dataset but also on existing shadow detection benchmarks, which is a notable strength. The paper is clearly written and easy to follow, and the extensive experimental results effectively demonstrate the efficacy of the proposed approach.", "questions": "I have several questions regarding the design choices made in the paper:\n\n- The paper adopts a cumulative learning strategy, but it is not clear why this is necessary.\n    - How crucial is this strategy to the final model performance?\n    - Have the authors conducted any ablation or sensitivity analysis to validate its effectiveness?\n    - How were the related hyperparameters selected?\n- Can shadow removal models also be used for shadow detection?\n    - For instance, given both the shadow and shadow-free images (as in the FSD dataset), one could compute an opacity map or binary mask from their difference.\n    - Could such derived maps serve as a competitive baseline to the proposed method?\n    - If so, is there a specific reason the authors chose not to compare against such approaches?\n- The baseline comparisons seem outdated. According to Table 2, the most recent papers used for comparison are from 2021 (for ISTD) and 2020 (for FSD). This is a significant limitation. The authors should compare their method against more recent state-of-the-art works, especially dealing with soft shadows.\n    - *HomoFormer: Homogenized Transformer for Image Shadow Removal*, CVPR 2024\n    - *Detail-Preserving Latent Diffusion for Stable Shadow Removal*, CVPR 2025\n    - *OmniSR: Shadow Removal under Direct and Indirect Lighting*, AAAI 2025\n    - *DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence*, AAAI 2024\n\nPlease let me know if I have misunderstood any part of the paper.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n- The paper is very well written, making it easy to understand both the shadow detection field and the specific problem addressed by the authors.\n- The authors have made a clear effort to enhance the quality of the paper through comprehensive experimental results, comparative tables with existing datasets, and well-designed model architecture figures.\n- The inclusion of a limitations section and an analysis of failure cases in the appendix contributes positively to the paper’s credibility and transparency.\n\n**Weakness**\n\n- Please refer to the Questions section below for specific concerns.", "was_revised": true, "final_justification": "After carefully revisiting both the authors’ rebuttal and the main paper, I acknowledge that the authors have addressed most of my concerns—particularly those regarding the ablation of the cumulative learning strategy and the use of straightforward or outdated baselines. Given the constraints of the rebuttal phase, where visual results cannot be included, the authors appear to have provided the best possible clarification.", "rebuttal": "Thanks for the thoughtful review and positive comments on the paper’s clarity, comprehensive experiments, and limitation discussions.\nGiven the opportunity, we will incorporate all suggestions with details in the revision.\nBelow, we address all concerns raised.\nWe will be very happy to clarify further concerns (if any).\n\n**Q1: The paper adopts a cumulative learning strategy, but it is not clear why this is necessary...How were the related hyperparameters selected?**\n\nA1: Thanks for the valuable suggestion.\nThe cumulative learning strategy (see L148–L151) dynamically balances position and opacity features by adjusting the fusion weight η over training epochs, optimizing fine-grained shadow detection accuracy. \nWe validate the effectiveness of this strategy by conducting ablation studies on cumulative learning and hyperparameter selection in the FSD dataset, with the results shown below. \nWe can see that the cumulative learning strategy is crucial for fine-grained shadow detection, and η = 0.5 achieves the best performance.\n\n| Method                     | BER (↓) | RMSE (↓) | WSE (↓) |\n|----------------------------|---------|----------|---------|\n| w/o cumulative learning     | 2.18    | 0.1505   | 10.83  |\n| η = 0.3                    | 1.93    | **0.0635**   | 6.21   |\n| η = 0.4                    | 1.83    | 0.9840    | 6.16   |\n| η = 0.6                    | 1.80    | 0.0675   | 6.17   |\n| **Ours (η = 0.5)**            | **1.78**| 0.0669| **5.78**|\n\n**Q2: Can shadow removal models also be used for shadow detection...Could such derived maps serve as a competitive baseline to the proposed method?**\n\nA2: Yes. Shadow removal models can also be used for shadow detection.\nHowever, we did not conduct such an experiment for two reasons.\nFirst, shadow removal is often considered as a downstream task of shadow detection, and thus is rarely used as an input for shadow detection.\nSecond, even the SOTA shadow removal models cannot ensure the complete removal of all shadows in the image.\nThus, using shadow removal models for shadow detection may contain a lot of noise with false positives and false negatives.\nTo validate the above explanations, we conducted an experiment on the FSD dataset.\nHomoFormer, the shadow removal model, is used for shadow detection.\nThe BER, RMSE, and WSE results are 5.31, 0.1071, and 8.32, respectively, which are significantly worse than our method (1.78, 0.0669, 5.78).\n\n**Q3: The baseline comparisons seem outdated...The authors should compare their method against more recent state-of-the-art works [1-4], especially dealing with soft shadows.**\n\nA3: Sorry for the confusion. \nNote that our motivation is that explicitly modeling the shadow opacity can help the downstream applications like shadow detection and shadow removal.\nThus, the experimental comparisons are separated into two aspects, i.e., shadow detection and shadow removal.\n\nFor shadow detection, as there is no existing method that predicts soft shadows explicitly, we compare with SOTA shadow detection methods in Table 2.\n\nFor shadow removal, we have compared our method with SOTA shadow removal methods in Table 4, including HomoFormer[1] and DeS3[4].\nFor OmniSR [3], as it involves additional inputs (i.e., Normal Map and Depth Map) in the training process with a new synthetic dataset, it would be difficult to compare the shadow removal performance directly.\nWe further compare the results with Detail-Preserving Latent Diffusion (CVPR 2025) [2].\nOn the SRD dataset with varied shadow opacities, the PSNR and SSIM results of [2] are 33.63 and 0.968, while our method achieves 34.84 and 0.974, respectively, confirming once again the effectiveness of our method on downstream applications like shadow removal.\nWe will make the discussions more clearly, and conduct additional comparisons with recent SOTA methods in the revision.\n\n\n**References:**\n\n[1] Xiao et al., HomoFormer: Homogenized Transformer for Image Shadow Removal, CVPR 2024\n\n[2] Xu et al., Detail-Preserving Latent Diffusion for Stable Shadow Removal, CVPR 2025\n\n[3] Xu et al., OmniSR: Shadow Removal under Direct and Indirect Lighting, AAAI 2025\n\n[4] Jin et al., DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence, AAAI 2024"}]

Author rebuttal:

Meta-review: This paper initially received mixed to slightly negative ratings, with most reviewers leaning toward rejection or borderline accept due to concerns about novelty, clarity of motivation, missing comparisons, and limited evaluation. After the rebuttal, however, all reviewers acknowledged that the authors had addressed most of these concerns satisfactorily. Several reviewers explicitly noted that the rebuttal resolved their key issues, though they still encouraged the authors to strengthen the paper with additional results, updated comparisons, and deeper analysis.

Overall, while the paper remains borderline in its current form, the consensus is that the idea is technically sound and the contributions are promising. The authors should revise the paper accordingly for the camera-ready version to include experiments or discussions in the rebuttal and fully address the remaining concerns.
Decision: Accept (poster)