Timeline and Boundary Guided Diffusion Network for Video Shadow Detection

Authors: haipeng  zhou
honqiu  wang
tian  ye
zhaohu  xing
jun  ma
ping  li
qiong  wang
lei  zhu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-11 10:32:29

Abstract: Video Shadow Detection (VSD) aims to detect the shadow masks with frame sequence. Existing works suffer from inefficient temporal learning. Moreover, few works address the VSD problem by considering the characteristic (i.e., boundary) of shadow. Motivated by this, we propose a Timeline and Boundary Guided Diffusion (TBGDiff) network for VSD where we take account of the past-future temporal guidance and boundary information jointly. In detail, we design a Dual Scale Aggregation (DSA) module for better temporal understanding by rethinking the affinity of the long-term and short-term frames for the clipped video. Next, we introduce Shadow Boundary Aware Attention (SBAA) to utilize the edge contexts for capturing the characteristics of shadows. Moreover, we are the first to introduce the Diffusion model for VSD in which we explore a Space-Time Encoded Embedding (STEE) to inject the temporal guidance for Diffusion to conduct shadow detection. Benefiting from these designs, our model can not only capture the temporal information but also the shadow property. Extensive experiments show that the performance of our approach overtakes the state-of-the-art methods, verifying the effectiveness of our components. We release the codes at https://github.com/haipengzhou856/TBGDiff.

Main paper: 
 introduction: 
 Shadow detection is increasingly important in vision analysis and applications. Vision tasks can suffer from the shadows, including incorrect segmentation [52] , inaccurate object detection [15, 25] , and flawed tracking [10, 37] . Hence, shadow detection has become a highly focused area of research. Recently, one can witness significant progress in single Image Shadow Detection (ISD) [11, 25, 26, 79, 80] , whereas in the dynamic scenario, Video Shadow Detection (VSD) is much more challenging.Temporal correspondence information matters in video shadow detection. For example, the SC-Cor [16] focuses on the relationship between shadow and optical-flow. It relies on a contrastive loss to explore the temporal correspondence for adjacent frames. However, like other Unsupervised Video Object Segmentation (UVOS) [41, 43, 50, 56, 63, 73] methods, it still depends on additional clues (i.e., optical-flow) and lack of semantic correspondence. We point out that the adjacent frames usually change slightly, leading the model to focus on the consistent area while distracting on the deformation region, which is more crucial for shadow detection.In addition, few works notice the characteristic of shadow to propose a task-specific model. We are attracted to recent shadow removal works [19, 38] , which use the boundary information to guide the restoration model to remove shadows. This indicates the boundary can provide potential clues to identify the shadows. Meanwhile, the contexts of boundary often contain higher uncertainty [59, 76, 78] making it difficult for the model to perform accurate segmentation. These observations motivate us to explore the boundary information in VSD. Moreover, the shadow detection works are dominated by CNNs [10, 11, 14, 16, 25, 26, 35, 79, 80] and Transformers [32, 34, 60, 69] . Advanced architectures can improve performance. For instance, Scotch&SODA [32] , equipped with a Transformer-based backbone, surpasses other cutting-edge VSD methods by a large margin. Recently, Diffusion models present remarkable results in image generation [1, 4, 24, 44, 48] and semantic segmentation [2, 7, 9, 27] . Whereas little effort has been made to explore the temporal guidance of Diffusion model for Video. In the scheme of conditional Diffusion, guidance matters since it can instruct Diffusion to approach a distribution in a specific direction. Therefore, it is worth studying effective temporal guidance for Diffusion models to enhance the performance in VSD.To tackle the aforementioned problems, we propose Timeline and Boundary Guided Diffusion (TBGDiff) for video shadow detection. To our best knowledge, this is the first work introducing Diffusion model for shadow detection. The core idea of our method is to utilize the temporal information in the clipped video and explore the boundary contexts for the Diffusion network to conduct VSD. In detail, (1) we design a Dual Scale Aggregation (DSA) module which is plug-and-play to aggregate the temporal features. Inspired by the residual operation in ResNet [22] , we rethink the affinity [13, 39] in video condition. We adopt the vanilla affinity to capture the consistent context for short-term frames and propose a residual affinity to encourage the model to focus on the deformation area of shadows for long-term frames as well. (2) We present a Shadow Boundary-Aware Attention (SBAA) to encourage the model to represent the characteristics of shadows. We embed the boundary position into the attention mechanism [51] to guide the model to more accurately distinguish between shadow and nonshadow areas. (3) We explore three different temporal guidance for Diffusion model to detect shadows in video scenario. Via considering the timeline frames (i.e., the past and future frames), we develop the best practice called Space-Time Encoded Embedding (STEE) to inject the temporal guidance into the conditional Diffusion model. Instead of using heavy U-Net [45] to predict noise, our TBGDiff can progressively decode the mask via the reverse process.In summary, our four-fold contributions are:• We develop Timeline and Boundary Guided Diffusion (TBGD -iff) for video shadow detection, which is the first work introducing Diffusion model to conduct shadow detection. Our TBGDiff outperforms state-of-the-art methods by a large margin, verifying the effectiveness of our approach.• To guide the Diffusion to learn temporal information, we propose three different ways to produce guidance. The devised Space-Time Encoded Embedding (STEE) enables our model to capture the representation from a timeline sequence (past and future frames), resulting in the best performance. • We develop a Shadow Boundary-Aware Attention to help the model understand the boundary context. Our model can be further improved by benefiting from focusing on the boundary-aware region. • We introduce a plug-and-play method for video understanding, Dual Scale Aggregation (DSA), to explore the affinity in video sequence. Our DSA is able to conduct short-term consistency context learning and long-term visiting. 
related works 2.1 video object segmentation: 
 Different from Semi-Supervised Video Object Segmentation (SSVOS) [12, 13, 29, 30, 39, 54] which provides the first frame mask to initialization during the testing stage, the Video Shadow Detection (VSD) follows the paradigm of Unsupervised Video Object Segmentation (UVOS) [28, 36, 41, 47, 50, 63] where we detect the shadow without the ground truth in the initial frame. In VOS, it usually deploys auxiliary encoder to extract temporal information, like optical flow [41, 43, 50, 63] , vanilla encoder [36, 55, 57] , or salient map [28, 47] . Recent studies [12, 13, 39, 54, 75] focus on the affinity between the past and current frames such that one can build a memory-bank, which aims to utilize the sequential information and past prior for video understanding. Not only in the past, but we also reconsider the information of future frames, i.e., in a timeline sequence. We rethink the affinity in dual temporal scales to consider timeline aggregation instead of a sequential way.Previous shadow detection works [11, 20, 25, 58, 79, 80] focus on the image shadow detection (ISD). For example, Zhu et al. [80] design a bidirectional FPN [31] to extract local and global contexts for detecting shadow. With respect to the dynamic scenarios, i.e., video shadow detection, it encounters huger challenges on account of the complexity of the real-world. Chen et al. [10] collect the first video shadow detection dataset named ViSha, and introduce TVSD-Net to apply collaborative training in different videos to learn the shadow context. Similarly, STICT [35] deploys Teacher-Student model [23] to achieve video consistency learning. Considering the temporal correspondence, SC-Cor [16] enables the network to focus on the anchor pixel of shadow via contrastive learning. In order to balance temporal learning and contrastive learning, in Scotch&SODA [32] the authors apply trajectory Transformer to conduct VSD and set a new record on ViSha dataset. Differently, motivated by the shadow removal works [19, 38, 68] which utilizes the boundary information to guide the image restoration, we first design a specific shadow boundary-aware attention to detect the characteristic of shadows.Diffusion model has shown remarkable promise in visual generation [4, 24, 42, 44, 48] , and it enlightens other tasks like object detection [6] , segmentation [2, 7, 9, 27] classification [21, 71] . For segmentation tasks, the denoise process is not suitable [8, 27] introduces a simple and generic way to embed and analog the discrete mask into continuous signals, and it also presents a simple concatenation strategy to process VOS. While there is still room to improve the temporal guidance for Diffusion to tackle VOS. Moreover, the efficiency should be considered as well. Though several works [4, 48, 49] dedicate to accelerate the inference, the conventional Diffusion models [1, 4, 24, 40, 44, 48, 74, 77] use heavy U-Net for noise estimation. Huge parameters make it hard to go on the downstream works. Instead, we further explore the feasibility and temporal understanding of the Diffusion model for VSD. 
methodology 3.1 overview: 
 Given 2𝑖 + 1 frames, our TBGDiff can simultaneously detect the shadow for all clipped sequences. In brevity, we illustrate the workflow of the 𝑇 -th frame 𝑥 𝑇 in Fig. 1 . First, we take an encoder to extract the temporal-agnostic features for all the frames, then the features are sent to the Dual Scale Aggregation (DSA) module to implement temporal aggregation. With an Auxiliary Head, these aggregated features are used to produce pseudo masks and boundary masks. Next, we input the 𝑇 -th aggregated feature 𝐹 𝐷𝑆𝐴 𝑇 , pseudo mask 𝑦 𝑇 , and boundary b𝑇 to Shadow Boundary-Aware Attention (SBAA) to further explore the characteristic of shadows. To utilize the timeline temporal information for Diffusion, we use a guidance encoder to yield Space-Time Encoded Embedding (STEE) via encoding the past and future pairs (pseudo masks and images). Finally, we adopt bit analog strategy [7] to embed noise and conduct the denoise process to predict the final shadow masks.When it comes to video-related works, the matching-based methods [13, 18, 39, 61, 62, 70, 72, 73, 75] usually adopt affinity to read and visit the space-time correspondences. However, the vanilla affinity will introduce the concern: it intends to give more weight to the adjacent frames because the contexts of close-range sequences change smoothly and slightly, and the interval frames gain less attention due to time-shift. We argue that both of the temporal scales of features should be considered, and to alleviate the temporal bias we propose a Dual Scale Aggregation (DSA) module where we rethink the affinity considering short-term and long-term scenarios jointly.The core of affinity is to compute the similarity between the query feature and the memory feature to retrieve the temporal and spatial feature. Given the query feature 𝑄 ∈ R 𝐶 𝑄 ×𝐻𝑊 , the memory key feature 𝐾 ∈ R 𝐶 𝐾 ×𝑁 𝐻𝑊 , and memory value feature 𝑉 ∈ R 𝐶 𝑉 ×𝑁 𝐻𝑊 (the 𝐻 and 𝑊 are the spatial dimensions, the 𝐶 {𝑄,𝐾,𝑉 } denote the channels, and 𝑁 represents the memory length), the affinity 𝑀 ∈ R 𝑁 𝐻𝑊 ×𝐻𝑊 is computed as:EQUATIONwhere 𝑓 (•) is L2 similarity function [13] . Such that, we can readout the aggregated feature 𝐹 ∈ R 𝐶 𝑉 ×𝐻𝑊 via the matrix multiplication: , we can obtain the boundary-aware embedded tokens serving as the query. We also use the pseudo mask to weight the coarse shadow regions via element-wise multiplying these tokens to produce key and value. Such that, we can implement attention mechanism and FFN to output the boundary-aware and shadow-oriented features.EQUATIONSuch a vanilla affinity can be deployed for short-term aggregation. With the current frame feature 𝐹 𝑇 as 𝑄, and the concatenated adjacent features {𝐹 𝑇 -1 , 𝐹 𝑇 +1 } as the 𝐾 𝑠 and 𝑉 𝑠 (see Fig. 1 ), we readout the short-term aggregated feature via the vanilla affinity:EQUATIONConsidering the long-term visiting, Eq. 1 indicates that similar areas among different frames occupy higher weight. This leads the model to distract from the deformation region. To encourage the network to focus on it, we propose residual affinity to enhance the weight of deformation areas. Similarly, we have the query of 𝐹 𝑇 , the key and value 𝐾 𝑙 = 𝑉 𝑙 = {𝐹 𝑙 } where 𝑙 ∈ [𝑇 -𝑖, ...,𝑇 -2,𝑇 + 2, ...,𝑇 + 𝑖], and the long-term aggregated feature can be obtained by a residual operation with the affinity matrix:EQUATIONwhere 𝑀 𝑠𝑒𝑙 𝑓 is a self-affinity anchored the consistent areas and the 𝑄 ′ is the broadcasting version of 𝑄 to match the size. An explanation is that the close-range frames usually contain consistent area leading to higher similarity, while the discrepancy area receives less attention. Adopting a subtraction operation on the affinity matrix, the residual area will reveal the difference region which is crucial to track the shadow deformation. With 𝐹 𝑠 𝑇 and 𝐹 𝑙 𝑇 , our DSA can produce the dual scales aggregated features for 𝑇 -th frame via a simple convolutional residual block:EQUATIONConsidering the property of shadows, previous shadow removal works [19, 38, 68] suggest that the marginal contexts of shadows indicate crucial clues to identify the shadow and non-shadow regions.Motivated by this, we design a Shadow Boundary-Aware Attention (SBAA) for specializing in detecting the shadows. First, we introduce an Auxiliary Head in which we input the aggregated features 𝐹 𝐷𝑆𝐴 to produce the pseudo mask 𝑦 and boundary mask b. Benefiting from this design, the pseudo mask can produce the semantic guidance for our Diffusion (see Sec. 3.4), and the boundary mask helps the model capture the characteristics of shadows. Here, we use the auxiliary loss to supervise the production via:EQUATIONwhere 𝑏 and 𝑦 are the ground truths of the boundary and shadow masks, respectively. We adopt Binary Cross Entropy (𝐿 𝑏𝑐𝑒 ) to compute loss. Note that all the frames are conducted.Then move to a specific frame at 𝑇 , given the aggregated feature 𝐹 𝐷𝑆𝐴 𝑇 ∈ R 𝐶 ×𝐻 ×𝑊 from DSA, the predicted boundary b𝑇 , and the pseudo mask 𝑦 𝑇 , we regard b𝑇 as the position embedding such that the boundary-aware embedded tokens can be obtained by:EQUATIONwhere E is a patch embedding projection [17] , and we flatten the boundary mask to obtain E 𝑏𝑝 serving as the boundary-aware position embedding. To highlight the shadow region, we propose the SBAA which can be described as:EQUATIONSBAA = Softmax(EQUATIONwhere 𝑊 𝑞 , 𝑊 𝑘 , and 𝑊 𝑣 are the learnable matrices, and 𝐶 is the number of channel to scale. We implement broadcasting mechanism to extend the channels of 𝑦 𝑇 to match the size of 𝐹 ′ 𝑇 , and 𝑦 𝑇 can serve as the weights of probability to emphasize the shadow-relevant areas. A visual depiction of our SBAA can be found in Fig. 2 .By doing so, the boundary-aware query is able to better retrieve the shadow regions. And based on the attention, we can deploy feed-forward network (FFN) [51] (i.e., the MLP linear layer) on it to produce the output feature which is used in our Diffusion process to give the final prediction.Recently, Diffusion Models have indicated powerful ability in segmentation tasks [2, 7, 27, 65] . When it comes to VOS, Pix2Seq [7] adopts a straightforward way that just concatenates the past predicated masks into Diffusion as guidance to conduct VOS. While, the potential of conditional Diffusion in VOS has yet to be further explored. Motivated by this, we try to seek more effective temporal guidance for Diffusion models and deploy it in VSD.Briefly, Diffusion model [24, 48] contains a forward process 𝑞 and a reverse process 𝑝. The 𝑞 aims to gradually add Gaussian noise to corrupt the distribution of a mask 𝑦 0 making it close to a normal distribution, which can be illustrated as:EQUATION)The 𝑡 denotes the timestep, and ᾱ𝑡 is a noise scheduler which could be adjusted by a cosine or linear style. And the reverse process 𝑝 𝜃 is parameterized by a network 𝜃 (𝑦 𝑡 , 𝑔, 𝑥), which is used to predict 𝑦 0 Instead of predicting noise like conventional Diffusion models, we predict the mask since the robust representation and bit analog strategy [8, 27] controlled by a scale weight enable the model to directly decode the mask rather than relying on the heavy U-Net. The detail of the Diffusion's operation are provided in Supplementary Material, and the ablation studies of the hyperparameters (scale weight, noise scheduler, and sampling steps) are given later. Here, we mainly discuss the importance of the guidance 𝑔.In the scheme of Diffusion, the conditional guidance is usually accessible (e.g., vectors of text or images). Considering the VOS, Pix2Seq [7] introduces a loop to predict the masks frame by frame such that the obtained predictions can serve as a temporal guidance to promote the following segmentation. It simply downsamples the masks and concatenates them with the latent features, and we denote it as Past Concatenate Embedding (PCE, see Fig. 3(a) ). While, existing Latent Diffusion models [4, 44] have proved that in latent space the Diffusion can perform better, which suggests us rethink the embedding ways. Intuitively, we propose other two methods to embed the guidance: Past Encoded Embedding (PEE, see Fig. 3(b) ) and Space-Time Encoded Embedding (STEE, see Fig. 3(c) ). We concatenate the masks with corresponding frames to form pairs, then a light-weight guidance encoder is deployed to embed them. Compared to PCE, the encoded guidances from PEE and STEE are much more robust to visit the temporal information.Here, we point out that the unidirectional embeddings (PCE & PEE) are not the best solution. They bring the following concerns:(1) The PCE and PEE access the guidance sequentially, leading to lower efficiency. Since the embedding is conducted frame by frame, they are restricted to real-time online shadow detection. Moreover, the uncertainty can accumulate as well. ( 2) The future prediction is agnostic, resulting in limited temporal guidance usage. All the space-time clipped frames should be considered to provide a temporal context. To address the aforementioned issues, we devise STEE to use all the space-time information in an efficient way. Instead of using the predicted masks, we take use of the pseudo masks produced by an Auxiliary Head (see Fig. 1 and Sec. 3.3). Because all the pseudo masks are available, we can execute the guidance encoding in a parallel manner rather than wait for the last prediction. As a result, our Diffusion can visit all the timeline temporal information leading to better performance.As mentioned in the last section, we directly predict the masks rather than the noise. Hence, the objective loss is similar to the segmentation task. Here, we adopt the Binary Cross Entropy [66, 67] and lovasz-hinge loss [3] to restrict training. Considering the auxiliary loss, the total term of our loss function is computed as:EQUATION) 
experiment: 
 We use Video Shadow dataset (ViSha) [10] to conduct our experiments and make comparisons with state-of-the-art methods. The ViSha dataset contains 50 videos for training and 70 videos for testing. Following previous studies [10, 16, 32, 35, 60] , we deploy Mean Absolute Error (MAE), Intersection over Union (IoU), F-measure score (F 𝛽 ), and Balance Error Rate (BER) as evaluation metrics for quantitative comparisons. Regarding the BER, we also compute the S-BER score at the shadow regions and the N-BER score at non-shadow regions, respectively.We utilize AdamW optimizer [33] with a learning rate of 3e-5 to train our model. The batch size is 4, and the clipped sequence is 5 frames. Four A6000 GPUs are used to conduct our experiments, and a fixed random seed ensures the reproduction. For a fair comparison, following Scotch&SODA [32] , we use MiT-B3 [64] as our feature extraction backbone, and all experiments are conducted with a resolution of 512×512. Note that the boundary masks are obtained by utilizing the canny operator on the shadow masks. More setup details are provided in Supplementary Material.State-of-the-art Methods 4.2.1 Compared Methods. We compare our network against 20 cutting-edge methods, including Image Object Segmentation (IOS) [5, 27, 31, 64] , Image Shadow Detection (ISD) [11, 14, 25, 69, 79, 80] , Video Object Segmentation (VOS) [7, 13, 36, 39, 53] , and Video Shadow Detection (VSD) [10, 16, 32, 35, 60] . Note that the Semi-Supervised video segmentation methods like STM [39] , STCN [13] ,and ShadowSAM [60] are not given the label of the first frame during testing, we reproduce them by predicting the initial frame for fair comparisons in line with previous VSD methods and ours.Table 1 : Quantitative comparisons with state-of-the-art methods. We compare with several methods from the Image Object Segmentation (IOS), Image Shadow Detection (ISD), Video Object Segmentation (VOS), and Video Shadow Detection (VSD). The † denotes the methods that require ground truth for initialization in testing, for fair comparisons we directly predict the first frame instead of using the label. Bold indicates the best performances, and underline indicates the second-best performances. either miss some shadow pixels or detect many non-shadow areas in their results. For example, in the 1 𝑠𝑡 row of Fig. 4 , other methods tend to wrongly identify the black headphone as shadows, while our method can alleviate such mistakes. In the 3 𝑟𝑑 and 5 𝑡ℎ rows of Fig. 4 , compared methods fail to identify the complex shadow regions, while our network can better detect these complex shadows due to integrating the shadow boundaries. From these visual depictions, we can conclude that our approach provides an effective solution to address the challenging video shadow detection task. See more visual comparisons in our Supplementary Material. 4.2.4 Efficiency Comparisons. We also compare the efficiency of our method with other video shadow detection works in Tab. 2.TasksThough the parameters of our model (102.3MB) are slightly larger than the smallest one (101.3MB), our approach takes the first rank in terms of the FPS and performance metrics, indicating our solution is much more effective and efficient for video shadow detection. We first conduct an ablation study to evaluate the effectiveness of our two modules (i.e., SBBA, and DSA) on the Diffusion model.To do so, we build a "Baseline" by removing both SBBA and DSA modules from our network. Then "M1" and "M2" are reconstructed by adding the SBBA module and the DSA module into "Baseline". Tab. 3 reports the quantitative results of our network and three constructed networks ("Baseline", "M1", and "M2").3.1 Effectiveness of SBBA. In Tab. 3, we can find that "M1" achieves an IoU improvement of +1.2%, a BER improvement of +9%, an 𝐹 𝛽 improvement of 0.3%, and an MAE improvement of 0.2%, when compared to "Baseline". Moreover, "M2" encounters apparent degradation without using SBAA compared to our final models. These From the results, it can be observed that each scale of aggregation can improve the shadow detection performance. By utilizing both long-term and short-term aggregation, our approach achieves the best results. We also visualize the readout results in Fig. 5 , where we select the mid-frame (i.e., the third one over five frames) for the best view. The long-term readout will focus on the deformable areas in the second column. Though it model (see Sec. 3.4), including the noise scheduler, scale weight, and the sampling step, and report the corresponding quantitative results in Tab. 6. According to the numerical results, we find that our network has the best video shadow detection performance when using a cosine scheduler, a scale weight of 0.01, and a sampling step of 20. Hence, we empirically adopt a cosine scheduler, 20 sampling steps, and a scale weight of 0.01 in our experiments. 
conclusion: 
 In this work, we propose a Timeline and Boundary Guided Diffusion (TBGDiff) network which is the first work to use Diffusion model for video shadow detection task. The main idea of our TBGDiff is to extract temporal guidance for Diffusion and to utilize the boundary information to capture the characteristics of shadows. In detail, we propose a Dual Scale Aggregation (DSA) module to aggregate the temporal signals by rethinking the discrepancy of the affinity among short-term and long-term. We also devise an Auxiliary Head to yield boundary masks and pseudo masks, which can be used for extracting the boundary context of shadows by Shadow Boundary-Aware Attention (SBAA) and producing timeline temporal guidance via Space-Time Encoded Embedding (STEE) for Diffusion, respectively. Experimental results show that the developed designs are effective and our approach can outperform state-of-the-art methods. 
 
 Appendix:  


Decision: A timeline and boundary guided diffusion network has been proposed for the task of video shadow detection. Temporal information is considered through a Dual Scale Aggregation (DSA) and a Spatial-Temporal Encoding Embedding (STEE), while boundary information is captured through a Shadow Boundary Aware Attention (SBAA). 

The proposed method is novel, being the first to apply diffusion model for shadow detection. Exploring the temporal guidance for the diffusion model to conduct shadow detection is also interesting. The experiments are solid, and the results are promising.

Most of the concerns previously raised by reviewers have been addressed in the rebuttal. The authors should also incorporate these changes into the paper upon acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a timeline and boundary guided diffusion network for the task of video object detection. The work successfully embeds the diffusion model into the video shadow detection network and achieves SOTA performance.
strength_weakness: strength_weakness: 1. The paper is clear and well-organized. 
2. The experiment shows a strong performance.
limitations: limitations: 1. Although DSA can capture temporal information from both short and long term sequences. What is the exact parameter of short and long terms? Then the table. 4 shows that DSA with short temporal scales obtains worse performances than the model without DSA, why does the short temporal information weakens the proposed model?
2. It seems that the model predicts the shadow mask of a frame using both past and future frames, does this strategy make the last frames of a video unpredictable and how to solve this problem?
3. Self-attention and diffusion models are both time-consuming modules, and the table.2 shows the model has higher fps than the previous works, so how does the author improve the speed of the model?
paper_summary: paper_summary: The paper presents a Time and Boundary Guidance Diffusion (TBGDiff) network for video shadow detection. TBGDiff integrates past-future temporal guidance and boundary information, designs a Dual Scale Aggregation (DSA) module for better understanding of temporal information, and proposes an exploration of Spatial-Temporal Encoding Embedding (STEE) to inject temporal guidance into the diffusion model. For boundary information, it introduces Shadow Boundary Aware Attention (SBAA) to capture shadow characteristics.
strength_weakness: strength_weakness: 1. The proposed model TBGDiff effectively combines past-future temporal guidance with boundary information, providing a robust solution for video shadow detection.
2. The DSA module is well-designed and capable of handling both short-term and long-term temporal information, enhancing the model's comprehension of temporal context.
3. The SBAA module incorporates boundaries and masks as auxiliary information and combining them with the attention mechanism, it captures shadow boundary characteristics, strengthening the model's ability to distinguish between shadow and non-shadow areas.
4. The STEE  module allows for parallel guidance encoding and incorporation of temporal information across all timelines by obtaining masks in advance, which is efficient and performs exceptionally well.
5. The paper conducted comprehensive experiments to demonstrate the superior performance of the proposed method.
limitations: limitations: 1.  Although the introduction of the diffusion model in video shadow detection is innovative, other components such as the Dual-Scale Aggregation (DSA) and Shadow Boundary Awareness Attention (SBAA) do not show significant innovation based on existing concepts.
2. Regarding the methods proposed in the paper, such as DSA and SBAA, what is the impact on GPU memory usage when computing long sequences simultaneously?
paper_summary: paper_summary: The paper introduces a diffusion-based method for video shadow detection. The key ideas include leveraging cross-frame temporal guidance and edge information to improve shadow detection. The method is consists of three modules. A dual scale aggregation module is designed for better temporal understanding, while a shadow boundary aware attention mechanism captures shadow characteristics. The usage of space-time encoded embedding enhances temporal guidance for shadow detection. Experimental results demonstrate that TBGDiff outperforms state-of-the-art methods in both performance, model size, and speed.
strength_weakness: strength_weakness: 1.	The paper introduce the first diffusion-based method for VSD.
2.	Extensive experiments demonstrate its effectiveness.
3.	Ablation studies verify that each component is an optimal solution.
limitations: limitations: 1.	There is some writing issues may hinder audience from understanding. 1) Table 3 should not list the column of diffusion, if you have not conduct ablation study for it. 2) The name of PCE and PEE is confusing. The actual difference between them and STEE is directional (causal). 3) Auxiliary head and loss take respond for DSA, but it appears at Sec 3.3 SBAA.
2.	The implementation details of diffusion model and guidance encoder are not clear. What lightweight guidance encoder is used?
3.	There are a lot of boundary-aware methods[1,2,3,4] developed in dense-prediction tasks, such as salient object detection, parsing, segmentation, etc. The authors have not discussion the difference between the proposed method and existing ones on handling boundary information. Considering the rationale of the proposed methodology, it should be contained in the introduction and ablation study for clarity.

[1] InverseForm: A Loss Function for Structured Boundary-Aware Segmentation, CVPR21

[2] PlaneSeg: Building a Plug-In for Boosting Planar Region Segmentation, TNNLS23

[3] Decoupled multi-task learning with cyclical self-regulation for face parsing, CVPR22

[4] Synthesize Boundaries: A Boundary-aware Self-consistent Framework for Weakly Supervised Salient Object Detection, TMM23
paper_summary: paper_summary: The paper introduces a diffusion-based method for video shadow detection (VSD). In this work, a timeline and boundary guided diffusion network (TBGDiff) is proposed to address VSD. The proposed method presents the dual scale aggregation and shadow boundary aware attention modules. Good detection performance is achieved by introducing space-time encoded embedding to guide the diffusion model.
strength_weakness: strength_weakness: 1. The use of diffusion models to address VSD is novel.
2. The experiment results on the ViSha dataset are promising.
limitations: limitations: 1. In Section 1, it is important to provide a more detailed and clear articulation of the motivation behind the proposed method (e.g., advantages and benefits of using the Diffusion model for video shadow detection) and the reasons for its effectiveness. This will help readers better understand the rationale for the work.

2. In Section 4.1, the AdamW optimizer is not cited.

3. Notations.
1) In Eq. (9), the transpose symbol could not be represented using ‘T’ because it has already been defined as a frame sequence number in the previous sections.
2) In L273-L274, ‘C’ denotes the channels. Also, ‘C’ is the number of the channel to scale in Eq. (9). Are these two C’s the same?
3) In Eq. (1), ‘M’ denotes a function. However, the description above says that ‘M’ is an affinity matrix. Please use different symbols for this.
4) In Fig. 1, what does the D in the green block indicate? If it is a decoder, what is its exact structure?

4. Typos
1) In L112, ‘(3) we explore’ -> ‘(3) We explore’.
2) In L274, ‘the C denote’ -> ‘the C denotes’.
3) In L915, ‘our TBGiff’ -> ‘our TBGDiff’.