Language-Driven Interactive Shadow Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:08:40

Abstract: Traditional shadow detectors often identify all shadow regions of static images or video sequences. This work presents the Referring Video Shadow Detection (RVSD), which is an innovative task that rejuvenates the classic paradigm by facilitating the segmentation of particular shadows in videos based on descriptive natural language prompts. This novel RVSD not only achieves segmentation of arbitrary shadow areas of interest based on descriptions (flexibility) but also allows users to interact with visual content more directly and naturally by using natural language prompts (interactivity), paving the way for abundant applications ranging from advanced video editing to virtual reality experiences. To pioneer the RVSD research, we curated a well-annotated RVSD dataset, which encompasses 86 videos and a rich set of 15,011 paired textual descriptions with corresponding shadows. To the best of our knowledge, this dataset is the first one for addressing RVSD. Based on this dataset, we propose a Referring Shadow-Track Memory Network (RSM-Net) for addressing the RVSD task. In our RSM-Net, we devise a Twin-Track Synergistic Memory (TSM) to store intra-clip memory features and hierarchical inter-clip memory features, and then pass these memory features into a memory read module to refine features of the current video frame for referring shadow detection. We also develop a Mixed-Prior Shadow Attention (MSA) to utilize physical priors to obtain a coarse shadow map for learning more visual features by weighting it with the input video frame. Experimental results show that our RSM-Net achieves state-of-the-art performance for RVSD with a notable Overall IOU increase of 4.4%. We shall release our code and dataset for future research.

Main paper: 
 related work 2.1 shadow detection.: 
 Shadow detection is crucial in computer vision, aiming to generate binary masks for all shadows [6, 10, 15, 17, 24, 45, 53] . Early techniques employed physical illumination and color models to analyze spectral and geometrical shadow properties [36, 38, 41] . As the machine learning development, subsequent approaches built models using handcrafted attributes like texture [42, 51] , color [12, 21] , and edges [18, 21] . These models were then combined with classifiers like decision trees [21, 51] and support vector machines [12, 18, 42] to differentiate shadow. However, the limited representational ability often restricted their effectiveness in various scenarios. Recent deep-learning methods have made significant strides in shadow detection by effectively learning from shadow images. Khan et al. [20] first propose a framework where relevant features are automatically learned through multiple convolutional deep neural networks. Shen et al. [40] employ structured CNNs to analyze shadow edges' local features. Vicente et al. [43] recommend utilizing quickly obtained, partially accurate image labels, refining them automatically for enhanced performance. Hu et al. [17] develop a network using spatial RNNs for direction-aware context analysis in shadow detection. Chen et al. [4] propose a semi-supervised, multi-task model combining consistency loss from unlabeled data with supervised loss from labeled data.All the above investigations primarily focus on shadow detection for images, while there have been recent endeavors in video-based shadow detection as well. Chen et al. [3] curate a new video shadow detection dataset and develop a triple-cooperative network for enhanced accuracy. Liu et al. [27] introduce shadow deformation attention trajectory, a new video self-attention module meticulously crafted to tackle substantial shadow deformations within videos. Differing from conventional video shadow detection that generates a universal binary shadow mask, we delve into a pioneering realm of referring video shadow detection. This approach facilitates the segmentation of each specific shadow through associated language expressions, enhancing user-friendly and personalized applications.The aim of referring segmentation is to precisely outline the particular object referred to in a natural language expression within an image. This task combines computer vision and natural language processing, similar to our RVSD approach. Hu et al. [16] first tackle the novel challenge of image segmentation guided by natural language expressions. They utilize a hybrid recurrent-convolutional network, encoding expressions with recurrent neural networks and generating response maps using fully convolutional networks from images. Liu et al. [26] present a convolutional multimodal LSTM to encode interactions among words, visual, and spatial information components, enforcing a more effective word-to-image interaction.Instance shadow detection is a subtask of shadow detection, and the goal of the current instance shadow detection is to detect the object and segment the corresponding shadow region. Wang et al. [46] pioneer instance shadow detection and built an image-based dataset for finer segmentation. Xing et al. [50] expand the scope of instance shadow detection from static images to dynamic videos, introducing a new framework to extract shadow-object associations in videos with paired tracking.However, our RVSD stands out from previous approaches in two distinctions. Firstly, RVSD is a novel interactive technique for shadow segmentation, which is more user-friendly. RVSD receives the user's linguistic description and the corresponding video frames as input to segment the relevant shadows, whereas instance shadow detection only inputs image data. Secondly, in contrast with previous approaches that focus on detecting objects in the foreground and segmenting their associated shadows separately, RVSD offers greater flexibility by enabling the segmentation of any shadow of interest, including those of multiple objects or background shadows (objects are invisible), as illustrated in Fig. 1 . 
rvsd dataset: 
 In this section, we introduce the newly established RVSD dataset. We begin by detailing the video collection, annotation and validation process in Section 3.1, followed by an examination of dataset statistics and analysis in Section 3.2. 3.1.1 Video Collection. We perform data collection and re-annotation using the most extensive publicly available video shadow dataset to date, which comprises 120 videos, known as ViSha [3] . After reviewing 120 potential candidates, we carefully exclude shadows that are blended together (hard to differentiate them through textual descriptions). We also remove highly fragmented shadows. Meanwhile, we re-mask the original binary shadow mask to label the shadows of different instances separately. Eventually, with an emphasis on quality over quantity, we meticulously select 86 videos to construct a benchmark dataset that encompasses diversity and represents a broad spectrum of real-world video scenarios. We also conduct an analysis of various scenarios where shadows are present in the video. Further details can be found in Table 1 . According to the tag combinations for different shadow scenarios, we give reference suggestions for the corresponding language descriptions. The following section will provide a more detailed explanation of the language expression annotation and validation process.The fundamental methodology and procedure for language annotation in RVSD are following previous works [8, 19, 39] . It employs an interactive approach involving multiple annotators taking turns to annotate and validate. We invited multiple people with no computer vision-related background to participate in labeling with the guidance in our Table 1 . The annotator is required to select one or multiple shadows from the video and generate corresponding referring descriptions according to the guidance for annotating language expressions. It is worth noting that our guidance in Table 1 contains three elements, the recommended , optional •, and not requiredones. In this setup, there is a distinct advantage. While ensuring the accurate depiction of various shadows, annotators have the freedom to select the description's content, thus maximizing the richness and flexibility of the sentences.After the initial annotation, we conduct validation tasks for all annotations. The validation process commences by presenting the video along with the corresponding expression, prompting the validator to identify the shadow referred to in the expression. The validator is required to independently find the target shadow and record it. The shadow chosen by the validator is subsequently cross-referenced with the annotations provided by the annotator. If they align, the annotation passes the validation; otherwise, it undergoes re-labeling and subsequent validation. By implementing the validation procedure, we strive to ensure the accurate language representation of shadows within our dataset, while maintaining sentence diversity.3.2.1 Video and Shadow. The selection of video and shadow is based on diversity, which makes the dataset both comprehensive and challenging. Firstly, the number of shadows is a factor to consider, since RVSD in a scene with multiple shadows generally is more challenging than one with a single shadow. Multiple shadows, often close in location and similar in shape, can lead to confusion during model interpretation, thereby making precise segmentation of a specific shadow challenging. In the RVSD dataset, there are 53 videos containing multiple shadows, and 33 videos containing only one shadow. Next, we consider the type of shadow, namely hard and soft shadow. A hard shadow has a distinct boundary, whereas a "the soft shadow is located below" Example 2: "the hard shadow of a person who is holding an umbrella and walking is in the upper left corner" soft shadow's boundary might be fuzzy or indistinct. Soft shadow is more challenging to segment than hard shadow. The RVSD dataset includes 156 hard shadows and 42 soft shadows. Finally, the diversity of scenes is also an important factor, diverse scenes are closer to the real world, making the dataset more comprehensive. The RVSD dataset includes 74 daytime videos and 12 nighttime videos, as well as 24 indoor videos and 62 outdoor videos.3.2.2 Language Description. One significant characteristic of the RVSD dataset is assigning language descriptions to each shadow on a frame-by-frame basis. Firstly, we incorporate language descriptions for shadows based on the ViSha dataset. The preceding referring object segmentation dataset [39] offered two kinds of language annotations: full-video expression and first-frame expression. The first-frame expression solely relies on the static attributes of the first frame image, whereas the full-video expression takes into account the whole video. However, neither of these expressions can perfectly align with the state of each target in the video (due to the movements of objects within the video), leading to instances where certain frames do not correspond with their language descriptions. Therefore, when adding language descriptions, we add corresponding language descriptions for each shadow target and its corresponding frames, making the content of the language descriptions match each frame. Meanwhile, we've established guidelines for the method of annotating language description, making the description more closely reflect the status of the target. In the language descriptions, we include the type of shadow, followed by the static attributes, the position information of the shadow, and the action information of the object corresponding to the shadow. Table 2 presents the basic statistics of our language descriptions. Fig. 3 displays the word cloud for RVSD, featuring terms associated with shadow type (e.g., "hard shadow", "soft shadow"), actions of the shadow-casting subjects (e.g., "walking", "skateboarding"), position (e.g., "left", "upper right"), and the objects casting the shadows (e.g., "person" or "tree"). 
methodology 4.1 overview: 
 Our approach is designed to generate a binary mask of the specified shadow of the given input video based on the corresponding natural language expression. Fig. 4 presents a schematic overview of our RSM-Net, specifically devised for the RVSD task. RSM-Net takes language descriptions and video frames as input, as depicted in the "Language and Image Integration" box to the right of Fig. 4 . It reads the previously stored Twin-Track Synergistic Memory and processes the image through the MSA module to jointly achieve the accurate segmentation of the target shadow. We first present how the twin-track synergistic memory is constructed and the content of our Triple-Entity memory propagation in Sec. 4.2. Subsequently, Sec. 4.3 is dedicated to a comprehensive exposition of our language and image integration, encompassing aspects of hierarchical memory reading and memory propagation. Lastly, in Sec. 4.4, we describe how our MSA effectively utilizes prior knowledge to facilitate the RVSD task.In response to the dynamic nature of shadows, we incorporate a twin-track memory structure into our framework. This encompasses both past hierarchical inter-clip memory and intra-clip memory, established before processing the current frame. The integration of these sequential memories in our network allows for a nuanced capture of evolving information across the video sequence, significantly contributing to the precise segmentation of the current frame. We emphasize that our hierarchical memory is dynamically updated. Specifically, when processing the current frame 𝑐𝑙𝑖𝑝 𝑖 , we constructed memory using the preceding five clips {𝑐𝑙𝑖𝑝 𝑖 -5 , ..., 𝑐𝑙𝑖𝑝 𝑖 -1 }. This dynamic updating and utilization in memory enhances our network's flexibility and efficiency while avoiding the disturbances from significant variations in very early video frames, allowing it to continuously adapt to the varying characteristics of shadows during the segmentation process. First, our hierarchical inter-clip memory is generated from multiple clip blocks. The details of the clip block are in the yellow box in the lower left corner of Fig. 4 . It mainly contains triple-entity memory propagation. In the triple-entity, the 𝑡𝑟𝑎𝑐𝑘𝑖𝑛𝑔 𝑏𝑜𝑥 𝑡 𝑏𝑜𝑥 ∈ R 𝑁 𝑞 ×4 denotes the predicted shadow detection box, 𝑡𝑟𝑎𝑐𝑘𝑖𝑛𝑔 𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑎𝑡𝑖𝑜𝑛 𝑡 𝑟𝑒𝑝 ∈ R 𝑁 𝑞 ×𝑑 refers to the encoded multi-modal features, and 𝑙𝑒𝑎𝑟𝑛𝑎𝑏𝑙𝑒 𝑞𝑢𝑒𝑟𝑦 𝑡 𝑞𝑢𝑒 ∈ R 𝑁 𝑞 ×𝑑 signifies the shadow query. Our hierarchical inter-clip memory stores 𝑡 𝑟𝑒𝑝 𝑠 from three different time scales {T1,T2,T3}. Meanwhile, we pick up the triple-entity of the previous frame of the currently pending frame from 𝑐𝑙𝑖𝑝 𝑖 -1 as the intra-clip memory. Next, we will elaborate on the processing of the current frame and the efficient retrieval of information from the twin-track memory.In line with earlier researches [2, 49, 52] , our learnable query-based referring segmentation largely follows a well-established paradigm, the Deformable DETR [52] . This involves processing a video frame, a textual expression, and a set of learnable queries. The output includes the target bounding box, segmentation mask, and associated output embeddings that match the input language expression. To process a particular 𝑡 𝑡ℎ frame image 𝐼 𝑡 , our network first performs attention enhancement of potential shadow regions via the MSA module, followed by extracting image feature F 𝑖 via the visual encoder. Meanwhile, the associated language description 𝜀 = {𝑒 𝑖 } 𝑁 𝑖=1 with 𝑁 words, where 𝑒 𝑖 is the 𝑖 𝑡ℎ word is tokenized and fed into the text encoder, yielding F 𝜀 . Next, the visual and linguistic embeddings are linearly projected to a unified embedding space with the same dimension and concatenated to form a multi-modal embedding, denoted as F 𝑚 = {F 𝑖 , F 𝜀 }. This is then input into the transformer encoder, realizing cross-modal information fusion and interaction.In the reading phase, it receives the triple-entity from the preceding frame along with the hierarchical memory, subsequently outputting the 𝐸 𝑚 ∈ R 𝑁 𝑞 ×𝑑 . Initially, we apply memory embedding to the hierarchical memory as shown in Fig. 5 , thereby fusing features across various temporal scales as follows:EQUATIONwhere ΦMLP represents a three-layer MLP (Multi-Layer Perceptron). 𝐶𝑜𝑛𝑐𝑎𝑡 and 𝐹𝐶 stand for concatenated operations and fully connected layers, respectively. The ℎ mem is fed into the Multi-Head Self-Attention module as both the Key (K) and Value (V), while the Query (Q) originates from the intra-clip 𝑙𝑒𝑎𝑟𝑛𝑎𝑏𝑙𝑒 𝑞𝑢𝑒𝑟𝑦 𝑡 𝑞𝑢𝑒 . During the propagation phase, the 𝑡 𝑟𝑒𝑝 advances the output as ΦTo generate the final mask, our segmentation pyramid network implements a cross-modal FPN (Feature Pyramid Network) [25] to enable multi-scale fusion between the linguistic features and visual feature maps (see Sec. 5 for model details). To summarize, the training objective of our network is to minimize the loss function as follows:EQUATIONwhere L box denotes the loss associated with bounding boxes, which is a composite of L1 loss and GIoU loss [37] . L mask signifies the loss pertaining to masks, aggregating DICE loss [35] and the focal loss for binary masks. The coefficients for the losses are denoted as 𝛼 box for L box , and 𝛽 mask for L mask .The premise and foundation of the RVSD task is that the network can recognize and pay attention to the shadow region. Therefore, we design a Mixed-Prior Shadow Attention (MSA) module to utilize the physical prior knowledge [7, 11] to generate a weight map 𝑚 𝑎 to help the network focus more on the shadow region. Given the complexity and variability of the shadow background, our MSA identifies the shadow region from two different color spaces to improve robustness and generalization, as illustrated in Fig. 6 . Grayscale Thresholding. For a given image 𝐼 𝑡 , the grayscale representation is obtained by converting the RGB channels into a single intensity channel 𝐼 gray . Mathematically, the binary shadow mask 𝑀 gray is obtained as:EQUATIONwhere the thresholds [𝑇 min , 𝑇 max ] specify the acceptable range of grayscale values for identifying shadow regions.HSV Thresholding. The input frame 𝐼 𝑡 is also converted to the HSV color space 𝐼 HSV . In the HSV color space, shadow regions typically exhibit lower values in the Saturation (𝑆) and Value (𝑉 ) channels due to color desaturation and reduced brightness caused by occlusion. Thus, two thresholds 𝑇 𝑆 and 𝑇 𝑉 are applied on the Saturation and Value channels respectively to discern the shadow regions. The binary shadow mask 𝑀 HSV is obtained as:EQUATIONThe thresholds 𝑇 𝑆 min ,𝑇 𝑆 max ,𝑇 𝑉 min , and 𝑇 𝑉 max specify the acceptable range for shadow regions in the 𝑆 and 𝑉 channels.The combined shadow mask is then obtained by: 𝑀 combined (𝑥, 𝑦) = Ψ 𝑚𝑜𝑟 (𝑀 HSV (𝑥, 𝑦)) ∪ Ψ 𝑚𝑜𝑟 (𝑀 gray (𝑥, 𝑦)). (5) The morphological operation (Ψ 𝑚𝑜𝑟 ) here specifically refers to the "opening" operation function with a kernel of size 5 × 5. This operation comprises an erosion followed by a dilation, which helps to remove noise and small interference while keeping the structure of the referring shadow. The final combined mask 𝑀 combined is then obtained by taking the union (logical OR operation denoted by ∪) Pub. P@0.5↑ P@0.6↑ P@0.7↑ P@0.8↑ P@0.9↑ Overall↑ Mean↑ mAP↑ URVOS [ of these morphologically processed masks. The 𝑀 combined is then weighted as attention map 𝑚 𝑎 on the input frame 𝐼 𝑡 . 
experiments 5.1 evaluation metrics and data setting.: 
 Following referring segmentation works [2, 9, 44] , we employ standard metrics for quantitative comparisons. These metrics are Precision@K, Overall IoU, Mean IoU, and mAP (mean Average Precision) across an IoU (Intersection over Union) range from 0.50 to 0.95 (a step of 0.05). IoU measures the overlap between predicted and ground truth regions, while precision@K evaluates the proportion of test instances surpassing the IoU threshold of K. The mAP metric computes the mean precision over varying IoU thresholds. To ensure a rich variety of scenarios in both the training and testing phases, the dataset is reasonably divided into training and testing subsets. The training subset encompasses 54 videos paired with 9,856 sentence-shadow combinations, while the testing subset contains 32 videos accompanied by 5,155 pairs.We utilize ResNet50 [13] as the visual backbones for extracting features. For the text encoding, we employ RoBERTa [30] and freeze its parameters throughout the training process. Following [48, 52] , the final three-stage features from our visual backbone serve as inputs for both the Transformer encoder and the FPN [25] . We perform all experiments using PyTorch on an NVIDIA GeForce RTX 3090 GPU with 24 GB of memory for training. Our model optimization employs the AdamW optimizer [32] with an initial learning rate set at 1𝑒 -5, while the visual backbone is adjusted at a lower rate of 5𝑒 -6. The training spans 20 epochs, and the featuring learning rate reductions by a factor of 0.1 after the 3rd and 5th epochs. The initial frame's query number 𝑁 𝑞 is set to 5. Each video frame is resized to ensure a minimum dimension of 320 on the shorter side and a maximum of 576 on the longer side.We compare our network against state-of-the-art methods on the RVSD dataset. These compared methods include URVOS [39] , CMPC-V [28] , LBDT [9] , MTTR [2] , ReferFormer [49] , SgMg [34] , R2VOS [23] and OnlineRefer [48] . In Table 3 , we report the quantitative performance of our network and eight referring video segmentation methods on the RVSD datasets. According to Table 3 , we can find that our RSM-Net clearly outperforms all compared methods, since our RSM-Net achieves the highest scores across nearly all evaluated metrics. Moreover, the last row ("Gain") in Table 3 highlights our method's improvements over competing approaches, and our network has an Overall IOU increase of 4.4%.Moreover, Fig. 7 shows the visual comparisons of RVSD results produced by our network and other state-of-the-art methods. Apparently, with the same text descriptions and input video frames, our network can better identify the referred shadow regions than the compared methods, and our results are more consistent with the ground truths. For example, the first, second, and third rows of Fig. 7 show that our method can accurately segment the described region when there are multiple shadow regions present, whereas other methods (MTTR [2] in the second and third rows) may segment irrelevant shaded regions or may fail to recognize the corresponding shaded region (ReferFormer [49] and SgMg [34] in the first row). The fourth and fifth rows illustrate the segmentation for the shadow of interest through text descriptions when the object is not entirely visible or absent in the image, a feat not achievable with current instance shadow segmentation techniques. Our results can be observed to more closely align with the ground truths.We further conduct ablation studies to validate the effectiveness of our MSA and TSM designs. To do so, we construct a baseline (denoted as "Baseline") by eliminating our MSA and TSM from our RSM-Net. Subsequently, we incrementally integrate MSA and TSM into the "Baseline" to formulate four networks, which are denoted as "M1", "M2", "M3", and our RSM-Net. Table 4 reports the quantitative results of these networks. Apparently, "M1" outperforms "Baseline", showcasing the effectiveness of the coarse shadow mask attention in our MSA for RVSD. The improvement in metrics from "M2" to "M1" demonstrates the contribution of the intra-clip memory features in our network. The progression from "M2" to "M3" with greater metric results further suggests that the incorporation of inter-clip features at a single temporal scale can bolster RVSD. while the larger metric results of "M3" than "M2", which further indicates that the inter-clip features at a single temporal scale can also enhance RVSD. In the end, our network has a superior performance over "M3", which demonstrates our hierarchical inter-clip features enable a better RVSD performance in our network. 
conclusion: 
 In this study, we pioneer the RVSD task, which integrates linguistic prompts with video shadow detection, paving the way for new potential applications, such as interactive video editing. Our first contribution is the development and annotation of the dataset for RVSD, comprising 86 videos paired with 15,011 text descriptions and corresponding shadow masks. Furthermore, we devise a Twin-Track Synergistic Memory (TSM) module to learn intra-clip and hierarchical inter-clip memory features to boost segmentation performance and a Mixed-Prior Shadow Attention (MSA) module to learn a coarse shadow attention map for refining shadow areas for RVSD. Experimental results demonstrate that our method achieves better performance than other leading comparative methods. 
: 
 promising results [6, 10, 15, 17, 24, 53] . However, these methods do not allow for the segmentation of specific shadows described by users, due to their lack of interactivity and flexibility which is essential in the era of advanced multimedia interaction.Therefore, we propose Referring Video Shadow Detection (RVSD) as a novel task dedicated to meeting this demand by interactively segmenting the specific shadows in videos based on descriptive natural language prompts, as illustrated in Fig. 1 . The significance of RVSD can be envisaged in numerous applications. Advanced video editing tools can benefit from this RVSD task by allowing editors to precisely manipulate shadows based on verbal instructions [5, 22] . In augmented and virtual reality environments, understanding and reacting to user-specified shadows can enhance immersion and realism [1, 31] . Furthermore, remote sensing may also benefit from this RVSD task to achieve more precise results [29] .In this work, we introduce the first dataset dedicated to the RVSD task. This dataset collected 86 videos with 15,011 paired rich textual descriptions and corresponding shadow mask annotations, with some examples showcased in Fig. 2 . It has been meticulously curated to cover a wide array of scenarios, and shadow dynamics, which can provide a comprehensive benchmark for evaluating future RVSD methods. To the best of our knowledge, our dataset is the first one for RVSD with high-quality annotation.Compared to classical referring video object segmentation [2, 39] , achieving an accurate RVSD is more challenging. Firstly, shadows lack rich appearance features and are easily confused with the dark regions of the input video [3] . Moreover, shadows are naturally influenced by ever-changing environmental factors, demonstrating severe temporal shape transformations. Hence, RVSD requires richer temporal information and physical prior information to accurately identify the specific shadow referred by users' descriptions.Based on our annotated RVSD dataset, we develop a novel Referring Shadow-Track Memory Network (RSM-Net) for addressing the RVSD task. In our RSM-Net, we devise a Twin-Track Synergistic Memory (TSM) to store hierarchical inter-clip features and intraclip features, and then propagate these two memory features via a memory read module to refine the current video frame for video shadow detection. Moreover, we propose a Mixed-Prior Shadow Attention (MSA) module that leverages physical knowledge to generate a preliminary shadow map, guiding the network to focus on potential shadow regions attentively.Our main contributions are summarized as follows:• This work is the first one to explore the RVSD task, which presents a fresh paradigm for language-driven interactive and flexible shadow detection with potential benefits for numerous downstream tasks. • To address the RVSD task, we collect and annotate the first dataset for RVSD consisting of 86 videos with 15,011 paired video frames and the corresponding text descriptions. This is the first dataset dedicated for the RVSD task. • We propose an RSM-Net for RVSD. Here, we devise a TSM module to learn intra-clip and inter-clip features and store both of them in a memory to refine the current video frame for video shadow detection. Moreover, an MSA module is developed to generate the coarse shadow map for focusing on potential shadow areas for RVSD. • Extensive experimental results show that our RSM-Net clearly outperforms state-of-the-art methods for the RVSD task. 
 
 Appendix:  


Decision: After rebuttal, all of the reviewers gave their comments towards recommending this paper be accepted. The AC also agrees with them. However, the authors are supposed to add a discussion about Reviewer hUQR's question -- How to define "blurry" and "clear". Your answer is too subjective. Besides, why can shadows be "stable" and "moving"?

Reviews: review: paper_summary: paper_summary: This paper pioneers an interesting task, termed as Referring Video Shadow Detection (RVSD), to flexibly and interactively identify and segment the object shadows with textual expressions. To this end, a novel and large-scale benchmark dataset is collected and produced. Besides, to achieve this task, an innovative baseline architecture is proposed which is comprised by two modules: Twin-Track Synergistic Memory (TSM) and Mixed-Prior Shadow Attention (MSA). These two modules respectively account for storing memory features and obtaining coarse shadow maps.
strength_weakness: strength_weakness: The English usage of this paper is fluent and well. Figures and Equations are illustrated accurately and unambiguously. Besides, from the showcase of empirical results, the proposed approach is not only solid but also effective and efficient.
limitations: limitations: Although impressive, there are also some confusing and erroneous aspects needed to give more detailed explanations. 

1. Why needing the consideration of hard and soft shadow. Although giving coarse explaination corelating the meanings of hard and soft shadow, it still makes me confused about these two concepts. It is better given a more detailed explaination or illustration with the help of drawing figure.
2. There are an increasing number of unexplained names in Table 1. For example, what are the meanings of stable, moving, and stable and moving. 
3. It is said "neither of these expressions can perfectly align with the state of each target in the video (due to the movements of objects within the video)" in section 3.2.2, why the full-video expression is inability to align with the state of each target, does the full-video expression contain no movement information?
4. It is better to integrate Mixed-Prior Shadow Attention (MSA) into the overall architecture in Fig. 4 to perfect the entire structure.
5. Table 4 is disorganized. The names in caption is dismatched with the table. For example, what are the InterC-S and InC-H?
paper_summary: paper_summary: The authors innovatively propose the concept of Referring Video Shadow Detection, which segments specific shadows in videos based on natural language descriptions. They specifically created a well-annotated RVSD dataset, laying a solid foundation for this new field of research, and introduced the RSM-Net to address the RVSD task.
strength_weakness: strength_weakness: The paper introduces a novel task, RVSD (Referring Video Shadow Detection), which is innovative and potentially impactful for applications in video editing, virtual reality, and more. It is evident that the authors have undertaken significant efforts, including compiling a dataset for the RVSD task, proposing the RSM-Net to tackle this challenge, and thoroughly evaluating the method's effectiveness through experiments. This manuscript is well-structured. The clear demarcation of sections and coherent presentation greatly enhance the readability of the paper.
limitations: limitations: I think the paper still has the following issues:
1. Page 3, left column, line 263: “After reviewing 120 potential candidates, we carefully exclude shadows that are blended together (hard to differentiate them through textual descriptions). We also remove highly fragmented shadows.”.
You should provide specific examples of the excluded shadows. This would help readers understand which types of shadows cannot be detected by this method, thus clarifying the limitations of the approach.
2. Table 1 mentions moving and stable shadows. You should provide the distribution of these two types of shadows in Section 3.2.1.
3. In Section 5.3, the proposed method is compared with several referring video object segmentation methods. Since current referring video object segmentation methods can already address the problem of RVSD, I hope you can elucidate the most significant breakthrough of the RVSD method compared to these referring video object segmentation methods. Is it the enhanced specificity for shadows?
4. The proposed method segments specific shadows based on natural language descriptions, but there is no mention of whether variations in the complexity of language descriptions impact the accuracy of shadow segmentation.
paper_summary: paper_summary: This paper presents a new shadow detector task: Referring Video Shadow Detection (RVSD), and they curated a well-annotated RVSD dataset. The authors proposed a novel Refer-Ring Shadow-Track Memory Network (RSM-Net) to solve the RVSD task. In RSM-Net, the Twin-Track Synergistic Memory (TSM) module and the Mixed-Prior Shadow Attention (MSA) module are used to learn frame features and guide the network to pay attention to potential shadow areas. Some experiments show that RSM-Net is better than existing methods in RVSD tasks.
strength_weakness: strength_weakness: The authors collected and annotated the first RVSD dataset, which includes 86 videos and 15,011 frame-text description pairs, they proposed a novel method (RSM-Net) to solve the RVSD task, and experiments show that RSM-Net is better than the latest methods in the RVSD task.
limitations: limitations: 1. Authors present the Referring Video Shadow Detection task, but they don’t explain the prospects of the task in detail with related downstream applications. 
2. There seem to be no indoor scenes in the type of scenario of the RVSD dataset, and it’s difficult to determine the generalization of the method in indoor scenes. 
3. There is an online method in the compared methods, so it’s best to show the runtime efficiency of the proposed method.
4. The proposed method should be tested in other shadow detector datasets except the RVSD dataset.
5. Most of the compared methods are video object segmentation methods, it’s best to compare with other shadow detector methods.
paper_summary: paper_summary: This paper proposes a new task, namely Referring Video Shadow Detection (RVSD), which detects arbitrary shadow areas of interest based on descriptions in natural language prompts. Meanwhile, it provides a well-annotated RVSD dataset to validate the performance of methods for the RVSD task. Also, this paper proposes a network called RSM-Net to solve RVSD. This network comprises two modules, namely Twin-Track Synergistic Memory (TSM) and Mixed-Prior Shadow Attention (MSA). TSM stores intra-clip memory features and hierarchical inter-clip memory features to refine features of the current video frame for referring shadow detection. MSA utilizes physical priors to generate a coarse shadow map, which is then weighted with the input video frame to facilitate the learning of more visual features.
strength_weakness: strength_weakness: 1. A meaningful new task, RVSD, is proposed along with the corresponding dataset. The rationale behind the dataset is explained from multiple perspectives.
2. The compared methods are relatively new, and the network proposed in this paper shows significant performance improvements on the RVSD task.
3. The workload is substantial.
limitations: limitations: 1.Some tags in Table 1 are confusing, such as 'Only shadow' and 'Shadow and object'. What do they respectively indicate?
2.In Figure 4, certain modules are not clearly explained. For example, where is the clip block reflected in the overall architecture? How is the temporal scale obtained? Additionally, which losses correspond to Lbox and Lmask in Equation 2?
3.Some experiments regarding model efficiency are missing, such as inference time.
4.Can you give another experiment that the input text is “all shadows in the picture” and compare the results with other SOTA methods?
5.Can you present the effect of each module, namely MSA and TSM using visualization?