SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-11 14:54:03

Abstract: Sign language retrieval, as an emerging visual-language task, has received widespread attention. Different from traditional video retrieval, it is more biased towards understanding the semantic information of human actions contained in video clips. Previous works typically only encode RGB videos to obtain high-level semantic features, resulting in local action details drowned in a large amount of visual information redundancy. Furthermore, existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training, and adopt offline RGB encoder instead, leading to suboptimal feature representation. To address these issues, we propose a novel sign language representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which integrates Pose and RGB modalities to represent the local and global information of sign language videos. Specifically, the Pose encoder embeds the coordinates of keypoints corresponding to human joints, effectively capturing detailed action features. For better context-aware fusion of two video modalities, we propose a Cross Gloss Attention Fusion (CGAF) module to aggregate the adjacent clip features with similar semantic information from intra-modality and inter-modality. Moreover, a Pose-RGB Fine-grained Matching Objective is developed to enhance the aggregated fusion feature by contextual matching of fine-grained dual-stream features. Besides the offline RGB encoder, the whole framework only contains learnable lightweight networks, which can be trained end-to-end. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods on How2Sign, PHOENIX-2014T, and CSL-Daily datasets.

Main paper: 
 introduction: 
 Sign language is the primary means of communication for people with hearing impairments. Sign language understanding is an important research field dedicated to overcoming communication barriers between hard-of-hearing individuals and non-signers. The classic sign language comprehension task [11, 12, 18, 21, 26, 27, 48, [57] [58] [59] 64] includes sign language recognition and translation, intending to identify the gloss sequence in sign language videos and further translate it into natural language. In this paper, we focus on the emerging sign language retrieval task [10, 15] . Unlike sign language recognition and translation, sign language retrieval task is targeted at retrieving videos or texts that are most similar to the query from a video or text database.As a special case of traditional text-video retrieval (TVR) tasks [14, 17, 19, 30, 39-41, 43, 55, 63] , sign language retrieval comprises two sub-tasks, i.e., text-to-sign-video (T2V) retrieval and sign-video-totext (V2T) retrieval. Unlike traditional TVR tasks, sign TVR focuses on the semantic information of human actions that correspond to natural language words contained in video clips (Fig. 1(b) ), rather than descriptive information of objects in video frames (Fig. 1(a) ). Sign language retrieval task is more similar to translating a language expressed through video clips, and then conducting cross-language retrieval [10] . Therefore, in sign language videos, a video clip containing a single action is the smallest unit of complete information, which can be defined as a sign visual word. This requires the presence of a sign encoder in the model to extract action semantic information from video clips.Recently, as a common practice for text-to-video retrieval models, contrastive loss [22] is involved in training, which significantly improves performance by utilizing visual prior knowledge from the pre-trained CLIP vision transformer [51] . In contrastive learning, multiple negative samples are required in a single batch. However, previous sign language retrieval works, such as SPOT-ALIGN [15] and CiCo [10] , use RGB-based sign encoders that embed dense visual data. The massive memory cost results in two-stage training instead of end-to-end training, which potentially reduces the quality of the feature representation. Meanwhile, sign language usually utilizes two types of signals, i.e., global visual signals including body position and facial expressions, and local action signals including hand actions and palm movements. Therefore, the sign encoder needs to pay attention to both the semantics of the global visual information and the local action information. In contrast, RGB encoders always embed global visual signals to obtain high-level semantic features, leading to local action details drowned in a large amount of visual information redundancy. Apart from the lack of local information, such bias also results in potential robustness issues, affecting the performance of model due to the difference in video backgrounds or signers between the training and test sets.In order to alleviate the above issues, inspired by previous sign language tasks [18, 24, 29, 35] , we introduce Pose modal knowledge to sign language retrieval task and propose a new framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which includes an online Pose encoder and an offline RGB encoder extracting features from two different perspectives. Compared to the offline RGB encoder, the remaining modules within the framework such as Pose encoder, fusion module and interaction transformers, are lightweight enough to be trained end-to-end, which effectively improves the feature quality of sign language videos. The introduction of Pose keypoints of both hand parts and entire body skeleton not only enables the Pose encoder to supplement the details of local action movement, but also effectively mitigates the bias for similar visual scenes introduced by the RGB modality. For the process of fusing RGB and Pose modalities, inspired by recent work GASLT [60] , we propose the Cross Gloss Attention Fusion (CGAF) module. This module keeps its attention on video clips locally, aggregating adjacent clips with similar semantic information from intra-modal and inter-modal. This enhances the semantic representation of sign visual words in video clips.To ensure optimal retrieval performance through fusion features, we propose an explicitly supervised Pose-RGB fine-grained matching objective. This objective performs contextual matching on fine-grained dual-stream features, matching the corresponding clip features in the two modalities. Therefore, it ensures that the same clips in both modalities have a higher chance of matching the same word feature with more similar semantics. The distribution of the fine-grained similarity matrices of Pose-Text and RGB-Text are implicitly aligned, allowing the CGAF module to fully consider the representations from both modalities during the fusion process.The main contributions of our work are summarized as follows:• To address the limitation of previous sign language retrieval works, we propose a framework called Semantically Enhanced Dual-Stream Encoder (SEDS) that first introduces Pose modality knowledge into sign language retrieval task.  
related work 2.1 sign language understanding: 
 Sign language understanding aims to fully understand the semantic information in sign videos and connect it with natural language. The current mainstream sign language understanding tasks include isolated sign language recognition [25, 33-35, 61, 68] , continuous sign language recognition [11, 12, 21, 26, 27, 48] , sign language translation [18, 57-59, 64, 66] , sign language spotting [1, 20, 46] , and our focused sign language retrieval [10, 15] task.As the foundation of sign language understanding, sign language recognition task aims to convert an input sign language video into a gloss sequence, which corresponds to the sign visual words in natural language. The encoder used for extracting sign language video features is mainly based on the CNN [32] architecture, including 3D-CNN [8, 36, 49] and 2D+1D CNN [13, 66] . The sign language spotting task [1, 20, 46] is a variant of the sign language recognition task, aimed at locating and recognizing sign word instances in an untrimmed video. The spotting features can be used as features of sign visual words, matching with the sign language annotation text at a fine-grained level. Therefore, we leverage the I3D [6] network pre-trained on BSL-1K [20] for sign spotting as offline RGB encoder.For sign language understanding tasks, there are two types of methods: gloss-based [5, 8, 9, 53, 65, 66] and gloss-free [2, 36, 44, 60, 62] methods. The difference between them is whether or not to use gloss annotations as auxiliary supervision. Recently, GASLT [60] argues that gloss plays a role in restricting attention to local sequences in sign language tasks, which is more consistent with logicality of sign language videos. Inspired by it, we propose Cross Gloss Attention Fusion (CGAF) module, which aggregates local information with similar semantics from intra-modal and inter-modal.Pose is an important modal information in the process of sign language video understanding, which is generally extracted offline by the Pose extraction model [28, 52, 56] . Compared to RGB data, Pose has less redundant information and is more targeted towards the position and motion information of human movements. There are many methods using Pose information. Works like [47, 66] RGB videos and feature maps into different parts based on Pose keypoints. Some other works [9, 67] use Pose heatmap to supplement the human skeleton information in RGB, while sharing the architecture with the RGB encoder. However, due to their strong correlation with RGB, these methods still cost excessive memory. Following previous work [4, 24, 29] , we use Graph Convolutional Networks (GCN) to model Pose keypoints into a graph, defining edges based on the connections between human skeleton joints. This allows us to extract features that represent precise local position and motion information of the human body. Furthermore, the sparse input of Pose keypoints enables end-to-end training.Traditional text-video retrieval [14, 17, 19, 30, 39-41, 55, 63] is a fundamental vision-language task, which primarily focuses on videos of more general daily life scenarios. Early works [14, 17, 39, 40, 55] mainly align the features of text and video by training text and video encoders from scratch. Due to the great success of the large-scale image-text pre-training model CLIP [50] , many CLIP-based text-video retrieval works [19, 30, 41, 63] have achieved significant performance improvements.Sign language text-video retrieval can be seen as a special case of text-video retrieval. CLIP-based text-video retrieval [43] mainly involves sparse frame sampling of the original video, but sign language videos often require multiple frames to express a sign visual word. Therefore, before sending it to the vision transformer, a sign encoder is needed to extract clip features. Lots of CLIP-based works [37, 38, 45, 54] have proposed multi-grained modal alignment methods, including both supervised coarse-grained and finegrained feature alignment, fine-grained aggregation feature alignment, etc. However, our proposed Pose-RGB fine-grained matching objective is different from these methods. It matches the contextual features of two video modalities at a fine-grained level, implicitly aligning the distribution of fine-grained similarity matrices between Pose, RGB and Text. This allows for a full consideration of the characteristics of both modalities during the fusion process. 
method: 
 In this section, we introduce our Semantically Enhanced Dual-Stream Encoder (SEDS) framework (Fig. 2 ), which is composed of three parts: 1) Pose and RGB Feature Extraction Module, which extracts local action information and global visual information.2) Cross Gloss Attention Fusion (CGAF) Module, which fuses the adjacent clip information from both inter-modal and intra-modal perspectives. 3) Pose-RGB Fine-grained Matching Objective, ensuring the contextual alignment of fine-grained dual-stream features.Sampling Clips in Sign Video. Sign language videos and natural language serve as two distinct modalities of expression with different word orders, yet they convey the same semantics. The fundamental sign visual words depicted in sign language videos should correspond with words in natural language. For a single video, we begin by filtering out low-quality video frames, such as those with unclear or missing hand gestures and incomplete signers. Subsequently, we utilize a sliding window approach with a window size of 16 and a sliding interval of 1 to generate a number of video clips, and then select 𝑇 clips at equal intervals in time dimension. Each video clip is then fed into online Pose and offline RGB sign encoders, resulting in Pose features 𝒇 𝑝′ ∈ R 𝑇 ×𝐷 and RGB features 𝒇 𝑟 ′ ∈ R 𝑇 ×𝐷 for this sign video. Next, we feed these two sequence features separately into two Transformer encoders for interaction in their respective modalities, finally obtaining features 𝒇 𝑝 ∈ R 𝑇 ×𝐷 and 𝒇 𝑟 ∈ R 𝑇 ×𝐷 for the further fusion. Online Pose Encoder. In order to obtain the Pose data in the video, we use the open-source RTMPose [28] to extract the Pose data from the original RGB sign language video. To better utilize the motion information of different skeletal parts of the human body, we divide the extracted Pose keypoints into three groups: 21 keypoints 𝐺 𝐿 for left hand, 21 keypoints 𝐺 𝑅 for right hand, and 7 keypoints 𝐺 𝐵 for body. We use two GCN network modules for feature extraction of Pose data and one 1D convolutional fusion module for information fusion in the time dimension in Pose encoder. We share the same GCN network for both hands. In the GCN module, we use edges to represent the connections of the human skeleton in hand and body group, further constructing the group-specific adjacency matrix 𝑨 𝐺 . The single-layer convolution operation for a specific group of keypoint features is as follows:EQUATIONwhere 𝒇 𝐿 𝑠 denotes the 𝐿-layer feature vector of the corresponding 𝑠 frame, and the feature vector is the original coordinates of keypoints when 𝐿 = 0. 𝑾 𝐿-1 𝐺 is the 𝐿 -1-layer weight matrix for performing feature linear transformation, and 𝜎 is a nonlinear transformation. 𝐼 represents the self-connection of points. 𝚲 𝑖𝑖 𝐺 is the normalized diagonal matrix 𝑗 (𝑨 𝑖 𝑗 𝐺 + 𝑰 𝑖 𝑗 ). After extracting spatial features from all frames of a single video, we obtain three groups of features 𝒇 𝑠 𝐿 , 𝒇 𝑠 𝑅 , 𝒇 𝑠 𝐵 ∈ R 𝐹 ×𝐷 , where 𝐹 represents the number of frames. Then we concatenate them into feature 𝒇 𝑠 ∈ R 𝐹 ×3𝐷 . Next, based on the pre-defined video clip boundaries, we sample these frame features into many sets of clip features 𝒇 𝑠′ ∈ R 𝑇 ×16×3𝐷 containing 16 frame features. In the temporal dimension, we use a 1D convolutional fusion module to aggregate information within clips. After information aggregation, the features 𝒇 𝑝′ ∈ R 𝑇 ×𝐷 will be ultimately fed to a Transformer encoder for intra-modal information interaction.Offline RGB Encoder. Meanwhile, there have been significant advances in pre-trained sign spotting [20, 46] research. When applied to downstream tasks, these RGB-based models [20] greatly enhance the understanding capabilities of convolutional neural networks for sign language videos. Therefore, we employ an I3D network pre-trained on BSL-1K [1] as the offline RGB encoder to extract 𝒇 𝑟 ′ ∈ R 𝑇 ×𝐷 features. If the RGB encoder performs end-toend learning simultaneously with the Pose encoder, the semantics of specific sign language video can be better captured. However, in the actual training process, extracting features from such contrastive video samples consumes a considerable amount of memory. Consequently, we freeze the pre-trained I3D network and extract features from each sign language video clip offline. For the process of fusing Pose and RGB modalities, there are some naive fusion methods such as concatenation or cross-attention. However, those naive methods ignore the logic and coherence of sign language videos. Specifically, the semantic relevance of adjacent clips in sign language videos is significantly higher than distant clips. Inspired by a new attention mechanism called gloss attention [60] in recent sign language translation tasks, we propose a Cross Gloss Attention Fusion (CGAF) module. Compared to simple addition or concatenation, our module emphasizes the attention interaction between local information rather than global information including irrelevant distant features, allowing for context-aware integration of features from inter-modal and inter-modal.The CGAF Module combines the Pose modality and RGB modality features of sign language videos in two main stages. The first stage aggregates local information between the two modalities, while the second stage merges the features obtained after aggregation. In the first stage, we start by obtaining the Pose vectors 𝒒 𝑝 , 𝒌 𝑝 , 𝒗 𝑝 ∈ R 𝑇 ×𝐷 and RGB vectors 𝒒 𝑟 , 𝒌 𝑟 , 𝒗 𝑟 ∈ R 𝑇 ×𝐷 , which are computed as linear projections of the Pose modality feature𝒇 𝑝 and RGB modality features 𝒇 𝑟 . Similar to traditional cross attention modules, we divide these two groups of vectors into two groups {𝒒 𝑝 , 𝒌 𝑟 , 𝒗 𝑟 } and {𝒒 𝑟 , 𝒌 𝑝 , 𝒗 𝑝 }. Taking group {𝒒 𝑝 , 𝒌 𝑟 , 𝒗 𝑟 } as an example, we first obtain 𝑇 × 𝑁 constant attention positions 𝑷 𝑝 ∈ R 𝑇 ×𝑁 for each query in 𝒒 𝑝 . Then we compute 𝑁 dynamical offset through 𝑶 𝑝 = 𝑾 𝑝 𝑜 𝒒 𝑝 ∈ R 𝑇 ×𝑁 ,EQUATIONwhere 𝒉 𝑝 ∈ R 𝑇 ×𝐷 is the attention vector obtained after cross gloss attention calculation. The method for calculating attention vectors for group {𝒒 𝑟 , 𝒌 𝑝 , 𝒗 𝑝 } is also the same. In the first stage, we use two layers of cross gloss attention to obtain the final output features f 𝑝 , f𝑟 ∈ R 𝑇 ×𝐷 , as shown in Fig. 3 .In the second stage, the dual stream features f 𝑝 and f𝑟 are concatenated and fed into a Multi-layer Perceptron (MLP) for interaction. Then these three features are added together to obtain the final feature 𝒇 𝑣 . The entire operation is as follows:EQUATIONwhere [• , •] represents the concatenation operation, 𝒇 𝑣 is the feature obtained after fusion.Text-Video Cross-Modal Alignment. For each sign language video, we have obtained three sets of features: Pose features 𝒇 𝑝 , RGB features 𝒇 𝑟 , and Fusion features 𝒇 𝑣 from the CGAF Module.For annotation of length 𝐿, we use a Text encoder initialized with CLIP for feature extraction, resulting in word features 𝒇 𝑤 ∈ R 𝐿×𝐷 . According to the CiCo [10] approach for text-video cross-modal alignment, we start the matching process by choosing clip features 𝒇 𝑘 𝑖 and word features 𝒇 𝑤 𝑗 , and computing a fine-grained similarity matrix 𝑬 𝑘 (𝑖, 𝑗) = 𝒇 𝑘 𝑖 • [𝒇 𝑤 𝑗 ] T ∈ R 𝑇 ×𝐿 , where 𝑘 ∈ {𝑝, 𝑟, 𝑣 } indicating three types of video features. Subsequently, we normalize the columns and rows of 𝑬 𝑘 (𝑖, 𝑗) using the softmax function to obtain 𝑬 𝑘 𝑐𝑜𝑙 (𝑖, 𝑗), 𝑬 𝑘 𝑟𝑜𝑤 (𝑖, 𝑗), and derive 𝑬 𝑡 2𝑘 (𝑖, 𝑗) = 𝑬 𝑘 (𝑖, 𝑗) • 𝑬 𝑘 𝑐𝑜𝑙 (𝑖, 𝑗) and 𝑬 𝑘2𝑡 (𝑖, 𝑗) = 𝑬 𝑘 (𝑖, 𝑗) • 𝑬 𝑘 𝑟𝑜𝑤 (𝑖, 𝑗). Finally, summing the elements across columns and rows yields 𝑬 ′ 𝑡 2𝑘 (𝑖, 𝑗) ∈ R 𝐿 and 𝑬 ′ 𝑘2𝑡 (𝑖, 𝑗) ∈ R 𝑇 , whose averages are used as the similarity scores between text and video, 𝑴 𝑡 2𝑘 (𝑖, 𝑗) and 𝑴 𝑘2𝑡 (𝑖, 𝑗).For a set of 𝐵 text-video pairs, this method efficiently generates a pair of matrices that capture the text-video and video-text similarities, denoted as 𝑴 𝑡 2𝑘 , 𝑴 𝑘2𝑡 ∈ R 𝐵×𝐵 . We employ the InfoNCE loss [23] as text-video cross-modal alignment training objective, which is as follows:L 𝑡 2𝑘 = - 1 𝐵 𝐵 ∑︁ 𝑖=1 log exp 𝜏 • 𝑴 𝑖𝑖 𝑡 2𝑘 𝐵 𝑗=1 exp 𝜏 • 𝑴 𝑖 𝑗 𝑡 2𝑘,L 𝑘2𝑡 = - 1 𝐵 𝐵 ∑︁ 𝑖=1 log exp 𝜏 • 𝑴 𝑖𝑖 𝑘2𝑡 𝐵 𝑗=1 exp 𝜏 • 𝑴 𝑗𝑖 𝑘2𝑡 , L 𝑡 -𝑘 = 1 2 (L 𝑡 2𝑘 + L 𝑘2𝑡 ) ,where 𝜏 is a learnable temperature parameter. For the three groups of the similarity matrices, we calculate the InfoNCE loss function separately and then sum them:EQUATIONwhere 𝛼 is used to balance the proportion of auxiliary loss in textvideo alignment. During the actual inference process, we only use the fusion features to match the text.Pose-RGB Fine-grained Matching Objective. The Pose modality branch and RGB modality branch are essentially learned independently. This leads to the significant difference between the distributions of Pose-Text and RGB-Text fine-grained matching matrices, which results in poor fusion performance due to the same clips between modalities may match with different word features. To address the issue, explicit supervision on fine-grained matching is required. However, the absence of clear labels in fine-grained clipword matching is a problem. Therefore, we utilize the two video modality features for auxiliary supervision. EQUATIONEQUATIONwhere 𝑺 𝑝2𝑟 is the Pose-to-RGB similarity matrix, and 𝑺 𝑟 2𝑝 is the RGB-to-Pose similarity matrix. 𝑑𝑖𝑎𝑔[•] represents taking only diagonal elements of a matrix. The summing of the diagonals in the fine-grained similarity matrix is conducted because the elements on the diagonals represent the similarity between different modalities for the corresponding clips. So this process can be considered as supervised learning with clip-level labels. As a result, the corresponding clips in the Pose and RGB modalities are matched, and their features are aligned. This increases the likelihood of the same clips in both modalities matching the same word features, implicitly aligning the distributions of the fine-grained similarity matrices of Pose-Text and RGB-Text.Upon acquiring video-video level similarity, the InfoNCE [23] loss is employed. Like the loss of text-video alignment, the loss of video-video alignment L 𝑝 -𝑟 consists of two parts, L 𝑝2𝑟 and L 𝑟 2𝑝 . These two parts correspond to 𝑺 𝑝2𝑟 and 𝑺 𝑟 2𝑝 . The whole process of Pose-RGB fine-grained matching objective is shown in Fig. 4 .Joint Loss Optimization for SEDS. We have established the text-video cross-modal alignment objective and Pose-RGB finegrained matching objective, the overall loss function during training Model T2V V2T R@1↑ R@5↑ R@10↑ MedR↓ R@1↑ R@5↑ R@10↑ MedR↓ SA-SR [15] 18.9 32. of our framework is as follows:EQUATION)where 𝛽 is the proportion of Pose-RGB fine-grained alignment loss. 
experiments 4.1 setups: 
 Datasets. Based on the dataset settings of previous sign language video retrieval work [10, 15] , we evaluate our model on the How2Sign [16] , CSL-Daily [65] , and PHOENIX-2014T [3] [10, 15, 19, 41, 43] , we use standard text-video retrieval metrics to measure retrieval performance, specifically Recall at K (R@K, higher is better) and Median Rank (MedR, lower is better). R@K denotes the proportion of correct results retrieved among the top K videos. We use K=1,5,10 in actual evaluation. MedR represents the median ranking of the correct options for all queries.Implementation Details. The offline RGB encoder is an I3D [7] network pre-trained on BSL-1K [20] . The online Pose encoder consists of two GCN modules and a temporal convolution module. The GCN module of both hands is initialized using Signbert [24] . For the RGB Interaction Transformer and Pose Interaction Transformer, we set the number of layers to 12 with a hidden size of 768 and initialize it with the image encoder of CLIP (ViT-B/32) [50] , but the input 1D convolution kernel is changed to the 1024 and 1536 channels corresponding to the RGB encoder and Pose encoder, then the feature channel is reduced to the same 512 channel as the vision token. For the Text encoder, we use the CLIP (ViT-B/32) text encoder for initialization. For all the datasets, we change the frame rate of sign language videos to 24 FPS, sample a maximum of 64 clips per video, and set the maximum length of words to 32. We use an Adam [31] optimizer with a cosine warm-up strategy [42] . During the training process, we freeze the offline RGB encoder, set the learning rates of the online Pose encoder and Cross Gloss Attention Fusion module to 1e-4, and set the learning rates of two Interaction Transformers and Text encoder to 1e-5. We set the batch size to 128 and train for 200 epochs. The hyper-parameters 𝛼 in Eq. 4 is 0.8 and the 𝛽 in Eq. 7 is 0.4.T2V V2T R@1↑ R@5↑ R@10↑ MedR↓ R@1↑ R@5↑ R@10↑ MedR↓ SA-SR [15] 30. 2 Model T2V V2T R@1↑ R@5↑ R@10↑ MedR↓ R@1↑ R@5↑ R@10↑ MedR↓ CiCo [10] We compare our framework SEDS on various datasets, including How2Sign, PHOENIX-2014T, and CSL-Daily, with previous works i.e. SPOT-ALIGN [15] and CiCo [10] , where SPOT-ALIGN builds the final combination (COMB) model by integrating its primary cross-modal (CM) model and sign recognition (SR) model.For the How2Sign dataset, Table 1 shows the performance comparison between SEDS and existing methods. Benefiting from the large-scale image-text pre-training model CLIP [50] , CiCo and our method SEDS achieve significant performance improvements compared to SPOT-ALIGN. These consistent performance improvements demonstrate that utilizing prior visual knowledge of CLIP and contrastive learning paradigms greatly enhances the ability of model in visual-language alignment. Compared to current strongest competitor CiCo, SEDS achieves 62.5(+6.3) R@1 on sign text-tovideo retrieval task, and 57.9(+6.3) R@1 on sign video-to-text retrieval task. The outstanding performance demonstrates the effectiveness of our SEDS, which pays attention to both global visual information and local action information, and enables the most representative features of two modalities to fully interact and fuse.In Tables 2 and 3 , we evaluate the performance of SEDS on the PHOENIX-2014T and CSL-Daily datasets. Our model achieves a significant improvement of 76.8(+7.3), 85.8(+10.5) T2V R@1, and 78.7(+8.5), 85.4(+10.7) V2T R@1 on the PHOENIX-2014T and CSL-Daily datasets, respectively. These results indicate that SEDS has good generality and robustness in various sign language languages from different countries and regions.Performance of Sign Encoders with Different Modalities. In Table 4 , we report the performance of Pose encoder and RGB encoder individually on How2Sign, PHOENIX-2014T, and CSL-Daily datasets. We observe that Pose encoder performs better on CSL-Daily dataset, while RGB encoder has higher performance on PHOENIX-2014T dataset. As a professionally recorded dataset of daily life sign language, CSL-Daily has clearer and more standardized human body details and movements, and the differences between different sign gestures are significant, so it is more necessary to pay attention to local hand movements. PHOENIX-2014T R@1↑ R@5↑ R@10↑ R@1↑ R@5↑ R@10↑ ✓ How2Sign [16] 55.9 69.6 74.9 50.9 64.7 69. Structure T2V V2T R@1↑ R@5↑ R@10↑ R@1↑ R@5↑ R@10↑ Add-MLP 60.4 73.5 78.5 55. 5 as a sign dataset collected from German weather forecasts TV program, focuses more on the specific field of weather. So its sign gestures are more similar and need to be combined with global visual signals such as facial expressions for further semantic understanding. The content of these two datasets exactly corresponds to the conditions where Pose and RGB modalities are applicable. For How2Sign sign language dataset, the performance of Pose encoder and RGB encoder is basically similar. As the largest sign language dataset with a wide variety of topics, How2Sign needs to emphasize both local action details and global visual signals. This is also in line with the design intention of our framework SEDS, which has demonstrated outstanding performance on all three datasets. Different Feature Fusion Methods. To validate the effectiveness of our introduced Cross Gloss Attention Fusion (CGAF) Module in the fusion of Pose and RGB features, we compare it with several other fusion methods in Table 5 . Add-MLP adds the sequence features of two modalities at corresponding positions and passes them to a Multi-Layer Perception (MLP), resulting in the second-best performance. Concate-MLP method replaces the add operation with concatenate, and its performance is slightly inferior compared to the Add-MLP. Concate-Trans concatenates the two sequence features and feeds them into a shallow-layer Transformer for interaction, then separates them for addition at corresponding positions. This method exhibits a notable decrease in performance when compared to the first two methods, as the feature aggregation at specific positions incorporates irrelevant distant features from both modalities. Similarly, Cross-Atten also aggregates irrelevant distant features from the opposite modality, leading to a certain decline in performance. The CGAF Module effectively integrates only local features with similar semantics surrounding corresponding positions from inter-modal and intra-modal while ignoring distant features. As a result, it achieves the best performance among various fusion methods.Analysis of Pose-RGB Fine-grained Matching Objective. In our framework, the role of the Pose-RGB fine-grained matching Pose-RGB Alignment T2V V2T Yes No R@1↑ R@5↑ R@10↑ R@1↑ R@5↑ R@10↑ objective is to explicitly align the corresponding Pose and RGB clip features before fusion, then implicitly reduce the difference between distribution of Pose-Text and RGB-Text fine-grained matching matrices. We demonstrate the effect of this objective on our CGAF module in Table 6 . CGAF module benefits significantly from Pose-RGB fine-grained alignment. This objective not only enhances the fusion of corresponding position features, but also leads to better aggregation of semantically similar neighbors from two modalities in the latent space. Consequently, the performance improvement of the CGAF method is +1.7 T2V and +2.1 V2T R@1.The Effect of Factors 𝛼 and Factors 𝛽. As depicted in Fig. 5 , we investigate the effect of the auxiliary loss factor 𝛼 and the Pose-RGB matching factor 𝛽 on T2V and V2T tasks. For the auxiliary loss factor 𝛼, there is a significant improvement in performance when it increases from 0 to 0.1. The performance continues to rise steadily with the increase of 𝛼, reaching a peak at 0.8, after which it gradually decreases. This indicates that the existence of auxiliary loss effectively provides supervision for both branches. However, an overly large auxiliary loss may overly emphasize the training of two branches. For the Pose-RGB matching factor 𝛽, performance consistently improves until it reaches 0.4, after which it begins to decline gradually. A smaller value enables fine-grained alignment between Pose and RGB clip features, while an excessively large 𝛽 will overemphasize this objective, compromising the representation ability of Pose and RGB modalities.Visualization of the Fine-grained Similarity Matrices. To provide a more intuitive demonstration of the effect of Pose-RGB finegrained matching objective, we select a pair of sign language video and annotation. Then we calculate and display the fine-grained similarity matrices between the Pose, RGB, and Fusion features with text in the form of heat maps, both with and without this objective. As shown in Fig. 6 , the heat maps in the upper part of the figure represent the matrices when the Pose-RGB matching objective exists, while those in the lower part represent the situation when the objective is absent. The distribution differences between the Pose and RGB similarity matrices are smaller in the upper part than in the lower part. Additionally, the mean values of the distributions are essentially the same for the upper matrices, while they differ significantly for the lower matrices. The Fusion similarity matrix in the upper part contains more high similarity scores compared to the lower part, and the overall distribution has a higher mean value. This observation suggests that the presence of the Pose-RGB matching objective allows for the fusion of features to consider the characteristics of both the Pose and RGB features, enabling them to better match the corresponding words in sign language annotation. In contrast, the focus during fusion tends to favor a single modality with higher mean scores of fine-grained matrix when the objective is absent, resulting in poor fusion performance.Retrieval Results of Different Sign Encoders. The Top-3 TVR results are displayed in Fig. 7 . Specifically, in the first example, we observe that the Pose encoder focuses more on the similarity of detailed hand movements and key joint positions, while the RGB encoder emphasizes the overall action and expression of the person. In the second example, which includes two different signers, the RGB encoder shows a clear preference for matching specific individuals, even though there are significant differences in the details of local part movements. The Pose encoder, due to its input modality, eliminates this potential bias. By combining the strengths of both the Pose modality and the RGB modality, our framework SEDS retrieves the correct results in both examples. It is capable of focusing on both local details and the global semantics of the videos, while also exhibiting great robustness and generalization. 
conclusion: 
 In this paper, we propose a novel framework called Semantically Enhanced Dual-Stream Encoder (SEDS) for the emerging Sign Language Retrieval task. SEDS uses online Pose encoder and offline RGB encoder to extract features from sign language videos, focusing on local action details and global visual semantics simultaneously. In the fusion stage, we propose a Cross Gloss Attention Fusion (CGAF) Module to fuse local information with similar semantic information from two modalities. We also develop a supervised Pose-RGB fine-grained matching objective, to match the contextual fine-grained dual-stream features. Our SEDS outperforms previous work significantly on How2Sign, PHOENIX-2014T, and CSL-Daily datasets. We hope that our work helps future exploration of sign language video retrieval tasks and promotes the development of multimodal alignment in retrieval tasks.  
 
 Appendix:  


Decision: The authors provide some new arguments to address the critical issues highlighted by reviewers. Most of the reviewers are satisfied with the response. I recommend that the authors incorporate the reviewers' insightful feedback into their revision.

Reviews: review: paper_summary: paper_summary: This paper introduces a new sign language retrieval framework called Semantically Enhanced Dual Stream Encoder (SEDS). The SEDS framework utilizes online pose encoders and offline RGB encoders to extract features from sign language videos, while paying attention to local action details and global visual semantics. In the fusion stage, a Cross Gloss Attention Fusion (CGAF) module is proposed to fuse local information with similar semantic information between two modalities. In addition, a supervised Pose RGB fine-grained matching target has been developed for matching fine-grained dual stream features in the context. The experimental results of SEDS on multiple datasets such as How2Sign, PHOENIX-2014T, and CSL-Daily, are significantly better than previous works. The author hopes that this work can promote the exploration of future sign language video retrieval tasks and the development of multimodal alignment in retrieval tasks.
strength_weakness: strength_weakness: 1. The structure of this paper is clear and easy to follow. 
2. In response to the existing problems in sign language retrieval work, the SEDS framework effectively solves the problem of local action details being overwhelmed by a large amount of visual information redundancy by introducing posture modal knowledge.
3. Through the Cross Gloss Attention Fusion (CGAF) module, effective aggregation of similar semantic information between two video modalities has been achieved, which helps to enhance the semantic representation of visual words in sign language video editing.
4. Extensive experiments on the How2Sign, PHOENIX-2014T, and CSL-Daily datasets have shown that the SEDS framework is significantly superior to existing state-of-the-art methods
limitations: limitations: 1. Although the framework includes lightweight networks, pose estimation and dual stream encoders may require higher computational resources, and the article lacks explanation of the possible increase in training or inference time.
2. There is no explanation in the paper whether it will release source code.
paper_summary: paper_summary: The paper presents a novel framework for sign language retrieval called the Semantically Enhanced Dual-Stream Encoder (SEDS). It integrates Pose and RGB modalities to capture both local and global information from sign language videos. The proposed framework includes a Cross Gloss Attention Fusion (CGAF) module to aggregate semantically similar features from both modalities and a Pose-RGB Fine-grained Matching Objective to enhance the representation of the fused features.
strength_weakness: strength_weakness: 1. Innovative Multi-Modal Approach: The paper introduces the Semantically Enhanced Dual-Stream Encoder (SEDS) framework, which uniquely integrates Pose and RGB modalities to capture both local action details and global semantic information, significantly enhancing the feature representation for sign language videos.

2. Effective Feature Fusion: The Cross Gloss Attention Fusion (CGAF) module and Pose-RGB Fine-grained Matching Objective are novel contributions that improve the fusion of semantically similar features and the contextual alignment of multi-modal features, leading to superior performance in sign language video retrieval tasks.

3. Comprehensive Experimental Validation: Extensive experiments on multiple datasets (How2Sign, PHOENIX-2014T, CSL-Daily) demonstrate that the proposed method significantly outperforms state-of-the-art approaches, validating its effectiveness and generalizability across different sign languages and scenarios.
limitations: limitations: 1. Lack of Motivation for Multi-Modal Data: While the introduction of Pose data aims to enhance feature representation, the paper does not sufficiently justify the necessity of multi-modal data over single-modality approaches. A more detailed explanation of why existing RGB-only methods are inadequate and how Pose data specifically addresses these shortcomings would strengthen the motivation.

2. Fairness in Experimental Comparisons: The introduction of Pose data introduces a new modality, raising concerns about the fairness of comparisons with existing methods that rely solely on RGB data. The paper should ensure that baseline methods are adapted to incorporate similar types of information or provide a detailed discussion on the limitations of single-modality approaches in capturing the required features.

3. Resource Consumption Analysis: The paper does not discuss the potential increase in computational resource consumption due to the additional data stream. An analysis of GPU memory usage, processing time, and computational complexity is necessary to provide a comprehensive evaluation of the method’s efficiency.
paper_summary: paper_summary: This paper proposes a framework to effectively reduce the huge memory cost of dense visual data embedding in end-to-end training, proposing a Cross Gloss Attention Fusion (CGAF) module and effectively enhancing the representation ability of fusion clip features.
strength_weakness: strength_weakness: introduce the skeleton modality
limitations: limitations: 1. The method section involves an excessive number of variables.
2. Visualize the two jointly optimized losses mentioned in the Fig.2.
3. The experimental section currently employs only two comparison methods. Could you provide a rationale for this choice? Furthermore, incorporating additional control experiments would strengthen the validity and reliability of the findings.
4.I wonder the motivation of introducing the skeleton modality, how to choose the skeleton detectors? What are the different influences with RTMPose openpose and mmpose? 
5.The organization of the methods section is confusing to me. The specific description of the proposed method is limited, whereas a significant amount of space is devoted to existing technologies such as infoNCE.
paper_summary: paper_summary: This paper proposes a sign language retrieval framework by utilizing extra human pose signal. In detail, authors design a cross gloss attention fusion (CGAF) module to aggreggate video clip features and human pose features. Besides, they use a Pose-RGB fine-grained matching objective to align two visiual modalities. Extensive experiments are conducted to show the effectiveness and efficiency of the proposed framework.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The idea to introduce human pose signal is very reasonable for sign language retrieval.
3. The performance improvement of the method presented in this paper is significant.
3. The authors have conducted a lot of experiments to prove the validity and rationality of design modules in detail.
limitations: limitations: 1. Compared with other methods, the proposed framework uses extra human pose signal. It is better to provide experimental results of other methods equipped with human pose signal. In other words, I wonder if the introduction of human pose signal could be used as a general method to improve the performance of sign language retrieval models.