MGR-Dark: A Large Multimodal Video Dataset  and RGB-IR benchmark for Gesture Recognition in Darkness

Authors: chalearn  isogd
ld-congr  ipn
jester  mgr-dark
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:42:54

Abstract: Gesture recognition plays a crucial role in natural human-computer interaction and sign language recognition. Despite considerable progress in normal daylight, research dedicated to gesture recognition in dark environments is scarce. This is partly due to the lack of sufficient datasets for such a task. We bridge the gap of the lack of data for this task by collecting a new dataset: a large-scale multimodal video dataset for gesture recognition in darkness (MGR-Dark). MGR-Dark is distinguished from existing gesture datasets by its gesture collection in darkness, multimodal videos(RGB, Depth, and Infrared), and high video quality. To the best of our knowledge, this is the first multimodal dataset dedicated to human gesture action in dark videos of high quality. Building upon this, we propose a Modality Translation and Cross-modal Distillation (MTCD) RGB-IR benchmark framework. Specifically, the modality translator is firstly utilized to transfer RGB data to pseudo-Infrared data, a progressive cross-modal feature distillation module is then designed to exploit the underlying relations between RGB, pseudo-Infrared and Infrared modalities to guide RGB feature learning. The experiments demonstrate that the dataset and benchmark proposed in this paper are expected to advance research in gesture recognition in dark videos. The dataset and code will be available upon acceptance.

Main paper: 
 introduction: 
 Gestures play a crucial role in human-computer interaction and language expression, serving as a significant means of information transmission. Numerous datasets [3, 15, 16, 19, 23, 34] and research [4, 17, 24, 33, [36] [37] [38] have been dedicated to gesture recognition. However, existing datasets primarily focus on ideal, well-lit environments, neglecting low-light conditions in real-world scenarios. This is partly due to the fact that current benchmark datasets for gesture recognition [3, 16, 23] are normally collected from web videos, which are shot mostly under normal illumination. Yet videos under normal illumination conditions are not available in many cases, such as night surveillance [14, 16] , and self-driving at night [4, 19] .In such scenarios, additional sensors like Infrared or Depth imaging sensors can be utilized to recognize gestures in darkness. Furthermore, existing datasets often lack high-quality video data due to limitations in early data collection sensors. With the advancement of sensors, the need for high-quality datasets has also become essential for practical applications.While multimodal gesture recognition progresses have certainly been made, there are yet many obstacles to be overcome, information transfer between RGB and Infrared poses particular challenges especially. Firstly, the use of different sensors for capturing RGB and Infrared data, along with their differing wide-angle perspectives, leads to horizontal misalignment of images. Secondly, directly using the infrared data stream as the teacher model to guide the student model's RGB data stream may result in the misalignment of feature hierarchies due to the significant modal differences. Many researchers [9, 22, 33] focus on RGB-IR cross-modal distillation. However, it is hard to let the student model simply mimic a pretrained teacher model when the capacity gap between the teacher and student is large [12, 18, 20] . For instance, Mirzadeh et al. [12] proposed the Teacher-Assistant KD method by gradually increasing the teacher size to foster the distillation process. Inspired by them, we consider that a new intermediate network can bridge the gap between the teacher and student.In this paper, a large-scale multimodal video dataset (MGR-Dark) is established for gesture recognition in darkness. To the best of our knowledge, it is the first dataset focused on human gesture recognition in the dark. MGR-Dark is distinguished from existing gesture datasets by three highlights. a). MGR-Dark draws attention to low-light gesture interaction. Unlike the existing datasets that record gestures in normal light, we capture gestures in a dark environment. Fig. 2 shows example frames sampled from different gesture datasets. It can be seen that the gestures in MGR-Dark are captured with a large field of dark light and are difficult to recognize, which poses a new challenge for gesture recognition. b). MGR-Dark provides three modalities stream for gesture recognition. In MGR-Dark, each gesture contains three paired videos,including synchronized RGB, Depth and Infrared stream. It should be noted that we also collected the paired data in daylight(See Figure 3 ), which provides researchers with more comprehensive data and brings more possibilities for solution gesture recognition in the real world. c). The videos collected in MGR-Dark are high quality. The Kinect V2 and XInfrared T3S, equipped with advanced sensors, are used to collect high-quality RGB, Depth, and Infrared video data. The RGB, Depth, and Infrared streams are captured synchronously with resolutions of 1920 × 1080, 512 × 424, and 1408 × 1068 respectively.To tackle the challenges of cross-modal learning between RGB and infrared data, we propose a novel Modality Translation, Crossmodal Distillation (MTCD) framework which integrates Modality Translation(MT), Progressive Cross-modal Distillation (PCD) module. The two modules address the RGB-IR problem at the image level, and the cross-modality feature level, respectively. First, we employ a modality translator to generate pseudo-Infrared data from the low-light RGB data. Subsequently, in PCD, the teacher network takes the Infrared data as input, the assistant network receives the pseudo-Infrared data as input, while the student model takes the low-light RGB data as input, all of network with features adapted by the attention-based feature converter. During training, the student network(RGB feature learning) is guided separately by the teacher network, the assistant network, and their combination. During inferring, just RGB data in darkness is employed for gesture recognition.In summary, we explore the task of gesture recognition in dark videos. The contribution of this work is three-fold and is summarized as follows:• We develop a new Multimodal Gesture Recognition in the Dark (MGR-Dark) dataset, dedicated to the task of recognizing gestures in real-world darkness environment, which, to the best of our knowledge, is the first dataset focused on human gesture recognition in dark videos.• We propose a Modality Translation and Cross-modal Distillation (MTCD) RGB-IR benchmark framework, exploring pseudo-Infrared stream and progressive cross-modal distillation methods to improve RGB gesture recognition accuracy.• We benchmark the performance of state-of-the-art gesture recognition models on our dataset and reveal challenges in the task of gesture recognition in dark videos. 
related works 2.1 gesture recognition datasets: 
 In the field of gesture recognition, there are several benchmark datasets [3, 15, 16, 19, 23, 34] . For example, the 20BN Jester [16] dataset and the IPN Hand [3] dataset instruct subjects to record gestures using their personal computers or laptops. Subjects sit in front of the computer camera and simulate computer operations using gestures. NvGesture [19] aims to enable gesture-based manipulation of cars, with gestures recorded within a car simulator. EgoGesture [34] concentrates on gesture interaction with wearable devices, collected using Intel RealSense SR300 RGB-D cameras mounted on the subject's head. In the case of ChaLearn IsoGD [23] , subjects perform gestures while standing within 1 meter of the Kinect V1 camera under sufficient lighting conditions. The LD-ConGR dataset [15] focuses on long-distance gesture interaction in normal daylight. However, although these datasets encompass a wide range of gestures, it's evident that the gestures are predominantly recorded under normal lighting conditions (See Fig. 1 ), which limits the capability of current gesture recognition models to videos with normal illumination. In many real-world scenarios, it is necessary to interact with the machine in a low-light environment. Therefore, to investigate gesture recognition performance in dark videos, we have collected a novel multimodal video dataset captured in dark environments.Knowledge distillation [11] is a powerful tool for transferring useful information between different domains, such as high-resolution and low-resolution images, RGB and Depth data, etc. There are many cross-modal distillation researches in action recognition. Shi and Kim [21] leverage skeleton data as privileged information to enhance the learning of RNN-based models for action recognition from depth sequences. Bargal et al. [8] trained a hallucination stream, by designing a multi-stage training paradigm and combining an Euclidean-distance metric loss with a generalized distillation loss, for action recognition. However, previous works either ignored the fact that the student modality can contribute useful information to the training of the cross-modal knowledge distillation process or it is not efficient to let the small student network learn directly from a pre-trained teacher network. Our work, instead, utilized both teacher and student modality data to build up assistant (middle)models to discover the relationship between RGB and thermal images. In this way, teacher network will tend to produce a model which is easier for the student network to understand from the human's learning analogy perspective. 
mgr-dark dataset: 
 Although a small number of videos taken in the dark do exist in current action recognition benchmark datasets, such as Kinetics [5] and ARID [26] , the task of gesture recognition in darkness environments has rarely been studied. This is partly due to the very low proportion of dark videos in current benchmark datasets, and a lack of datasets dedicated to gesture analysis in the dark. To bridge the gap in the lack of dark video data, we introduce a new Multimodal Gesture Recognition in the Dark (MGR-Dark) dataset. In the following subsections, we will introduce the classes, collection and annotation of the dataset, report data statistics, and make a comparative analysis of the MGR-Dark dataset and other gesture recognition datasets.Collection. The video clips in the MGR-Dark dataset are collected using Kinect V2 and Xinfrared T3S. We use Kinect V4 to collect RGB-D video data. Kinect V2 is equipped with a 12-megapixel RGB camera and a 1-megapixel Depth camera, ensuring the quality of the captured videos. Xinfrared T3S is a commercial camera equipped with a 12-megapixel Infrared camera. We synchronously record RGB, Depth, and Infrared streams with resolutions of 1920×1080,512× 424, and 1408 × 1068 respectively. The frame rate of 25fps. The MGR-Dark dataset includes a total of 10 gesture classes.The gesture classes include forward, backward, stop, keep down, turn left, turn right, upward, look for, and quiet. We collected videos in 9 outdoor scenes, including carparks, corridors, and groves, and over 17 indoor scenes, such as halls, stairs, and laboratories. The lighting condition of each scene is different, with no direct light shot on the actor in almost all videos. In many cases, it is challenging even for the naked eye to recognize the human gesture without tuning the raw video clips. Figure 2 illustrates sample frames for each of the 10 gesture classes captured in various scenes and lighting conditions in the MGR-Dark dataset. The videos are shot strictly during night hours. In order to expand the application field of the dataset, we further captured paired gesture videos during daylight. Figure 3 shows the samples for each modality in darkness and the paired samples in daylight. The distance from the recording spot to the camera is between 2m and 4m. The videos feature a total of 21 subjects, including 17 males and 4 females. All subjects are shown the standard gestures before recording, and the recorded video will be further checked whether the gestures are correct. The subjects are asked to perform gestures continuously, and a short break is allowed between two gesture instances. The data is only allowed for academic research and we will provide strict access for applicants who sign data use agreements. The subjects were informed of the uses of the data and signed informed consent. The gesture label with its video frames can be found in Supplementary.Annotation. In the original dataset, each video sequence contains gestures from 10 different classes, performed sequentially. We manually segmented the data and labeled the temporal segmentation information, indicating the beginning and end frames of each gesture in all MGR-Dark dataset videos. The gestures were synchronized based on time points. Since the RGB and Depth video streams are synchronized, only the RGB streams required labeling, with Depth video annotations obtained accordingly. We utilized automated codes for gesture segmentation based on temporal information, significantly reducing time and labor. Finally, isolated gesture datasets were created.The MGR-Dark dataset comprises a total of 31,020 video clips, with 15,510 clips for dark videos and an equal number for daylight videos, due to their paired collection. Each modality, whether dark or daylight, contains 5,170 video clips. With 10 gesture classes, each modality has 517 video clips per class. The dataset was divided randomly into a training set and a testing set based on subjects, with gesture clips from 16 subjects included in the training set and those from the remaining 5 subjects forming the testing set. All video clips maintain a fixed frame rate of 25fps. Clip lengths range from a minimum of 1.8 seconds (45 frames) to a maximum of 8.4 seconds (210 frames), with the entire dataset spanning 8,721 seconds. Video clips are stored in .mp4 format.In Tab. 1, we compare our dataset MGR-Dark with the publicly available gesture recognition datasets, including Jester [16] ], NvGesture [19] , EgoGesture [34] , ChaLearn IsoGD [23] , LD-ConGR [15] and IPN Hand [3] . Below we will make a detailed comparison and explain the advantages of our dataset from three aspects: light conditions, diversity, and video quality.Light conditions. To better understand the illumination of real dark videos, we compute and compare the statistics of the MGR-Dark(dark) dataset with other datasets. Figure 4 displays the RGB values histograms of datasets MGR-Dark(dark), MGR-Dark(daylight), Jester [16] , NvGesture [19] , EgoGesture [34] , ChaLearn IsoGD [23] , LD-ConGR [15] and IPN Hand [3] .The histograms of the first two columns represent MGR-Dark, with the first column showing gesture videos in darkness and the second column showing Diversity. Our dataset covers multimodality and various scenes, and dark gesture videos with paired daylight videos. The videos are collected with three modalities. As we can see in Tab. 2, our dataset provides RGB, Depth and Infrared video data compared to others except NvGesture [19] . Infrared stream is useful for various real-life scenarios, e.g., night surveillance, emergency call for help at night, and driving at night. Compared with NvGesture [19] focus on drive scenes, the videos are collected in more than 25 different scenes, including 9 outdoor scenes and more than 17 indoor scenes, such as carparks, corridors and groves for outdoor scenes, and halls, stairs and laboratories for indoor scenes. To our knowledge, our dataset is the first to provide paired daytime and dark action videos, further enhancing its uniqueness and utility.Quality. As we can see in Tab. 1, our dataset provides highdefinition RGB video data (1920 × 1080), while the highest resolution of other gesture datasets is only 640 × 480. In addition, the Depth streams (captured synchronously with RGB streams) are available in our dataset and have a higher resolution (512 × 424) compared to NvGesture [19] (320 × 240) and ChaLearn IsoGD [23] (320×240). Meanwhile, the resolution of the Infrared data stream we provide is 1408 × 1068. 
method: 
 The overview architecture of the MTCD framework is illustrated in Fig. 5 . To tackle the challenges of cross-modal learning between RGB and Infrared data, we propose a novel Modality Translation, Cross-modal Distillation (MTCD) framework, comprising the RGBto-IR Modality Translation (MT) and Progressive Cross-modal Distillation (PCD) modules. First, we employ a modality translator to generate pseudo-Infrared data from the low-light RGB modality data. Subsequently, we leverage the teacher network and assistant network to integrate the cross-modal knowledge and then guide the feature learning of the student network. The teacher network takes the Infrared data as input, the assistant network inputs the pseudo-Infrared data, and the student network takes the low-light RGB data as input. Three networks with features optimize by the attention-based feature converter. During training, guidance for the student model's RGB feature learning is provided separately by the teacher network, the assistant network, and their combination. During inferring, just the student network with RGB data in darkness is employed for gesture recognition.Motivated by recent advancements in Re-Identification [27, 31] , we introduce the Modality Translator (MT), based on the domain translation-based method [9] (SDA GAN), to generate pseudo data. Given a source RGB dataset, the MT translates RGB images into Infrared images while preserving the identity of the subjects. This results in a set of original RGB images and their corresponding translated Infrared images. These translated Infrared images serve as pseudo-Infrared data, aiding in guiding cross-model learning. In the experiment, we pretrained the proposed Modality Translation framework on widely used image RGB-IR datasets,SYSUMM01 [25] .Two feature encoders are first pre-trained with two different initialization seeds on SYSUMM01 for providing weight initialization. The encoder-decoder model is a feature transformation function that extracts feature representation 𝐹 (𝑥 𝑖 | 𝜃 ) from input image 𝑥 𝑅𝐺𝐵 𝑖 . The 𝜃 denotes the parameters of the feature encoder network. The training loss can be defined byEQUATIONwhere 𝐿 𝑅𝐺𝐵 (𝜃 ) is the cross-entropy loss, and 𝐿 Tri (𝜃 ) [29] is the triplet loss with hard samples. 𝜆 pre is the parameter weighting the two losses. 𝐿 𝑅𝐺𝐵 (𝜃 ) and 𝐿 Tri (𝜃 [29] are formulated as:EQUATIONEQUATIONwhere 𝑞(𝑖) is the predicted label probability and 𝑝 (𝑖) is the groundtruth label probability, 𝐾 is the total number of training RGB data in the dataset.⌊𝑥⌋ + = max(𝑥, 0), 𝜖 is the margin,𝐷 𝑖,𝑝 and 𝐷 𝑖,𝑛 denote the L2-norm distance between sample 𝑥 𝑖 and its hardest positive 𝑥 𝑖 ,𝑝 and negative samples 𝑥 𝑖 , 𝑛 in one batch, respectively, which can be calculated byEQUATIONwhere ∥ • ∥ is the L2-norm distance between samples. Unlike traditional translation-based methods [27] focus on homogeneous domains and lack cross-modality supervision, our approach leverages MT to construct cross-modality image pairs, thereby providing essential cross-modality supervision. This helps mitigate the multimodal discrepancy, specifically addressing the misalignment at the image level. Furthermore, the generated pseudo-Infrared data enhances cross-modality learning by enriching the feature space with additional cross-modality information.Inspired by [20] , we introduced a progressive KD learning procedure that requires the student network to learn from the teacher network step-by-step or in incremental method, which can help the student network better learn the information from the teacher network. Specifically, the teacher network takes the Infrared data(IR) as input and fuses, the assistant network receives the pseudo-Infrared(IR 𝑝 ) data as input, while the student network takes the low-light RGB data as input. We leverage the teacher network and assistant network to integrate the cross-modal knowledge and then guide the feature learning of the student network.As shown in Fig. 5 ,our Progressive Cross-modal Feature Distillation is divided into three parts: IR→RGB cross-modal distillation,IR→IR 𝑝 cross-modal distillation, and Fusion→RGB cross-modal distillation. In other words, the loss functions of cross-modal feature distillation are formulated as:EQUATIONEQUATIONEQUATION)where the feature of the teacher network is denoted as 𝐹 𝑇 ,𝐼𝑅 , the feature of the assistant network is denoted as 𝐹 𝐴,𝐼𝑅 𝑝 , and the feature of the student network is 𝐹 𝑆,𝑅𝐺𝐵 . To match the feature dimension, 𝑇 𝑡 , 𝑇 𝑎 and 𝑇 𝑠 respectively, we transform the feature 𝐹 𝑇 ,𝐼𝑅 , 𝐹 𝐴,𝐼𝑅 𝑝 and 𝐹 𝑆,𝑅𝐺𝐵 . A distance d between the transformed features is used as a loss function 𝐿 * → * 𝑘𝑑 . By incorporating the assistant network, the gap between the Infrared and RGB streams are narrowed, facilitating the student model in better mimicking the performance of the teacher network. Specifically, the student model will be updated one step towards the ground-truth labels, guided by the fusion of teacher and assistant features.The architecture of our attentive feature converter is depicted on the right side of Fig. 5 . Our proposed feature converter takes the initial feature 𝐹 from the feature extractor as input. It utilizes consecutive spatial attention and temporal attention modules to cooperatively adjust features in both spatial and channel dimensions. This process can be formally described as follows:EQUATIONwhere 𝐹 : [T, C, H, W] signifies the converted feature, while 𝐴𝑡𝑡 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 and 𝐴𝑡𝑡 𝑡𝑒𝑚𝑝𝑜𝑟𝑎𝑙 represent spatial attention and temporal attention map, respectively. We utilize a 3 × 3 convolution layer with sigmoid activation to generate a spatial attention map 𝐴𝑡𝑡 𝑡𝑒𝑚𝑝𝑜𝑟𝑎𝑙 and 𝐴𝑡𝑡 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 , except involving a global average pooling:EQUATIONAs depicted in Fig. 5 , the attention feature converter assumes distinct roles within the Teacher, Assistant and Student networks. In the Teacher and assistant network, the feature converter facilitates the adaptively fused of homogeneous domains in spatial and channel domains, thereby enhancing feature representation learning. In the Student Network, the attentive feature converter mainly focuses on alleviating the feature gap between the RGB feature and the pseudo-Infrared feature with our contrastive feature distillation.In summary, we train the student network and the final total loss for the student network is defined as follows: EQUATION 
experiments: 
 In this section, we will first evaluate the state-of-the-art methods in the field of gesture recognition on the MGR-Dark dataset and then discuss the proposed RGB-IR benchmark method on two datasets.All experiments are implemented on the PyTorch with two NVIDIA RTX 3090 GPU. To prepare the input data for training and inference, it is resized to 256 × 256 first and then randomly/center cropped into 224 × 224 with a length of 32 frames, respectively. During training, the batch size is set to 32, and the training lasts for 80 epochs on all three datasets. The initial learning rate is set to 0.0001, and we decay the learning rate by a factor of 0.1 after 20 epochs. We train all networks using the SGD optimizer. We select 32 frames from each action clip to generate input data, uniformly sampled in time. Additionally, the trade-off parameter 𝛼 and 𝛽 are 0.9 and 0.3.To evaluate the performances of the models, we use the recognition rate, L 𝑐𝑒 , as defined in :EQUATIONwhere 𝑛 is the total number of samples, 𝑝 is the predicted label, 𝑦 is the groundtruth label; if 𝑝 (𝑖) = 𝑦 (𝑖),𝑓 (𝑝 (𝑖); 𝑦 (𝑖)) = 1, otherwise 𝑓 (𝑝 (𝑖); 𝑦 (𝑖)) = 0. We refer to this metric as top-1 recognition rate, since we are only evaluating a model's best guess. Top-N recognition rate refers to the rate by which the true class label exists in a model's top-N predictions.We evaluate the state-of-the-art gesture and action recognition methods on the proposed MGR-Dark dataset, and the results are presented in Tab. 2.In order to mitigate the influence of different multimodal data processing methods, we conduct separate comparisons based on the RGB or Depth modality.As shown in Tab. 2, we assess MGR-Dark datset using four different backbones: ResNet [24] , 3D CNN [6, 17] , 3D ResNet [4] , and Transformer [36, 37] . 3DDSN [6] and C3D [17] methods are based on 3D CNN, which simultaneously extract spatial and temporal features via 3D convolutions. In contrast, TSN [24] model processes spatial and temporal information separately by segmenting the video and sampling one frame from each segment. Spatial features are then extracted using 2D CNNs from the sampled frames, while temporal features are represented by optical flow. I3D [4] and our method are based on 3D ResNet. Compared to 3D CNN networks, which can be challenging to train and optimize, the incorporation of residual connections in 3D ResNet reduces the difficulty of training deep networks and enables better capture of spatiotemporal information in video data. The results indicate that the performance The experiments are mainly implemented on the part of darkness in the MGR-Dark dataset.of TSN [24] is inferior to that of 3D ResNet-based methods on the MGR-Dark dataset. This is primarily because gestures with long duration may lose keyframes during segmentation and random sampling. Transformer [36, 37] does not perform as well as 3D ResNet due to the vast amount of data required for training transformer networks. Conversely, 3D ResNet demonstrates a strong ability to extract spatiotemporal features. Hence, we focus on 3D ResNetbased models for the backbone of our benchmark. We also notice that though our dataset is of relatively small size and has fewer classes than current normal illumination video datasets, there is plenty of room for improvement in accuracy. Additionally, we further employ NvGesture to test proposed RGB-IR benchmarks. NvGesture Dataset [19] is the only multimodal gesture recognition dataset that provides RGB, Depth, and infrared modalities. As presented in Tab. 3, our method shows its clear advantage against other multimodal methods overall. Although our accuracy is not the highest on this dataset, it still ranks in the top three, which demonstrates that the proposed method can improve the representation of RGB stream by progressive crossmodal distillation, making a better performance. It's worth noting that our method only requires single-modal RGB input during inference.The Effectiveness of Components. For a comprehensive understanding of our model, we conduct ablation experiments to demonstrate the improvements obtained by three important components, including Teacher network, Assistant network, Fusion of Teacher and Assistant network. We ensemble each module to the backbone network step-by-step, and compare the recognition performance on the MGR-Dark. For the convenience of comparison, we construct the baseline that uses the student network with only RGB as input. The results are illustrated in Tab. 4.A) Teacher network (L 𝐼𝑅→𝑅𝐺𝐵). We verify the effectiveness of the Teacher network module first. The results in the first row are obtained from the same depth baseline network, while those in the second row stem from the student network (RGB stream) being distilled by the teacher network (Infrared stream). A comparison of the results in the first and second rows of Tab. 4 reveals that RGB feature learning with the guided by teacher network shows an improvement of nearly 4%. This underscores the crucial role of crossmodal representation in enhancing gesture feature performance. B) Assistant network (L 𝐼𝑅→𝑅𝐺𝐵). With the addition of the Assistant Network (pseudo-Infrared stream), the performance of MTCD further improves. However, although there has been improvement, ). After the fusion of the teacher and assistant network, the results demonstrate that cross-modal representation learning can enhance the progressive cross-modal learning procedure, bringing a prominent performance gain (4.13%). The student network learns from the teacher network in a step-by-step or incremental fashion, thereby helping the student network better mimic the performance of the teacher network. Overall, MTCD significantly outperforms baseline models, underscoring the validity of the three components. The Effectiveness of hyperparameters. The parameter 𝛼 indicates the importance of L 𝐹𝑢𝑠𝑖𝑜𝑛→𝑅𝐺𝐵 𝑘𝑑 . We evaluate the scale range setting 𝛼 ∈ [0.0, 1.5] as shown in Figure 6 . We find that accuracy is improved from 85.26% when 𝛼 = 0.9. As a result, we adopt 𝛼 = 0.9 to achieve the best performance. The parameter indicates the importance of The parameter 𝛼 indicates the importance of L 𝐹𝑢𝑠𝑖𝑜𝑛→𝑅𝐺𝐵 𝑘𝑑 . In Figure 6 , we show the influence of the hyperparameter 𝛽. We also evaluate the scale range setting 𝛽 ∈ [0.0, 1.5]. We find that the model achieves the best performance at 𝛽 = 0.3, so we set 𝛽 = 0.3 as the default in practice. 
conclusion: 
 In this paper, we present a large multimodal video dataset MGR-Dark. MGR-Dark is distinguished from existing gesture datasets by its gesture collection in darkness, multimodal videos (RGB, Depth, and Infrared), and high video quality. To address gesture recognition in darkness, we propose a Modality Translation, Cross-modal Distillation framework. Moreover, representative methods of gesture and action recognition are evaluated and discussed on the MGR-Dark. We believe that our dataset and experimental studies can inspire research in many fields, including but not limited to gesture recognition, action recognition, and human-computer interaction.  
 
 Appendix:  


Decision: The paper proposes a new multimodal dataset that fills a rather unexplored area of study $-$ gesture recognition in the dark. This should benefit the current research community with all reviewers equally receptive to this proposal. The paper also goes to the extent of providing competitive baselines and introducing a new MTCD framework that aids the transfer (or distillation) of information and relations between the modalities. I project that this dataset would advance the current research in gesture recognition, especially since the possibility of adverse capture conditions can hinder well-performing methods. There are some presentational / writing issues in the paper, which were rightly pointed out by one reviewer. Also, some sections of the paper that lack clarity (as already answered in the rebuttal) should be properly addressed in the final version to increase the quality of the paper. There were some minor concerns regarding the proposed MTCD method but I think the paper has placed a tremendous emphasis on the initiation of the dataset, and hence it would be slightly unreasonable to overcrowd the paper with too many details. All things considered, I can recommend that this paper be accepted as a poster at ACM MM.

Reviews: review: paper_summary: paper_summary: The paper presents a significant contribution to the field of gesture recognition with the introduction of the MGR-Dark dataset and the MTCD framework. The research is well-conducted, and the results are promising. However, there is room for expansion and further exploration of the framework's limitations and potential applications.
strength_weakness: strength_weakness: 1. The introduction of a new dataset that fills a gap in the current research landscape.
2. The innovative approach to cross-modal learning between RGB and Infrared data.
3. Comprehensive experiments and ablation studies that validate the proposed framework.
limitations: limitations: 1. While the dataset is large, the number of gesture classes is limited, which might affect the generalizability of the results.
2. The paper could benefit from a broader discussion on the potential limitations of the MTCD framework and how it might be adapted or extended in future work.
3. Given the high resolution of the dataset, I am curious about the efficiency of the methods in processing this data.
paper_summary: paper_summary: This paper proposes a new large-scale multimodal dynamic gesture recognition database based on dark environments. Further, a distillation benchmark network is introduced using the database.
strength_weakness: strength_weakness: •	Gesture recognition is important for various applications. This paper presents a gesture recognition database collected in the dark environments. According to the author, this is among the first work to help the community in night surveillance and self-driving fields.
•	The database is high-quality and provides high-resolution images, which are very useful in improving accuracy. 
•	The experiments are well conducted.
limitations: limitations: •	This brightness, contrast and pixel intensity can be manipulated using the image processing techniques. Further, recent GAN-based methods can also generate the dark samples using the existing datasets.
•	If we use simulated images, then what is the performance using real (proposed dataset) and simulated datasets. It is needed to be verified. Further analysis among the RGB, Depth and Infrared images should be conducted to provide the characteristics difference between images captured in dark and simulated.
•	In Figure 5, should be re-drawn to clearly depict the information flow. The output of Modality Translation is not provided to other modules. Both arrows are shown as input to Modality Translation.
paper_summary: paper_summary: This paper introduces MGR-Dark as a multimodal gesture recognition dataset, where part of the data is collected under dark conditions.
strength_weakness: strength_weakness: The dataset is indeed interesting, expanding the gesture recognition data towards adverse environments, yet there are multiple concerns that the authors should address.
limitations: limitations: **1. Selection of class is unclear.** How are the 10 classes selected? MGR-Dark seems to have the smallest number of classes compared to other gesture recognition datasets. Why are these particular classes selected or collected? How are these classes characterized?

**2. Detailed statistics are missing.** Several key statistics are missing for the dataset and benchmark. For example, what is the train:test ratio of the dataset? Does the testing set include only dark videos or it also include daytime videos? How many videos are there for each video length (since the video length span from 45 frames to 210 frames, where the longest videos is almost 5 times the length of the shortest videos)? Also how many videos are there for each different depth? What are the specific scenes and why are these specific scenes selected?

**3. The dataset risks to be biased towards gestures performed by certain groups of people.** From the provided statistics and screenshot, we observe that the dataset is collected from a group of volunteers who are mainly (if not all) Asians and mostly male (male:female ratio over 4:1). If the dataset is biased towards gestures performed by such an ethnicity and gender, models trained on this dataset would not be generalizable towards the more general gesture recognition, and the dataset would prove rather useless.

Some further but rather minor concerns are as follows:

a. While the authors stressed on the multi-modality of the dataset, it should be noted that in real-world scenarios, it is common to use only the RGB modality due to the low cost of ordinary cameras. Is the trained model able to generalized to RGB-only scenarios?

b. The authors have not provided how the mosaic of the person filmed would be formed in the example provided, leading to doubt about whether the mosaic can protect the privacy of the person filmed.
paper_summary: paper_summary: This paper addresses the challenge of gesture recognition in dark environments, an underexplored area crucial for natural human-computer interaction and sign language recognition. The authors introduce MGR-Dark, a large-scale multimodal video dataset designed for gesture recognition in darkness, featuring high-quality RGB, Depth, and Infrared videos. They also propose a Modality Translation and Cross-modal Distillation (MTCD) RGB-IR benchmark framework. This framework includes a modality translator to convert RGB data to pseudo-Infrared data and a cross-modal feature distillation module to enhance RGB feature learning using Infrared data. The experiments show that this dataset and benchmark framework significantly advance research in dark environment gesture recognition.
strength_weakness: strength_weakness: The strengths of this paper are three-fold. Firstly, it introduces the MGR-Dark dataset, specifically designed for gesture recognition in dark environments, which fills a notable gap in existing research. Secondly, it develops a Modality Translation and Cross-modal Distillation (MTCD) RGB-IR benchmark framework that uses pseudo-Infrared data and progressive cross-modal distillation to enhance RGB gesture recognition accuracy. Lastly, the paper benchmarks the performance of state-of-the-art gesture recognition models on the MGR-Dark dataset, revealing specific challenges in recognizing gestures in dark videos.
limitations: limitations: 1. The paper contains numerous writing issues that significantly reduce its quality. 

a). There are many instances where necessary spaces are missing in the abstract and main text. For example, in the abstract: "multimodal videos(RGB, Depth, and Infrared)"; in the Introduction: "the paired data in daylight(See Figure 3),", "the student network(RGB feature learning)"; in the Method section: "As shown in Fig. 5,our Progressive"; in the experiments: "presented in Tab. 2.In order to mitigate"; and in several places in the references, like “Numerous datasets[3, 15, 16, 19, 23, 34]”、“recognition[3, 16, 23]” et al. The authors should carefully check the entire manuscript.
b). There are inconsistencies in the figure citation format, such as "Fig. 2, Fig. 5" versus "Figure 2, Figure 3, Figure 4". The authors should ensure consistent citation formatting throughout the paper.
c). There is an incorrect table citation in section 3.3 Diversity: "As we can see in Tab. 2, our...". The authors should thoroughly check for such errors.
d). References should be added to the dataset in Figure 1.

2. In Section 3.1, there is confusion regarding the equipment used for data collection: "The video clips in the MGR-Dark dataset are collected using Kinect V2 and Xinfrared T3S. We use Kinect V4 to collect RGBD video data. Kinect V2 is equipped with a 12-megapixel RGB camera and a 1-megapixel Depth camera, ensuring the quality of the captured videos." It is unclear whether Kinect V2 or Kinect V4 was used.  Additionally, the setup and positioning of Kinect and Xinfrared devices during data collection should be illustrated (use a graph or table to illustrate), including any angular differences and whether all collection environments were the same.

3. The information shown in Figure 4 seems unnecessary for the paper's quality. Calculating the mean and variance of grayscale images would suffice. It is recommended to integrate this information with Table 1.

4. In the Feature Converter Module, the rationale behind the sequence of Temporal attention followed by Spatial attention needs clarification. Does the order have any significant impact on the experimental results?

5. There are unfair comparisons in Table 2. For instance, the methods Decouple&Recouple [37] and UMDR [36] are originally based on RGB-D recognition, but the comparisons are made using only RGB or Depth, whereas the authors’ method uses RGB+Infrared during training. Furthermore, the process for obtaining the Depth results in Table 2 is unclear. Considering that Figure 5 describes a process from RGB to pseudo-Infrared, does a similar process exist for Depth? Or is the model trained based on the method described in Figure 5 and then tested directly using Depth?