Multi-Granularity Hand Action Detection

Authors:   <l-o>
  <o-o>
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 07:01:08

Abstract: Detecting hand actions in videos is crucial for understanding video content and has diverse real-world applications. Existing approaches often focus on whole-body actions or coarse-grained action categories, lacking fine-grained hand-action localization information. To fill this gap, we introduce the FHA-Kitchens (Fine-Grained Hand Actions in Kitchen Scenes) dataset, providing both coarse- and fine-grained hand action categories along with localization annotations. This dataset comprises 2,377 video clips and 30,047 frames, annotated with approximately 200k bounding boxes and 880 action categories. Evaluation of existing action detection methods on FHA-Kitchens reveals varying generalization capabilities across different granularities. To handle multi-granularity in hand actions, we propose MG-HAD, an End-to-End Multi-Granularity Hand Action Detection method. It incorporates two new designs: Multi-dimensional Action Queries and Coarse-Fine Contrastive Denoising. Extensive experiments demonstrate MG-HAD's effectiveness for multi-granularity hand action detection, highlighting the significance of FHA-Kitchens for future research and real-world applications. The dataset and source code will be released.

Main paper: 
 related work 2.1 ar & ad dataset: 
 Action Recognition (AR) Dataset. Pioneering AR datasets, such as KTH [49] and Weizmann [1] , have played a pivotal role in the advancement of this field, inspiring subsequent endeavors in constructing more challenging datasets. Existing studies on action recognition datasets can be divided into two main categories based on the types of actions: whole-body action and part-body action, such as UCF101 [54] , Kinetics [3] [4] [5] , ActivityNet [24] , FineGym [50] , and others [25, 28, 41, 51, 68] . These datasets primarily focus on whole-body actions, lacking fine-grained action information from specific body parts. Datasets like MPII Cooking Activities [47] and EPIC-KITCHENS [13] refine the action verb part and consider interacting objects, yet they do not describe the localization of action interaction regions or the relationships between interacting objects, crucial for HAD tasks. Representing hand actions solely with singledimensional verbs is insufficient given the diversity and complexity of real-world scenarios. To address this issue, the FHA-Kitchens dataset enriches hand action data by providing annotations for interaction region localization and interaction action categories. Action Detection (AD) Dataset. Compared to action recognition datasets, fewer datasets are available for action detection [22, 30] . This is due to the need to annotate the position and category of each action instance, which requires more efforts for dataset construction. The AVA dataset [22] focuses on human action localization, providing bounding box annotations for each person. However, this dataset primarily focuses on whole-body actions, providing location information for individuals rather than action interaction regions. Moreover, the provided action categories are mainly single-dimensional coarse-grained verbs (e.g., "sit", "write", and "stand"). FHA-Kitchens dataset addresses these limitations by providing precise bounding box annotations for each hand subinteraction region. Moreover, we refine the representation of action We provide precise localization information by meticulously annotating hand interaction regions and corresponding interaction objects using bounding boxes. (2) We offer two granularity for hand actions: coarse-and fine-grained. For fine-grained categories, we use multi-dimensional triplets to represent each sub-interaction region action, expanding the dimensionality of each frame to 9. (3)We not only focus on the interacting objects that generate interaction actions but also consider the active and passive relationships between these objects, capturing their contact areas.Action Recognition (AR) Method. Existing action recognition methods can be broadly summarized into two pipelines based on technical approaches. The first pipeline employs a 2D CNN [16, 19, 52, 61] to learn frame-level semantics and then aggregate them temporally using 1D modules. For example, TSN [60] divides an action instance into multiple segments, represents it with a sparse sampling scheme, and applies average pooling to fuse predictions from each frame. TRN [70] and TSM [34] replace pooling with temporal reasoning and shift modules, respectively. The second pipeline directly utilizes a 3D CNN [5, 15, 18, 40, 57, 62] to capture spatial-temporal semantics, such as I3D [5] , SlowFast [18] , and Video Swin Transformer [40] . On the other hand, AR methods can be categorized into coarse-grained [11, 12] and fine-grained [26, 37, 42, 43] based on the granularity of the actions. Some hand actions approaches [37, 42] use the EPIC-KITCHENS and MPII Cooking Activities datasets, from first-person and third-person perspectives, respectively. Another method [26] focuses on human whole-body actions in sports scenarios using the FineGym dataset [50] .Action Detection (AD) Method. Most state-of-the-art action detection methods [7, 17, 18, 45, 56] commonly follow a two-stage pipeline, utilizing separate 2D and 3D backbones for localization and video feature extraction, respectively. Since transformer [58] was introduced for machine translation, it has become a widely adopted backbone for sequence-to-sequence tasks [33, 59, 65] . Most recent methods [8, 9, 21, 63, 69] utilize a unified backbone to perform action detection. VAT [21] is a transformer-style action detector designed to aggregate spatiotemporal context around target actors. EVAD [8] , built upon the ViT framework, offers an end-to-end efficient video action detection method. WOO [9] and TubeR [69] are query-based action detectors that follow the detection frameworks of [2, 55] to predict bounding boxes and action classes, while STMixer [63] is a one-stage query-based detector that adaptively samples discriminative features. However, we observed that these methods primarily focus on individual human actions and overlook action interaction regions, interacting objects, and their relationships. Leveraging the advantages of transformer-based detection models, we propose an end-to-end solution capable of simultaneous hand action localization and recognition. 
fha-kitchens dataset 3.1 data collection and organization: 
 Data Collection. The proposed dataset is derived from the largescale action dataset Kinetics 700_2020 [53] , which comprises approximately 650K YouTube video clips and over 700 action categories. However, as the Kinetics dataset primarily focuses on human actions, most of the videos capture whole-body actions. To narrow our focus to hand actions, we performed filtering and processing operations on the original videos in three steps. (1) Content Localization: We observed that videos in kitchen scenes prominently showcase human hands. So we sought out and extracted relevant videos set against a kitchen backdrop. (2) Quality Selection: To ensure dataset quality, we selectively chose videos with higher resolutions. Specifically, 87% of the videos were recorded at 1,280 × 720 resolution, while another 13% had a shorter side of 480. Additionally, 67% of the videos were captured at 30 frames per second (fps), and another 33% were recorded at 24∼25 fps. (3) Duration Control: We imposed a duration constraint on the videos, ranging from 30 seconds to 5 minutes, to exclude excessively long videos. This constraint aimed to maintain a balanced distribution within the sample space. Finally, we collected a total of 2,377 video clips, amounting to 84.22 minutes of footage, encompassing 8 distinct types of dishes. Data Organization. The collected video data was reorganized and cleaned to align with our annotation criteria (Section 3.2). First, we split the collected video data into individual frames, as our annotated units are frames. Subsequently, we conducted further cleaning of the frames by excluding those that did not depict hands or exhibited meaningless hand actions. This cleaning process took into consideration factors such as occlusion, frame quality (i.e., without significant blur, subtitles, and logos), meaningful hand actions, and frame continuity. As a result, we obtained a total of 30,047 highquality candidate video frames containing diverse hand actions for the FHA-Kitchens dataset. Compared to the initial collection, 113,436 frames were discarded during the cleaning process.We recruited 10 voluntary annotators to annotate hand actions for each frame with high quality. Their responsibility was to annotate bounding boxes and multi-granularity action categories for each hand interaction region. To enhance annotation efficiency, we implemented a parallel annotation approach. We utilized the LabelBee tool for annotating bounding boxes and coarse-grained categories, while fine-grained action triplets were annotated on the Amazon Mechanical Turk platform. To ensure annotation quality, we conducted three rounds of cross-checking and corrections. The annotation content and criteria are detailed below.Bounding Box Annotation: We annotated the bounding boxes for both interaction regions (IR) and interaction objects (IO). ( 1 (2) IO: To better understand interaction actions, we also annotated the interactive object pair within each sub-interaction region using bounding boxes. For example, in L-O, we annotated objects directly touched by the left hand. In O-O, we annotated the interacting objects directly involved in hand actions (e.g., utility knife and carrot). However, during annotation, we may encounter overlapping bounding boxes, i.e., the same interacting object will satisfy two annotation definitions, for example, the utility knife in Figure 1 , which is both the object directly touched by the right hand in the R-O and the active force provider in the O-O. In this case, we annotate all the labels because the same object participates in different interaction actions and has different roles (Annotation details can be seen in supplementary material). Finally, we annotated a total of 198,839 bounding boxes, including 49,746 hand boxes, 66,402 interaction region boxes, and 82,691 interaction object boxes.Hand Action Annotation: We annotated coarse-and finegrained actions for each sub-interaction region. Coarse-grained categories, denoted by the generic terms "L-O", "R-O", and "O-O", represent the coarse actions within the sub-interaction regions. Different from existing fine-grained datasets. We annotate each fine-grained action category in a triplet format: <subject, action verb, object>. (1) Subject & Object: We considered the "activepassive" relationship between objects, where the "subject " refers to the active force provider (e.g., utility knife) and the "object" refers to the passive force receiver (e.g., carrot), and annotate them in order within the action triplet. In L-O or R-O, the subject represents the corresponding hand, while the object denotes the directly interacting object. Furthermore, to enrich the description of each action, we also considered the contact areas of interacting objects within each sub-interaction region. For example, as shown in the first green block in the middle of Figure 1 (b), we labeled the subject as "hand_left" and the object as "carrot_end". We referred to the EPIC-KITCHENS [13] dataset to define the object noun. (2) Action Verb: It describes the fine-grained hand action within the sub-interaction region. We used fine-grained verbs in the annotated action triplets and constructed the verb vocabulary by sourcing from EPIC-KITCHENS [13] , AVA [22] , and Kinetics 700 [4] .Overview of FHA-Kitchens. As summarized in Table 1 , we annotated hand action information for 30,047 frames from 2,377 clips, resulting in 880 action categories (including 877 action triplets), 130 action verbs, and 384 interaction object nouns. We have taken steps to refine the dataset by focusing on hand action categories and interaction regions, providing more precise localization bounding boxes and rich hand action categories for the three sub-interaction regions. Compared to the original action annotations in Kinetics 700_2020 [53] , the FHA-Kitchens dataset expands the action labels by 7 dimensions, increases the number of action categories by 52 times, and introduces 122 new action verbs. Furthermore, we provide bounding boxes for hand action regions (i.e., 66,402 interaction region boxes). This expansion significantly enhances the diversity of hand action annotations, provides valuable region-level contextual information for each action, and facilitates future research for a wider range of video understanding tasks. The FHA-Kitchens dataset is then randomly divided into the disjoint train, validation, and test sets, with a video clip-based ratio of 7:1:2.Annotation Statistics. Our annotation primarily focuses on hand interaction regions, interaction objects, and their corresponding interaction actions, resulting in a diverse array of verbs, nouns, and bounding boxes. Following the fine-grained annotation principles [13] , we ensured minimal semantic overlap among action verb-noun categories, rendering them suitable for multi-category action recognition and detection. (1) Verbs: The annotated dataset comprises 130 action verbs that have been grouped into 43 parent verb categories (Figure 2 and Figure 3 ). The three most prevalent parent verb categories, based on the count of sub-action verbs, are Cut, Hold, and Take, representing the most frequently occurring hand actions in kitchen scenes. Figure 3 visually depicts the distribution of all verb categories within FHA-Kitchens, ensuring the presence of at least one instance for each verb category. (2) Nouns:In the annotation process, we identified a total of 384 interaction object noun categories that are associated with actions, categorized into 17 super-categories. Figure 4 shows the distribution of noun categories based on their affiliations with super-categories. Notably, the super-category "vegetables & plants" exhibits the highest number of sub-categories, followed by "kitchenware", which aligns with typical kitchen scenes. (3) Bounding Boxes: We performed a comprehensive statistical analysis on the bounding boxes of the three sub-interaction regions and the corresponding interaction objects. Specifically, we focused on two aspects: the box area and the aspect ratio. Details can be found in supplementary material. Methods. We benchmark several representative action recognition methods [18, 40, 48, 59, 60] and detection methods [20, 46, 65, 71] with different backbone networks on the proposed FHA-Kitchens dataset based on the MMAction2 [10] and MMDetection [6] codebases. We establish three tracks using the provided dataset. SL-AD Track: The aim is to evaluate the supervised learning performance All models on the SL-AD, SL-AR, and DG tracks were trained and tested using NVIDIA GeForce RTX 3090 GPUs. For the SL-AD and DG tracks, we employ the mean Average Precision (mAP) [36] as the primary evaluation metric, while for the SL-AR track, Top-1 accuracy and Top-5 accuracy (%) are adopted. Detailed results of SL-AR and DG can be found in supplementary material.Results and Discussion. The results in Table 2 show that current detection methods perform well in learning single-dimensional coarse-grained categories like verbs or nouns. However, they struggle in learning multi-dimensional fine-grained action categories. Understanding the intricate nature of real-world hand actions, which encompass both coarse-and fine-grained information, underscores the significance of investigating multi-granularity action categories in HAD tasks, an area that poses significant challenges and remains largely unexplored. To fill this gap, we propose a novel method for multi-granularity hand action detection. 
a simple yet strong baseline 4.1 a multi-granularity framework: 
 Drawing inspiration from the image-based DINO [65] , we propose the novel MG-HAD method with specific novel designs in the decoder for multi-granularity hand action detection (Figure 5 ). MG-HAD consists of a backbone, a multi-layer Transformer encoder, a multi-layer Transformer decoder, and multiple prediction branch heads. Given a video clip, for each frame, we utilize backbones like ResNet [23] or Swin Transformer [39] to extract multi-scale features, which are then fed into the Transformer encoder along with corresponding positional embeddings. After enhancing features through the encoder layers, we initialize anchors as positional queries for the decoder using a mixed query selection strategy, following the design of DINO, without initializing content queries but leaving them learnable. It's worth noting that the original content queries focus on semantic information of single-dimensional categories, which is not suitable for fine-grained multi-dimensional categories in the new task. Therefore, we modify the single-content query to multi-content queries and introduce a Content Query Reorganization (CQR) module to obtain query sets focusing on three different sub-action dimensions, as detailed in Section 4.2. Additionally, similar to DINO, we have an extra CDN branch to perform contrastive denoising training. In contrast to the standard CDN method, we specifically devise a novel coarse-fine granularity contrastive denoising training approach to distinguish labels with different granularity levels, which will be discussed in Section 4.3.Comparing the results presented in Table 2 , it's clear that existing detection methods struggle with learning from multi-dimensional fine-grained labels. Fine-grained hand action detection poses a greater challenge compared to coarse-grained detection due to the need to discern subtle differences in similar hand actions. Additionally, multi-dimensional fine-grained labels provide important supervisory signals about subject, object, and action categories, as well as localization information. However, effectively encoding this information at different dimensions and leveraging these supervisory signals, particularly in terms of query design within the DETR series framework [2, 65, 71] , remains unexplored.Implementation: We observed that the current design of content queries mainly focuses on single-dimensional semantic information, i.e., single verb or noun categories. However, in fine-grained categories, we incorporate both verb and noun categories, generating multi-dimensional semantic information, i.e., <𝑐 1 , 𝑐 2 , 𝑐 3 > (𝑐 1 , 𝑐 3 ∈ nouns, and 𝑐 2 ∈ verbs), or more specific <s, a, o>. If we stick to the original design, content queries would consider <s, a, o> as a whole, learning global information from a single-dimensional perspective. To enhance the model's focus on local information of sub-categories, we transform a set of content queries 𝑄 = {𝑞 1 , ..., 𝑞 𝑛 } originally focusing on single dimensions into three sets of content queries, i.e., 𝑄 𝑠 , 𝑄 𝑎 , and 𝑄 𝑜 , focusing on different action dimensions. 𝑛 is the index of the original queries. Specifically, we first convert each query element 𝑞 𝑛 (bottom orange cubes in Figure 5 ) into three sub-queries, i.e., 𝑞 𝑛 𝑠 , 𝑞 𝑛 𝑎 , and 𝑞 𝑛 𝑜 , expanding 𝑁 original queries to 3 × 𝑁 sub-queries. Next, through our designed Content Query Reorganization (CQR) module, sub-queries focusing on the same action dimension (i.e., 𝑞 1 𝑠 , 𝑞 2 𝑠 , ..., 𝑞 𝑛 𝑠 ) are selected and reorganized to obtain a query set for each action dimension. Additionally, to ensure a comprehensive understanding of fine-grained categories, we introduce an action dimensional hyper-parameter 𝑤 𝑑 (𝑑 ∈ {s, a, o}), to add a certain proportion of weight to each query set, which is then sum with the global information (i.e., 𝑄 = {𝑞 1 , ..., 𝑞 𝑛 }). This process is formulated as:𝑄 𝑑 = 𝐶𝑄𝑅 𝑞 𝑛 𝑑 𝑁 𝑛=1 = 𝑁 ∑︁ 𝑛=1 𝑞 𝑛 𝑑 × 𝑤 𝑑 + 𝑄, 𝑑 = 𝑠, 𝑎, 𝑜𝑟 𝑜. (1)After passing through the CQR module, we obtain content query sets for the three action dimensions. These sets are then summed with the initialized anchors to yield multi-dimensional action queries. Each dimensional query set has a length of 𝑁 , resulting in a total length of 3𝑁 . Following the decoder layers, three query sets for different action dimensions (𝑄 ′ 𝑠 , 𝑄 ′ 𝑎 , and 𝑄 ′ 𝑜 ) are outputted.Subsequently, through the Split&Integration module, queries from different action dimensions with the same index (e.g., 𝑞 1 𝑠 , 𝑞 1 𝑎 ,𝑞 1 𝑜 ) are integrated to generate 𝑁 queries, each of which contains information from three action dimensions (top orange cubes in Figure 5 ):EQUATIONwhere 𝑆𝐼 represents the Split&Integration module. Finally, the matching process is conducted to predict hand action results. Analysis: In our design of multi-dimensional action queries, we introduce an action dimensional hyper-parameter 𝑤 𝑑 (𝑑 ∈ {s, a, o}), to control the proportion of local information (sub-categories) fused with global information (triplet categories). In the three action dimensions <s, a, o>, the a dimension is the most crucial for our task. Therefore, we use 𝑤 𝑎 as the central weight to dynamically adjust the weight proportions of the three action dimensions, with a total sum of 1. To determine the optimal weight distribution, we conducted a total of 10 comparative experiments under different backbones, with detailed results provided in supplement material.For object detection, DINO is highly effective in stabilizing training and accelerating convergence. With the help of DN queries, it learns to predict "no object" for anchors without nearby objects, thereby inhibiting confusion and selecting high-quality anchors (queries) for predicting bounding boxes. However, in HAD tasks where hand action categories may overlap or be similar, DINO primarily addresses the confusion of boxes but overlooks label categories, resulting in poor prediction capability for different granularity levels of hand action categories. To address this issue, we propose a Coarse-Fine granularity Contrastive DeNoising (C-F CDN) training approach to reject anchors with "incorrect granularity labels".Implementation: DINO introduces two hyper-parameters 𝛾 1 and 𝛾 2 , to control the scale of box and label noise, respectively. The generated noises are no larger than 𝛾 1 and 𝛾 2 , aiming to enable the model to reconstruct the ground truth (GT) from moderately noisy queries. We observed that DINO only designs two types of CDN queries for the box: positive and negative queries, while the label is set to be randomly generated. In the proposed method, while keeping the box settings unchanged, we further generate two types of CDN queries for the label: coarse-grained and fine-grained queries (dark and light yellow cubes in Figure 5 ). Moreover, unlike the strategy of randomly generating noisy labels, we add noise by specifying the noise position and noise category for different granularity labels. Specifically, coarse-grained queries add noise containing fine-grained information, while fine-grained queries add noise containing coarse-grained information, with the expectation of predicting the correct granularity label for each GT box. In Figure 5 , each CDN group comprises four types of queries: positivecoarse, positive-fine, negative-coarse, and negative-fine. If a frame has 𝑛 GT bounding boxes, a CDN group will contain 4 types of 2 × 𝑛 queries. Similar to DINO, we also utilize multiple CDN groups to enhance the effectiveness of the method. The reconstruction loss for bounding box regression includes l 1 and GIOU losses, while focal loss [35] is employed for classification.Analysis: When designing the noise label generation strategy, we replaced the "random" generation of noise with "specified", reducing randomness by specifying noise positions and categories. This ensures noise is added to different granularity labels, generating CDN queries encompassing various granularity. To determine the optimal setting, we considered the noise distribution of different granularity categories in real-world scenarios and ensured contrastive learning between coarse and fine-grained information.We conducted three sets of comparative experiments (Details can be found in supplementary material). The final selected setting, as shown in Eq. ( 3), exhibits the most significant improvement. Hence, subsequent experiments were conducted using this setting for further investigation. Our method's success lies in its ability to suppress confusion at the category level and select appropriate granularity to predict hand action categories, thus enhancing its ability to predict multi-granularity information.noise label = fine-grainedEQUATIONwhere 𝑖 indexes the multi-granularity action category for a specific instance (i.e., 0∼2 denote coarse-grained categories while 3∼C-1 denote fine-grained categories), and 𝐶 is the number of categories. "fine-grained" and "mixed-grained" denote that the noise label is chosen randomly from the fine-grained categories and the combination of the coarse-grained and fine-grained categories, respectively. 
experiments 5.1 experiments settings: 
 Dataset and Metric. Due to the absence of benchmarks for this new task, we evaluated our model and other representative detection models solely on the FHA-Kitchens dataset. We conducted experiments using two different backbones: ResNet-50 [23] pre-trained on ImageNet-1k [14] and Swin-L [39] pre-trained on ImageNet-22k [14] . All detection models utilize pre-trained weights on the MS COCO object detection dataset [36] . Furthermore, we not only report the overall validation results using mixed-grained labels but also separately report the validation results for coarse-grained and fine-grained sub-labels. We follow previous works and adopt mean Average Precision (mAP) [36] as the primary evaluation metric. Implementation Details. We trained the MG-HAD model on the FHA-Kitchen dataset using the MMDetection [6] codebase. Specifically, we utilized pre-trained weights on the MS COCO [36] object detection dataset and fine-tuned it on the hand action detection task on FHA-Kitchens. We trained the model under two different settings: 4scale-R-50 and 5scale-Swin-L. Following DINO [65] , we used the Adam optimizer [29] for model training, with an initial learning rate of 1 × 10 -4 and weight decay is 10 -4 . The experiments were conducted on the NVIDIA GeForce RTX 3090 GPUs, with a batch size of 2 for 4scale-R-50 and 1 for 5scale-Swin-L. By default, MG-HAD was trained for 12 epochs, taking approximately 5 hours. More details are provided in the supplement material.12-Epoch Setting. To demonstrate the effectiveness of our method for the multi-granularity HAD task, we compared it with representative strong baselines from the DETR series [2, 38, 65, 67, 71] on the FHA-Kitchens dataset under the setting of ResNet-50 backbone and 12 epochs. In particular, our method, DINO [65] , and DDQ [67] mainly report results under the 4scale setting. As shown in Table 3 , our method achieves much better accuracy in detecting mixedgrained hand actions, owing to the proposed C-F CDN module and multi-dimensional action queries. Specifically, it achieves an improvement of +2.3 AP on mixed-grained labels compared to the current strongest baseline DINO [65] under the same setting. Furthermore, compared to the classic DETR [2] , our method achieves a significant improvement of +14.7 AP. Note that our method not only performs well for mixed-grained labels but also shows improvement in the validation results for fine-grained sub-labels.Comparison with SOTA Detection Methods. To comprehensively and fairly validate the effectiveness of our method in enhancing the performance of multi-granularity hand actions, we compared it with other state-of-the-art (SOTA) detection methods on the FHA-Kitchens dataset, utilizing their optimal settings (refer to the MMDetection [6] codebase). DINO exhibits relatively fast convergence, achieving good results with just 12 epochs on the Swin-L backbone. Our method inherits the convergence capability of DINO but yields more significant improvements. We adopted the same settings as DINO [65] , utilizing both 4scale ResNet-50 and 5scale Swin-L backbones, trained for 12 epochs and 24 (2×) epochs, respectively. The results in Table 4 indicate the following:(1) Our method exhibits a significant improvement compared to the baseline [65] , which can be attributed to the design of handling finegrained information in our model; (2) Comparing models trained for 24 epochs and 12 epochs, the main improvement lies in the accuracy of fine-grained action detection. Since the FHA-Kitchens dataset contains overwhelming fine-grained categories over the coarse-grained ones, the model's representation capacity may be primarily utilized for fitting fine-grained categories;(3) Under the 5-scale Swin-L backbone, our method achieves a significant improvement of 59.4 AP for mixed-grained hand actions with just 12 epochs. This indicates that using a more powerful backbone [39] can improve both coarse-and fine-grained action detection accuracy. Specifically, the detection accuracy of fine-grained actions is increased by +3.1 AP and the detection accuracy of coarse-grained actions is increased by +1.3 AP. The visualization of the detection results can be found in the supplement material.Effectiveness of New Components: Our method utilizes the multi-dimensional action queries for multi-dimensional information processing, as introduced in Section 4.2, and the C-F CDN module for multi-granularity information processing, as described in Section 4.3. To further validate the effectiveness of these components, we separately isolated them from the model and evaluated the performance under two settings: 4scale ResNet-50 and 5scale Swin-L, as shown in Table 5 , where the baseline denotes the original design proposed by DINO [65] . As can be seen, while the strong baseline DINO [65] has already surpassed previous models, the proposed MG-HAD introduces two novel designs that notably boost performance in hand action detection. Each module significantly enhances the baseline on both backbones, and their combined effect further enhances performance, demonstrating their complementary role in understanding multi-granularity hand action information. 
conclusion: 
 In this paper, we present the first study on multi-granularity hand action detection, aiming to understand the diverse hand actions through localizing regions and recognizing various granularity categories of hand actions. We establish FHA-Kitchens, the first fine-grained hand action detection dataset, comprising 30,047 highquality video frames, 198,839 bounding boxes, and 880 hand action categories. Through systematic evaluation, we identify that existing detection methods excel in coarse-grained actions but struggle with fine-grained ones. To address this, we propose MG-HAD, a simple yet strong baseline model leveraging the Transformer detector with two novel designs. It outperforms previous methods across various granularities of hand actions. FHA-Kitchens and MG-HAD can serve as a valuable testbed and baseline for future research.  
: 
 The annotation of hand actions in existing relevant datasets, where UCF101 [54] and Kinetics700 [53] are whole-body action datasets, while MPII Cooking [47] and EPIC KITCHENS [13] are hand action datasets. (b) The annotation of hand actions in our dataset. The left shows some frames extracted from 8 dish categories. The right illustrates the annotation process of hand actions in "fry vegetable".Pioneering datasets such as MPII Cooking Activities [47] and EPIC-KITCHENS [13] have been developed to facilitate hand-action research. However, they exhibit limitations including insufficient representation of hand-action granularity, lack of annotation of hand-action interaction regions, and neglect of the relationships between interacting objects. As shown in Figure 1 (a), they offer only coarse-grained annotations for hand actions like "cut" rather than the fine-grained multi-dimensional categories like "<knife, cut slice, carrot>". These limitations hinder the study of detecting fine-grained hand actions and exploring their spatial relationship, leaving challenges in hand action detection unresolved. Therefore, establishing a large-scale benchmark with rich hand-action annotations is essential for advancing research in this field.To this end, this paper presents a novel dataset FHA-Kitchens, focusing on rich and fine-grained localization and categorization information of hand actions in kitchen scenes. The FHA-Kithcens dataset encompasses a total of 2,377 video clips and 30,047 frames from eight different dish types (Figure 1 (b) left). Each frame includes meticulously annotated hand action information, featuring high-quality annotations of hand interaction region boxes and corresponding coarse-and fine-grained categories. Our data were extracted from publicly available large-scale action datasets [53] , focusing on videos relevant to hand actions. Subsequently, frames underwent cleaning and were annotated by ten expert voluntary annotators. To excavate more hand action information, we refined the annotation process in two aspects (Figure 1 (b) right):(1) Hand interaction regions. These were subdivided into three sub-regions based on hand-object interaction: Left hand-Object interaction region, Right hand-Object interaction region, and Object-Object interaction region. Each sub-interaction region was annotated with bounding boxes and coarse-grained categories, denoted as "L-O", "R-O", and "O-O". (2) Hand interaction actions. To enhance the model's understanding of hand actions, we further refined the category of each sub-interaction region by expanding single-dimensional action categories into multi-dimensional ones, annotated in the format of triplets: <subject, action verb, object>, abbreviated as <s, a, o>, where "s-o" denotes interacting objects, and "a" represents the interaction action between the objects. Additionally, when annotating "s" and "o", we considered the specific contact area between the interacting objects, labeled as "name_contact area" (e.g., "carrot_end"). Overall, we meticulously annotated 880 hand action categories (coarse-and fine-grained) for approximately 220k bounding boxes, with each category corresponding to a subinteraction region's localization box. Fine-grained categories per frame have nine dimensions, resulting in 877 action triplets.Hand Action Detection (HAD) is a sub-area of Action Detection (AD) research, which has a close relation to generic object detection (OD) in the image domain. We systematically evaluated several representative AD methods [20, 46, 65, 71] on FHA-Kitchens, observing varied performance across different levels of granularity: "Coarse-grained" and "Fine-grained". Existing methods perform significantly worse under fine-grained labels compared to coarsegrained ones, indicating that these detection methods have a better understanding of single-dimensional coarse-grained labels (i.e., single verb or noun). However, real-world hand actions often involve both coarse-and fine-grained information simultaneously. Therefore, exploring the impact of multi-granularity action categories in HAD tasks is both interesting and practically significant.Among the state-of-the-art detection methods, DINO [65] showed relatively strong performance across different granularity hand actions. Building upon DINO, we propose MG-HAD, a novel baseline for hand action detection (HAD). MG-HAD contains a backbone, a multi-layer Transformer encoder, a multi-layer Transformer decoder, and multiple prediction branches. To better adapt to multigranularity hand actions, we propose two novel designs: (1) Multidimensional information processing: To enhance the model's understanding of fine-grained information, we replace the original single-dimensional content query in the decoder with multidimensional content queries to focus on multiple aspects of hand actions. Additionally, we introduce a Content Query Reorganization (CQR) module to generate three query sets focusing on different action dimensions as decoder inputs. (2) Multi-granularity category processing: We observed that the DINO's CDN (Contrastive DeNoising) module mainly focuses on bounding boxes for contrastive denoising training, while the labels are not specially designed. To enable the model to better learn and distinguish coarsegrained and fine-grained action labels, we devise coarse-grained and fine-grained sample queries for contrastive denoising training of the labels, by adding noise to different granularity categories with specified noise positions and classes. Besides, we investigate the pre-trained ResNet50 [23] and Swin-L [39] models as backbones to extract multi-scale visual features. During training, following the DN-DETR [32] method, we add ground truth labels and boxes with noises into the Transformer decoder layers to stabilize bidirectional matching, and also adopt deformable attention [71] for improved computational efficiency.In summary, our contributions can be summarized as follows:• To the best of our knowledge, we are the first to study the problem of multi-granularity hand action detection and establish the first hand-action dataset FHA-Kitchens, which includes both hand interaction region localization and multi-granularity category annotations. This dataset can serve as a benchmark for hand action detection tasks. • We systematically investigated the impact of different granularity hand action information in kitchen scenes on the hand action detection task and provided insights about the evaluation protocol, performance analysis, and model design. • We propose a novel multi-granularity hand action detection method named MG-HAD, which is designed from the perspectives of multi-granularity and multi-dimensionality. This method incorporates Multi-dimensional Action Queries and a Coarse-Fine Contrastive Denoising module to address the mixed-grained HAD problem. MG-HAD demonstrates its effectiveness in hand action detection and could serve as a strong baseline. 
 
 Appendix:  


Decision: This work is focused on detecting fine-grained hand action in videos. It proposed a new dataset for this task with detailed bench-marking/comparison of existing methods. It initially received 2x borderline accept and weak accept ratings. The main concerns were limited size of dataset, limited scope to kitchen settings, and weak comparisons. The authors provide a rebuttal and most of the concerns were addressed. The final ratings are 2x weak accept and borderline accept. The most critical remaining concerns are size of dataset and used baselines. These limitations are important, but the other strengths of this work makes this a good contribution to the community and AC recommends acceptance of this work.

Reviews: review: paper_summary: paper_summary: The paper focuses on the challenge of detecting fine-grained hand actions in videos, specifically within kitchen scenes, which has diverse real-world applications. To address this, the authors introduce the FHA-Kitchens dataset, comprising 2,377 video clips and 30,047 frames annotated with approximately 200k bounding boxes and 880 action categories. This dataset offers detailed annotations for both coarse- and fine-grained hand actions, filling a significant gap in the field.

Furthermore, the paper presents MG-HAD (Multi-Granularity Hand Action Detection), an innovative end-to-end method designed to handle multi-granularity hand actions. MG-HAD incorporates Multi-dimensional Action Queries and Coarse-Fine Contrastive Denoising to enhance detection accuracy. Extensive experiments demonstrate MG-HAD's effectiveness, highlighting the importance of the FHA-Kitchens dataset for future research and practical applications. The results underscore the method's superior performance in detecting hand actions across different granularity levels, establishing a new benchmark in the field.
strength_weakness: strength_weakness: Novelty: Introducing the FHA-Kitchens dataset, which provides comprehensive annotations for both coarse- and fine-grained hand actions, filling a critical gap in the field.

Technical Approach: The MG-HAD method is robust, incorporating Multi-dimensional Action Queries and Coarse-Fine Contrastive Denoising to effectively handle multi-granularity hand actions.

Evaluation: Extensive experiments demonstrate MG-HAD's superior performance on the FHA-Kitchens dataset, providing a detailed comparison with existing methods.
limitations: limitations: Lack of Novelty in Methodology: While the FHA-Kitchens dataset is novel, the proposed MG-HAD may not present significant innovation compared to existing methods. The paper could benefit from a clearer articulation of the unique challenges addressed by their approach.

Dataset Scope: Although FHA-Kitchens is detailed, it is limited to kitchen scenes. This narrow focus may restrict the generalizability of the findings to other contexts or environments where hand actions are critical.

Clarity on Annotation Process: While the annotation process is described, more details on the consistency and accuracy of the annotations would be beneficial. This includes how annotator bias was minimized and how inter-annotator agreement was ensured.
paper_summary: paper_summary: This article proposes a new video action recognition dataset for kitchen hand action interaction, which includes multi granularity interaction actions and detection boxes, filling a certain gap in the field of video action recognition. In addition, to address the multi granularity problem in hand action recognition tasks, the paper also proposes MG-HAD, an end-to-end multi granularity hand action detection method.
strength_weakness: strength_weakness: The article proposes a new dataset for human action recognition that includes multi granularity annotations. This dataset includes multiple annotation forms and can be applied to various tasks including action detection, action recognition, hand action recognition, object detection, action interaction recognition, etc. It provides a data foundation for subsequent research on multi granularity action recognition. In addition, the article also provides a baseline for solving multi granularity hand action recognition for comparison.
limitations: limitations: The number of videos and total frame rates in the new dataset are relatively small, and future practical research in application may be limited. In addition, the other models provided in the paper for comparative testing are not novel enough and are relatively old.
paper_summary: paper_summary: The core contribution of this paper is the multi-granularity hand action detection dataset named FHA-Kitchens, which provides fine-grained hand action localization and classification information. The dataset fills the gaps in action localization and fine-grained classification in previous full-body action datasets and hand action datasets. To better handle multi-granularity hand actions, the authors propose the MG-HAD method, which introduces multi-dimensional action queries and a coarse-fine contrastive denoising module.
strength_weakness: strength_weakness: - This paper is the first to study the problem of multi-granularity hand action detection. The proposed FHA-Kitchens dataset provides coarse-grained action category annotations while also offering fine-grained multi-dimensional annotations.

- The annotation work of the dataset was completed by 10 expert annotators and underwent three rounds of cross-checking to ensure annotation quality. The action verbs and interacting object nouns covered in the dataset are broad and well-distributed. 

- The MG-HAD method based on the DINO architecture, effectively solves the problem of multi-granularity hand action detection.

- The paper is well written.
limitations: limitations: - In addition to the interactions of "left hand-object", "right hand-object", and "object-object", should the interaction of "left hand-right hand" also be considered?

- For the macro task of "hand action detection", the dataset proposed in this paper lacks annotations of hand pose, so the description of "hand action" tends to focus on the interaction between hands and objects (action triplets), rather than hand actions in the traditional sense.

- How does the MG-HAD method perform on traditional datasets (such as UCF101, Kinetics700, EPIC-Kitchens) in terms of action recognition or action detection? Does the MG-HAD method of introducing multi-granularity hand actions exhibit fine generalization?